You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I am trying to run garak on an llm that responds in the form of event streams like "event:MESSAGE\ndata: He" , "event:MESSAGE\ndata: llo" and so on. The output is not json but I am using the rest generator. I have seen that some evaluators fail probably due to the other content that comes with the textual data. Is there a way to configure the rest generator to include or exclude certain tags?
Also, the LLM I am testing has a rate limiter, is there a configuration to add a delay between requests? Or should I do it over garak's source code?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hi, I am trying to run garak on an llm that responds in the form of event streams like "event:MESSAGE\ndata: He" , "event:MESSAGE\ndata: llo" and so on. The output is not json but I am using the rest generator. I have seen that some evaluators fail probably due to the other content that comes with the textual data. Is there a way to configure the rest generator to include or exclude certain tags?
Also, the LLM I am testing has a rate limiter, is there a configuration to add a delay between requests? Or should I do it over garak's source code?
Thank you in advance,
Beta Was this translation helpful? Give feedback.
All reactions