Comment by lionkor

Untrusted inputs to systems with agency or access to privileged data. Here’s a data exfiltration example in Google AI Studio:

https://x.com/wunderwuzzi23/status/1821210923157098919

That's a fair point - technically it protects the application from malicious attempts to subvert the desired LLM experience. The more specific language (and I think we could do better here) would be that Arch ensures users remain within the bounds of an intended LLM experience. That at least was the intention behind "ensure safe user interactions"...

Jailbreak ensures a smooth developer experience by controlling what traffic from user make its way to the model. With jailbreak (and other guardrails soon to be added) developers can short-circuit response and with observability developers can get insights on how users are interacting with their APIs.