Comment by edude03
13 hours ago
Envoy is legendary in (dev)ops circles, but I don't understand what it lends to the AI space. I feel like building a separate backend service that runs behind envoy would make more sense but that's just me.
13 hours ago
Envoy is legendary in (dev)ops circles, but I don't understand what it lends to the AI space. I feel like building a separate backend service that runs behind envoy would make more sense but that's just me.
We agree Envoy is legendary - and per se doesn't lend anything to the AI space. That's essentially what we are doing here, building on top of Envoy to add capabilities specifically for AI and prompts. For instance, we use Envoy's filtering capabilities to handle and process prompts - this was get to keep all the robustness and security features for TCP/HTTP from Envoy and solve the critical but undifferentiated tasks related to prompts like safety, observability, routing, function calling, etc.