What Is an AI Gateway? The Tech Behind This Week’s LiteLLM Breach

What it is
An AI gateway is a piece of software that sits between your applications and the AI services they call. Think of it like the switchboard at an office that routes calls to the right extension. Instead of every app inside your business holding its own OpenAI key, Anthropic key and Azure OpenAI key, they all talk to the gateway, and the gateway talks to the AI providers on their behalf. LiteLLM — the open-source tool patched this week for a critical vulnerability reported by The Hacker News — is one of the most popular gateways out there, with more than 45,000 stars on GitHub and use across plenty of Australian engineering teams.

Why it matters right now
A gateway is meant to make AI use safer. Central logging, key rotation, rate limiting, cost tracking — all the things you can’t easily do when every app handles its own provider keys. But it also concentrates risk. Every credential, every prompt and every model response routes through one box. If that box is exposed to the internet and has a vulnerability — like the SQL injection bug being actively exploited in LiteLLM — an attacker doesn’t need to hit ten apps. They just need to hit the gateway. That’s exactly what happened over the past week, with attackers pulling provider API keys and proxy configuration straight out of the database within roughly 26 hours of the bug becoming public.

What it means for your business
If anyone on your team has spun up an AI gateway — including a “just for testing” deployment a developer set up months ago — treat it the way you’d treat any production system. Keep it patched, put it behind your firewall or VPN where possible, and rotate the keys it holds on a schedule. If you don’t know whether you have one running, that’s the first answer to track down. Shadow AI almost always lives outside the IT inventory, and you can’t protect what you don’t know about.

The All IT Services cybersecurity team can help you find the AI tools running across your business and put proper controls around them.