Old problems, new surface: security in AI systems

The system assumption

The assumption around new AI features is that they are new, so the risks must be new too, and attention goes to the model. Meanwhile the plumbing gets built quickly: an internal endpoint that calls the model, a service that stores prompts and responses for debugging, a self-hosted tool to watch it all. That plumbing is ordinary software making ordinary mistakes, and because everyone is looking at the model, the plumbing gets less scrutiny than a normal feature would.

Where the assumption breaks

The classic failures come first

Across AI-adjacent systems we have found internal endpoints that drive the model with no authentication on them, and observability tooling reachable by anyone with the URL. These are not AI vulnerabilities. They are missing auth and exposed services, the same findings we would write for any application. The AI context did not change the bug; it just gathered the attention elsewhere while the bug shipped. (The default-privilege version of this is its own note: an AI-observability instance that made every new sign-up an owner. See verification is an access-control primitive.)

The observability store is a new sensitive asset

AI systems tend to log heavily: prompts, responses, intermediate reasoning, tool calls, sometimes the data fed into the model. That store is enormously useful for debugging and enormously sensitive, because it can contain whatever users and systems put in front of the model, which is often private data. An exposed trace store is a data breach of a kind that did not exist a few years ago. It needs the same access control, retention limits, and exposure review as any datastore holding user data, and it frequently gets less, because it is thought of as "logs."

The model's input is a trust boundary

This is the genuinely new one. A model treats the text it receives as instructions to follow, and if attacker-controlled content reaches that text, the attacker is partly writing the instructions. We have confirmed prompt injection in a model-response endpoint: crafted input steered the model's behavior. The severity depends on what the model is wired to do. A model that only returns text can be made to say the wrong thing. A model connected to tools, data, or actions can be made to do the wrong thing, on behalf of whoever's content it read.

Reconstructing it

  request flow around a model feature

  user input ---------------+
                            v
  [ internal model endpoint ]  <-- often missing auth
        |   builds a prompt from user + system text
        v
  [ model ]  <-- reads it all as instructions (injection surface)
        |   may call tools / read data / take actions
        v
  [ trace store ]  <-- records prompts + responses (new sensitive asset)

  three checkpoints, each a classic control applied to a new place:
  authenticate the endpoint, isolate untrusted input, protect the store.

Why the obvious defenses didn't solve it

Treating the model as the whole system. Hardening the model, or picking a "safer" model, does nothing for an unauthenticated endpoint or an exposed trace store. Those are perimeter and access-control problems the model has no say in.

Prompt-level filtering as the only injection defense. Blocklisting "ignore previous instructions" and similar phrases is the input-validation trap in a new costume. Instructions can be phrased countless ways, arrive indirectly through content the model retrieves, and hide in data the developer did not think of as instructions. Filtering the obvious phrases raises the bar slightly and does not close the boundary.

Assuming internal means unreachable. An endpoint described as internal is only internal if something enforces that. Several of the exposed AI endpoints we have seen were "internal" in intent and reachable in fact.

Root cause

The root cause is that AI features are held to a lower standard than the rest of the application, on the assumption that their risks are novel and separate. In practice they inherit every classic risk and add two assets to protect. So the model to hold is layered. First, treat the AI plumbing as ordinary software and apply the ordinary controls: authenticate every endpoint, do not expose internal services, default to least privilege, protect the trace store like the user-data store it is. Then, on top, treat the model's input as an untrusted channel: assume any content that reaches the prompt may be adversarial, and constrain what the model is empowered to do so that a successful injection has a small blast radius.

How to test for this class of failure

• Inventory the whole system, not just the model: every endpoint that calls it, every store that logs it, every tool it can invoke, every admin surface around it.
• Test each endpoint for authentication and authorization as you would any API. "It is only called internally" is a claim to verify, not accept.
• Find the trace/observability store and check its exposure, access control, and what it actually contains. Treat it as a datastore of user data.
• For injection, test whether untrusted input changes model behavior, and crucially whether the model can reach tools, data, or actions. Map the blast radius: what can a controlled model actually cause.
• Include indirect injection: content the model retrieves or summarizes, not just the direct prompt field.

How to design the control correctly

• Apply your existing controls to the AI plumbing. Authentication, authorization, exposure management, least privilege, and datastore protection are not optional because a model is involved.
• Isolate untrusted input in the prompt. Keep system instructions and untrusted content separated, and never let retrieved or user content silently become privileged instruction.
• Constrain the model's authority. Give it the least capability it needs. If it can call tools or take actions, gate those actions with their own authorization checks rather than trusting that the model will only do the right thing.
• Protect and minimize the trace store. Access-control it, limit retention, and avoid logging more sensitive data than debugging genuinely requires.

What we took away from it

The headline risk in AI security is prompt injection, and it is real. But in the systems we have actually tested, the first findings were the oldest ones, missing auth, exposed services, over-privileged defaults, sitting under a layer of new-technology attention. The practical lesson is unglamorous: hold AI features to the same security standard as everything else first, then add the model-specific controls. An AI system that has not solved authorization has not earned a conversation about adversarial prompts yet.