The layers of an AI agent: context, harness and connectors

In short. When AI agents fail in production, the cause is more often what the model received and what it was allowed to do than the model itself. In the ZeroFive.AI method the architecture of an agent is split into three layers: context, which decides what the model sees at each decision point; the harness, which executes its proposals within permissions, budgets and approval gates; and connectors, one per system of record, which bring in data with source and timestamp and carry the identity of the person doing the work. Permissions, authoritative sources and approvals are written in code, the model chooses only inside that perimeter, and every build phase is measured against evaluation cases taken from the real process.

If we open the logs of an AI agent that went wrong in production, the most frequent case is a model that reasoned correctly over the wrong information: a policy replaced months ago but still sitting in the index, the status of an order read from a nightly copy instead of the order system, the record of a customer whose name resembles the right one, an action executed twice because the process restarted from scratch after an interruption. None of these errors is fixed by changing the model, because they depend on what the model received and on what it was allowed to do.

Rakesh Sheshadri has published on Medium a very extensive map of what he calls the context layer, a hundred concepts in sixteen parts, from memory to retrieval, from state management to security, with a goal we share: give each agent the minimum sufficient, authorised, trustworthy and up-to-date information for the task it is performing. We start from there, but we reorganise the discussion around the separation we use in our projects and in building our own internal tools, the one between context, harness and connectors, because that separation determines where the model can choose and where a deterministic system has to choose instead.

The piece is technical and is written for people who design and build agents, but the question running through it is the one we also put to management before any project: which decisions we leave to the model and which ones we write in code.

The layers around the model

The separation we use places the model at the centre of three layers with distinct responsibilities: context determines what the model sees, the harness determines what it can do and how its proposals become actions, and connectors determine how the system reaches the company's data and applications. Sheshadri describes seven layers, from data sources up to agent execution, with governance and observability cutting across all of them; our version groups them by type of decision, because that boundary is where you establish what to entrust to a probabilistic system and what not to.

The model receives a context package and returns proposals; every read and every write towards systems of record goes through the harness and a connector, never directly from the model.

The context layer

Context is what the model sees at the moment it has to decide, and the first design choice concerns precisely that moment: an agent working on a multi-step process needs different information when it plans and when it executes a call, so context is assembled at every decision point rather than once at the start of the session. Sheshadri frames it as an optimisation problem, choosing from the available evidence the most useful subset for the task within constraints of tokens, latency, authorisation and freshness, a framing that shifts attention from quantity to selection.

In practice the context layer produces a structured package, with fields the model learns to recognise and that whoever is debugging can read: the task instructions, the current workflow state, the retrieved evidence with its provenance, verified memories, conflicts between sources and the information that is missing. The last two fields are the ones we most often see neglected, and they are what allow the model to say it does not have enough to go on instead of filling the gap with a plausible answer.

Every piece of evidence carries the metadata without which the model has no way to weigh it, namely the source, the time it was read and how reliable that source is for that type of data. The same CRM can be authoritative for a customer's contact person and count for little on the status of a payment, which belongs to billing, and this map of authoritative sources has to be written before building retrieval, because ranking by semantic similarity knows nothing about authority.

Freshness works in a similar way. A product description can stay valid for months while warehouse availability changes within minutes, and a system that puts both in the same index with the same update policy ends up serving the model old numbers presented as current; in our projects the index is for finding and the source system is for confirming, and anything that changes faster than the indexing cycle is read in real time through a connector.

Retrieval in the narrow sense is a choice made by type of question, and a planner that picks the strategy and reranks results before assembling them produces smaller and more precise contexts than any single pipeline applied to everything:

• semantic search for questions about concepts
• keyword search for exact identifiers, such as an invoice number
• relationship traversal for questions that move from one entity to another
• direct queries to operational systems for data that changes

Memory and workflow state

Memory and state are often treated as the same thing, because both exist so that the agent does not start from scratch, but they answer different questions and have to be designed separately. Memory keeps what is worth knowing from one interaction to the next (a preference, a fact about a customer, the outcome of a case already resolved), while state describes where a running process stands, what has been done, what is left and which actions have already had effects outside the system.

For memory, Sheshadri's distinction between semantic, episodic and procedural memory has practical consequences, because each type follows its own update policy: a fact is re-verified periodically, a past event is never rewritten, a procedure comes from a controlled source and the agent does not change it on its own. Every memory record we design carries the subject, the source supporting it, a confidence level, an access scope and a review date, and above all it separates what the user said from what the system inferred. A memory without these fields turns, within a few months, into an archive of assumptions that nobody is able to verify any more.

Writing to memory goes through the harness, like any other write. The model proposes what to remember, a deterministic check verifies type, scope and retention, and for the most sensitive information a person decides; it is the same pattern we use internally, where the notes an agent extracts from meetings and email enter the project workspace only after explicit approval, gathered in a package that is either accepted or rejected.

State calls for different tools. A workflow that lasts hours, perhaps paused while waiting for human approval, needs persistent checkpoints to resume after an interruption, an event log from which to reconstruct what happened, and idempotency keys on external actions, because an agent resuming from a checkpoint must not place the same order or send the same email twice. When work passes from one agent to another, the handover is a structured object with the goal, the verified results, the open hypotheses and the permitted actions; a transcript of the previous agent's reasoning adds tokens and rarely any information useful to the agent receiving it.

The harness

We call harness the execution environment around the model, the loop that passes it context, receives its proposal, executes tool calls, observes the result and decides whether to continue, stop or involve a person. In Sheshadri's map these functions are spread across the gateway, orchestration and state management; we prefer to keep them together, because they are the parts of the system where an error leaves the boundary of the answer and has effects on customers, orders and documents.

The harness determines which tools are available for a given task and under which identity they are called. Permission checks happen here, in code, before every call, and they do not depend on what the model understood or on what is written in a retrieved document. This is also the structural defence against prompt injection: text inside an email or a web page asking the agent to export data cannot extend the list of authorised tools, because that list does not pass through the model. A text filter reduces the risk, while a narrow tool perimeter limits the damage even when the filter fails.

Actions fall into two families with different rules. Reads can proceed autonomously within a budget of steps and cost, while writes, and irreversible ones in particular, go through an approval gate whose position depends on the level of delegation chosen for that process. In our vocabulary an AI-assisted process leaves the work to the person and uses the agent at the margins, an AI-first process lets the agent do the work and has every output reviewed, and an AI-native process hands full execution to the agent and asks people to write policies and guardrails. Technically, moving from one level to another is a change to the harness configuration, to where the gates sit and who opens them, with the same model as before.

Then there are responsibilities that are less visible but decisive in production: termination rules that prevent endless retrieval loops, retry handling with idempotency keys, and a complete log of every call together with the context that preceded it. That log is what lets you answer, after an error, questions such as which version of the policy the agent had read or whether it was allowed to see that record, without asking the model to reconstruct its own reasoning after the fact.

Connectors

A connector is the adapter between the system and a single system of record, such as the order system, the CRM, the document archive or email. MCP has made connectors much cheaper to build, because it standardises how an AI application discovers and invokes tools and resources exposed by an external server; the protocol, however, defines how you talk to the system, and leaves to the designer every choice that determines whether that connector is safe and useful.

Identity decides a large share of the risk. A connector using a service account with full access hands the model permissions that no user of the process holds, while a connector that propagates the identity of whoever started the task inherits the controls already present in the source system, including isolation between customers or between group companies. The same care applies to scope: read-only tools and write tools kept separate, each with only the parameters it strictly needs, because a generic interface of the "run a query" kind leaves to the model decisions that belong to the designer.

Every response from a connector should come with the metadata the context layer needs (record identifier, time of reading, source) and with explicit limits on pagination and size, so that a truncated response or a format error is recognised as such instead of ending up in the prompt as valid data.

How many connectors to build is an architectural choice. If every agent has its own access to the CRM, its own document index and its own customer memory, within a few months the company has several versions of the same reality and no single place from which to revoke a permission once. The model we follow provides one connector per system of record, shared across agents and governed by a single set of rules, and it is the approach we used to expose our own internal tools to agents, the management system and the project workspace, both through MCP and with writes always subject to approval.

Separation of responsibilities

The separation between layers becomes operational when every decision the agent makes has an explicit owner, which can be the model, a rule written in code or a person. The table is the one we use as a checklist during design.

Decision	Who decides	Layer
Which information is relevant to the task	Model, on already filtered candidates	Context
Which source is authoritative for a given piece of data	Rule written upstream	Context
Whether the evidence is sufficient	Model, with mandatory reporting of gaps	Context
Whether the user can see a record	Deterministic code	Connector
Which tool to call among those permitted	Model	Harness
Whether an irreversible action can go ahead	Person or rule, depending on the level of delegation	Harness
What goes into memory	The model proposes, a rule or a person approves	Harness
When to stop	Termination rule and budget	Harness

Every row in which the middle column says "model" where it should say "code" or "person" is a risk that no benchmark of the model will bring to light, because in that case the model is working exactly as it was configured.

Build phases

A context layer is not designed all at once, and the sequence we propose starts from measurement before code. Before writing retrieval you build a set of evaluation cases taken from the real process, with the expected answers, the sources that should support them and the access boundaries, deliberately including the difficult cases Sheshadri lists at the end of his piece: conflicting documents, a policy that has just been replaced, a revoked permission, retrieved content that contains instructions, a workflow interrupted halfway, a handover between two agents.

On that basis the phases follow the order he also proposes.

1. Hybrid retrieval with citations and source-level permissions.
2. Persistent state, checkpoints and memory with retention policies.
3. Connectors to operational systems, with freshness rules and separation between reads and writes.
4. Context shared across several agents, with handover contracts and separate memory spaces.
5. Entity relationships, adaptive retrieval and temporal reasoning.

The difference lies in the gate between phases. You move to the next phase when the previous one passes the agreed threshold on the evaluation cases, and the metric that decides is correct task completion, with retrieval precision and coverage used as diagnostics to understand where to intervene. Applied this way, the path often stops before the fifth phase, because for many processes a knowledge graph or agentic retrieval adds cost and latency without moving the result, and that is worth knowing before funding them.

The most frequent design errors

The errors we encounter most often, including those we made in our own first agents, almost all come from a responsibility placed in the wrong layer.

• Permissions left to the prompt, with instructions such as "do not show other customers' data" instead of a filter applied by the connector.
• An index used as the source of truth for fast-changing data, such as prices, stock levels or order status.
• A memory that keeps everything, without separating stated facts from inferences and without review dates.
• A different retrieval stack for each agent, with diverging indexes and permissions to revoke in several places.
• Summaries used instead of sources in decisions where the exception matters, such as a contract clause.
• External actions without idempotency, which are executed again after an interruption.
• Connectors with all-powerful service accounts, giving the agent more access than anyone using it.

In this architecture the model is the component bound to change most often, for cost, for performance or because of constraints on where data has to stay. If context, harness and connectors are separate, the change touches a configuration and is verified by rerunning the evaluation cases; if the rules live in prompts and permissions in instructions, every new model reopens a project to be validated from scratch. Counting how many of the agent's decisions would still hold on the day the model changes is the quickest way to see how much of the system was designed and how much was left to chance.