RAG and connectors: which channel for which data

In short. Semantic retrieval and connectors answer different questions. A vector index finds the passages of text most similar to the question, so it works on documents, procedures, contracts and material that changes rarely. A connector queries a system and returns the current value, with the permissions of whoever is asking, so it serves balances, stock levels, case status, deadlines. A system that puts a balance in a vector index is answering from an old snapshot without saying so.

This is the architectural choice that decides the perceived quality of an enterprise assistant, more than the choice of model.

Two channels with different properties

Property	Semantic retrieval (RAG)	Connector to a system
What it finds	Passages similar to the question	The record matching a query
Freshness	As of the last indexing run	As of the source system
Authority	Depends on the document retrieved	The system is the source
Permissions	Must be rebuilt on the index	Inherited from the system
Aggregates and counts	Unreliable	Native
Traceability	Reference to document and passage	Record identifier and timestamp
Marginal cost	Low per query, high per reindex	Depends on the system queried

The aggregates row explains half the failures. A question like how many contracts expire this quarter isn't answered by retrieving ten passages of text, because retrieval returns what resembles the question rather than the complete population that satisfies it.

How semantic retrieval actually works

Material is split into chunks, each chunk is turned into a numeric vector, and the question gets a vector of its own. The system returns the nearest chunks, almost always a fixed number, and passes them to the model as context.

The operational limits follow from that mechanism, and knowing them avoids promising behaviour the channel cannot deliver.

Chunk size decides what can be retrieved at all. Small chunks give precision and lose surrounding context, large chunks bring noise and dilute similarity. A contract clause referring to a definition twenty pages earlier retrieves badly in both settings, unless document structure is preserved in the metadata.

The fixed number of returned chunks introduces a less obvious effect. If the correct answer needs seven sources and the system brings five, the model answers anyway, with the five it has, without flagging that the picture is partial.

Near-identical documents are the third critical point. Four versions of the same procedure, differing in three paragraphs out of forty, have nearly overlapping vectors, and retrieval returns the wrong one with the same confidence as the right one. The fix isn't in the model, it is in version metadata and the filter that uses it.

The connector and its three properties

A connector turns a request into a query against the system that holds the data: an ERP, a CRM, a document repository, a service exposed by IT.

Three properties make it right for operational data. The result is current, because it is read at the moment of asking. Permissions come from the source system, so a person sees what they would see through its own interface. The result is identifiable, because it carries the record key and the time of reading.

One design consequence belongs in the budget. A connector requires someone to define which queries are allowed, with which parameters and which limits, otherwise an assistant can generate requests that put a production system under strain. That definition is a technical contract, and it gets versioned like everything else.

Which channel for which information

Type of information	Channel	Note
Procedures, policies, manuals	Semantic retrieval	With version metadata and validity dates
Contracts and tender documents	Semantic retrieval	With a filter by counterparty and period
Balance, stock, availability	Connector	Changes during the day
Status of a case or an order	Connector	The data sits in the system that governs it
Deadlines and contract dates	Connector on structured data	If they live only in PDFs, extract them first
Counts, totals, complete lists	Connector	Semantic retrieval doesn't enumerate
Decision history and minutes	Semantic retrieval	With a precise document reference
Master data and identifiers	Connector	Single source, or versions multiply

In that table the deadlines row holds the most common problem. Much information that looks structured lives only inside documents, and until it is extracted into a queryable system no channel returns it reliably.

Where semantic retrieval gets permissions wrong

An index built over every company document, open to anyone, exposes to anyone whatever the filter fails to exclude.

Three arrangements exist, at different costs. Filtering before the search, narrowing the space to documents that user may see, is the correct behaviour and requires permissions in the metadata of every chunk. Filtering after the search, discarding disallowed results, is simpler and produces worse answers, because the system has already spent its available slots on material it then throws away. Separate indexes per profile are the safest route and the most expensive to maintain.

Whichever arrangement is chosen, permissions have to be refreshed when they change in the source systems. An index holding six-month-old authorisations is a security problem nobody sees, because it only surfaces when somebody asks the right question.

Freshness, versions and invalidation

Every indexed chunk needs three pieces of information attached: which document it came from, in which version, and until when that version is valid.

Without the third, the system cites superseded procedures with the same confidence as current ones. With it, the filter can exclude expired material and, where needed, state that the version in force isn't present in the index.

Reindexing cadence depends on the material. Policies change a few times a year, price lists and operating procedures far more often, and an assistant answering across both worlds at the same refresh rhythm is systematically wrong about one of them.

Citation belongs among the requirements

Every answer resting on a document has to carry the precise reference, with document, version and section. Every answer resting on a system has to carry the record identifier and the time of reading.

This serves three purposes. It lets the reader verify without redoing the search. It makes it possible to reconstruct an answer months later, once the document has changed. It gives oversight a way to measure how often the system cites expired or out-of-scope material.

An assistant that doesn't cite is unusable wherever somebody answers for what they did on the strength of that answer.

The architecture that combines both channels

In systems that work, semantic retrieval locates the context and the connector brings the data, with rules deciding what to do with the result.

A request about a case arrives, retrieval finds the applicable procedure in its current version, the connector reads that case's current status, the rule compares the two and produces either the outcome or an exception. The model interprets the question and composes the answer, without being the source of either element.

The separation also makes oversight workable, because an error is attributed to one of three components rather than to an undifferentiated system.

The errors that recur

The first is indexing operational data. Exporting an ERP to PDF and dropping it into an index produces an assistant that answers with old numbers and doesn't know it.

The second is treating the number of indexed documents as a quality measure. An index of a hundred thousand unfiltered documents answers worse than one of two thousand curated, versioned documents with correct permissions.

The third is not measuring retrieval separately from the answer. When an assistant gets something wrong, the cause usually sits in the retrieved chunks, and without a measure of that stage you end up changing model while the problem is elsewhere.

The fourth is exposing a connector without limits. Queries with no volume or time-range constraints can degrade a production system, and the remedy belongs in the design rather than in the incident report.

Where to start

The distinction between components that interpret and components that decide is in probabilistic and deterministic.

Data preparation and quality are covered in data readiness and data governance for AI, while the role of knowledge structures is in ontologies as infrastructure.

To set the architecture of your assistant and decide what belongs on an index and what on a connector you can book an assessment session.