Eight questions to ask any agent vendor, including us

I wrote these after sitting through enough vendor meetings where the buyer asked about accuracy and price and never about the things that decide whether an agent is still running a year later. Each question has the answer we give for our own platform. Two of them are answers we are not proud of. Use the questions on anyone.

1. 01

   When does the agent not call a model, and how do I see that in the log?

   Ours

   Retrieval runs without a model. The model is called only for the steps that need judgment, and the log records each call, which step it belonged to and what it was given. In one of our invoice and filings pipelines, six steps became two model calls. Validation, lookup, normalisation and arithmetic are code.

2. 02

   Can every answer show the page it came from?

   Ours

   Yes. A citation check runs in code, against the passages that were actually supplied, before the answer is shown. On a production run that check cost under half a cent for a full answer.

3. 03

   What is written to the log, and when?

   Ours

   Who asked, what was retrieved, which model was called if any, what it cost, and the answer. It is written before the answer is displayed, so there is no answer without a record. The log sits in the customer’s own event store, in their tenant, so it is theirs to query.

4. 04

   What does one question cost, and is that known before it runs?

   Ours

   The budget is fixed before the run, as a cap per step and a cap on model calls. A retrieval-only question uses no model tokens.

   The honest part: one of our searches once spent $0.95 across 72 attempts and returned nothing, because a column was misnamed. Renaming the column fixed the next run to 7 of 7. The spend was bounded, which is the only reason we noticed on the same day and not on the invoice.

5. 05

   Where does our data go?

   Ours

   It stays with you. The execution cluster, the event store, the knowledge graph and the vector index are installed in your tenant, and the model is the one you bring. param does not see your documents or your questions.

6. 06

   Which model is it tied to, and what happens when we want to change it?

   Ours

   No single model. The model is chosen per step and can be changed without rebuilding the agent.

7. 07

   What happens when the agent does not know?

   Ours

   It says so, shows what it searched, and says what document would answer the question.

   One run, as it happened. Asked who had invested in a company and how much, the agent reported that the documents held no investor, shareholder or equity records. It listed the capital spending figures it did find and said plainly that those are not outside investment. It flagged a summary line that had expanded “TSR” as shareholder return when the rest of the corpus uses it for thermal substitution rate, and declined to treat it as evidence. It pointed to the CFO memo as the place the records would be. Nothing was invented to fill the gap.

8. 08

   Can you explain every field in your data model to the person who signs?

   Ours

   Not always, and this is the one we are least proud of. A field named “Domain” that nobody on our side could define sat in the model until a customer asked what it meant. It was the website. We renamed it, and the question became a rule: if a field cannot be explained to the person who signs, it does not ship.