Structured Data Gateways vs. Web Retrieval for AI Agents: What Our Study Found

Summary

We gave AI agents the same 48 procurement questions two ways: through a structured data gateway, and through web retrieval. Same facts, same only the agent data routing changed.

• *
  Accuracy: the gateway scored 100%. The web path scored 46% for a hand-built scraper and 51% for a real LLM (DeepSeek V4.1 Flash, 192 episodes).
• *
  Cost: web retrieval cost 7.2× (scraper) to 23× (LLM) more per correct answer.
• *
  Speed and tokens: the LLM on the web burned 10× the input tokens and took 12× longer.
• *
  Security: hidden prompt-injection text on just 3 of 69 web pages misled the agents. The gateway was never exposed.

The takeaway: for high-stakes AI agents, the data interface matters as much as the model. It's an early, controlled and we're building a bigger simulated benchmark next.

Why we ran this study

AI agents have stopped just answering questions — they compare quotes, shortlist suppliers and place orders. All of that hangs on one unglamorous step: getting the right data. Right now agents get it one of two ways:

• *
  Web search, fetch HTML, read pages written for humans.
• *
  Structured data LLM tool calling against typed APIs or MCP servers (Model Context Protocol) that return schema-validated records with provenance.

We picked procurement because small errors get expensive fast. Misread a GST-inclusive price as exclusive and every quote is 15% off. Trust a six-month-old price list and you overpay.

We couldn't find a study that held the facts fixed and changed only the interface. So we built one: a synthetic world of 36 New Zealand suppliers and 87 offers, rendered two ways — a gateway with four typed tools, and a 69-page simulated web carrying real-world mess like unlabelled GST, stale PDF price lists, out-of-date directories and hidden injection text.

What we found

Swapping web retrieval for a gateway took an LLM agent from 51% to 100% accuracy. Every one of the 47 disagreements between the two paths went the gateway's way (McNemar p = 1.4×10⁻¹⁴).

Hand-built scraper

• *
  Gateway: 100% accuracy, 0% no answer, 2,514 input tokens per query, $0.0034 per correct answer, 0.1s median latency, misled 0 of 10 times
• *
  Web: 46% accuracy, 17% no answer, 13,134 input tokens per query, $0.0244 per correct answer, 12.7s median latency, misled 2 of 10 times

LLM agent (DeepSeek V4.1 Flash)

• *
  Gateway: 100% accuracy, 0% no answer, 7,580 input tokens per query, $0.0060 per correct answer, 4.6s median latency, misled 0 of 20 times
• *
  Web: 51% accuracy, 33% no answer, 76,763 input tokens per query, $0.1379 per correct answer, 55.3s median latency, misled 3 of 20 times

Scraper = a hand-written “competent scraper” vs API client. LLM = DeepSeek V4.1 Flash, 48 queries × 2 runs per arm. Scraper latency is modelled tool time only.

1. Structured data for AI agents removes the guesswork

Once the data came price basis, as-of date, vendor ID the LLM made zero data errors. The gateway won biggest exactly where the web needs interpretation: fresh prices, GST and stock levels.

2. LLMs read the web better than scrapers but not well enough

The LLM beat the scraper on fresh prices (75% vs 0%) and on cheapest totals (63% vs 38%), because it read GST labels and validity dates in context. It was worse on filter-set questions (63% vs 88%). It never came close to the gateway.

3. Web scraping vs API for AI agents: five ways the web path broke

1. Unpublished data. All 16 stock-level episodes came back empty, because websites only say “In stock.”
2. Unlabelled GST. The agent reported $10.61 ex-GST with “high” confidence. The real figure was $9.23.
3. Stale sources agreeing with each other. Two out-of-date pages made a wrong price look double-checked ($25.63 vs the current $28.09).
4. Missing identity data. Māori-owned supplier questions scored just 25%, because genuinely Māori-owned suppliers didn't say so on their sites.
5. Runaway exploration. 26 of 96 web episodes hit the 14-step limit, some guessing URLs that didn't exist.

4. AI agent cost per correct answer is the real metric

Per query, the web looks only ~3× pricier. Per correct answer, it's 7–23×. And that's before the cost of being wrong: a 15% GST misread on a $105,000 laptop order is a $15,000 mistake.

5. Prompt injection in AI agents is a live channel

Three hidden paragraphs across 69 pages were enough. One fake page claimed to be Māori-owned, and the scraper recommended it as the cheapest Māori-owned option. The LLM never repeated the planted $1.00 price, but still named an injecting supplier in 3 of 20 exposed runs. The gateway returns typed fields, so there's far less room for instructions to hide.

What we'd do better

The results hold for our controlled world. Here's where we'd push harder next time:

• *
  The gateway's 100% is a ceiling we built in. A correct client of a correct API can't miss. Real gateways carry stale or wrong records too, and we didn't model that. Next time, the gateway gets noisy providers.
• *
  The web was too small and too clean. 69 pages and BM25 search are far simpler than the real web and commercial search engines.
• *
  We designed both sides. Failure rates for GST labelling or stale price lists came from experience, not measurement. They should be calibrated against real supplier sites.
• *
  Only one LLM finished. DeepSeek V4.1 Flash completed both arms. Our Claude Haiku 5.5 web run failed on API errors, so we can't yet say the gap holds across model families.
• *
  Latency is partly an artefact. 92 of 96 web episodes ran through DeepSeek's API vs 21 of 96 gateway episodes, so some of the 12× latency gap may come from the route. Tool latency and data fees were modelled, not measured.
• *
  Small per-type samples. Each query type rests on 16 LLM episodes. The overall gap is significant; the per-type numbers are directional.
• *
  No hybrid arm. The realistic future is gateway-first with a labelled web fallback. We didn't test it.

What's next: a bigger simulated benchmark

This was round one. We're building a larger, more realistic simulated AI agent accuracy benchmark that fixes the gaps above:

• *
  A much bigger, messier simulated web, with stronger search.
• *
  A gateway that can be wrong stale and noisy provider records so 100% stops being a given.
• *
  Several model families, open-weight and proprietary, run on the same route.
• *
  A hybrid arm: gateway first, labelled web fallback.
• *
  Measured latency and real data fees.

We'll publish the results and the code when it's done. Follow along if you want the numbers first.

Studies to follow

• *
  Cross-model replication: Does the gateway gap hold for proprietary models? Why it matters: shows the interface, not the model, drives accuracy.
• *
  Real-supplier validation: Do results hold with real, consenting NZ suppliers? Why it matters: moves the findings from lab to market.
• *
  Gateway data-quality stress test: How fast does accuracy fall as gateway records go stale or wrong? Why it matters: puts a real number on the 100% ceiling.
• *
  Hybrid routing: Does gateway-first plus a labelled web fallback beat either alone? Why it matters: tests the setup most teams will actually ship.
• *
  Prompt-injection red team: How do stronger injections hit web agents vs gateways? Why it matters: turns a 3-page probe into a proper security result.
• *
  Coverage vs accuracy: How many suppliers must connect before a gateway beats the web overall? Why it matters: answers the cold-start question for any gateway.
• *
  Other domains: Does the pattern hold in travel, logistics or healthcare pricing? Why it matters: shows whether this is procurement-specific.
• *
  Inclusive onboarding: Do small, rural and Māori-owned suppliers gain or lose visibility? Why it matters: checks the gateway doesn’t encode exclusion.

FAQ

What is an AI agent data gateway? A service that gives AI agents typed, schema-validated data through tool calls, such as an API or MCP server, instead of making them read web pages. Every record carries its source, as-of date and price basis.

Is web scraping or an API better for AI agents? In our benchmark, the structured gateway answered 100% of procurement queries correctly. Web retrieval managed 46–51%, at 7–23× the cost per correct answer.

What is agent data routing? The choice of where an AI agent gets its facts from: the open web, a structured gateway, or a mix. Our study shows that choice can swing accuracy by around 50 points.

Does an MCP server stop prompt injection? It shrinks the attack surface a lot, because it returns typed fields from known providers rather than free text. It doesn't remove risk: the gateway itself becomes a high-value target.

Why measure cost per correct answer? Cost per query hides wrong answers. Per query, our web path was ~3× pricier; per correct answer it was up to 23×.