What OpenAI, Anthropic, and Meta learned after putting data agents to work
A valid query can still produce the wrong business answer. Each company built a different context system to reduce that risk.
I read the internal data-agent write-ups from OpenAI, Anthropic, and Meta’s analytics team side by side.
One problem runs through all three: the agent has to know which table matters, what the metric means, whether the result makes sense, and what evidence a human needs before acting.
Note: Every figure below is a self-reported internal result. The companies used different users, questions, and evaluation methods. Treat the figures as clues, not a leaderboard. Each system combines several kinds of context. The labels above show the clearest difference in emphasis.
Same problem, different starting point
OpenAI reads the code
OpenAI’s data platform serves more than 3,500 internal users and spans more than 600 petabytes across 70,000 datasets. Its data agent combines six context layers: table usage, human annotations, code enrichment, institutional knowledge, memory, and live warehouse inspection. OpenAI’s write-up explains each one.
Code enrichment is the interesting part. Codex reads the code that produces a table and recovers assumptions, freshness logic, and business intent that may never appear in the schema.
The goal is to answer two very analyst questions: “what’s in here” and “when can I use it?”
Anthropic governs the meaning
Anthropic starts with a semantic layer owned by humans, then uses skills to steer Claude toward the right source and analysis process.
In Anthropic’s internal evaluations, accuracy did not exceed 21% without skills. With skills, aggregate accuracy stayed above 95%.
Then the documents went stale.
Offline accuracy fell from roughly 95% to roughly 65% over a month. Anthropic now updates the skill document in the same pull request that changes the data model.
The definitions have to ship with the data.
Meta starts with the analyst
Meta’s warehouse has millions of tables, but an individual analyst usually works within a few dozen.
Its agent uses personal query history and offline summaries to bound the working domain. Shared knowledge comes through Cookbooks, Recipes, and Ingredients, including semantic models, documentation, snippets, and memories.
Meta’s starting question is personal: what does this analyst already work on?
Old SQL can point. It cannot decide.
Meta found that 88% of data-scientist queries used only tables that person had queried in the preceding 90 days.
Recent history can narrow the search space.
Anthropic tested something else. It gave Claude raw access to thousands of old dashboard, transformation, and notebook queries. Accuracy moved by less than one percentage point. For about 80% of the missed questions, the answer was somewhere in that corpus.
The findings answer different questions.
Meta measured how well recent table use describes an analyst’s working set. Anthropic measured whether a large raw query archive improved answer accuracy.
OpenAI uses query history as one of six context layers, then combines it with code and human annotations.
These are separate internal findings, not a benchmark comparison.
Old SQL can show the neighborhood. It cannot choose the official metric. Someone still has to turn repeated patterns into approved definitions and trusted sources.
The shared loop
My synthesis of the three systems is simple:
QUESTION → CONTEXT → QUERY → EVIDENCE → CORRECTION
OpenAI and Meta describe agents that run SQL, inspect the result, and adjust. Anthropic puts more weight on governed sources and verification before the answer reaches a user.
All three keep the evidence close:
Meta places SQL front and center with each data point.
OpenAI summarizes assumptions and execution steps, then links the underlying results.
Anthropic shows the source tier, freshness, and owner in a provenance footer.
They also preserve corrections.
OpenAI saves memories. Meta adds reusable team knowledge through Cookbooks. Anthropic turns stakeholder corrections into documentation changes and evaluation cases.
A correction should change the next run.
A useful place to start
Pick one question that returns every week: “Why did conversion drop?”
Give the agent four things:
an approved metric definition
a trusted source and query path
the evidence required beside the answer
a place to save the correction
Run it again next week. If someone repeats the same filter, caveat, or source choice, the system did not learn.
This is the design I am building into ChatData: approved metric context, proof receipts, review state, and reusable answer paths.
A trusted answer should make the next answer faster to produce and easier to defend.
AI can do the analysis. You own the answer.