Data teams run checks. Pipelines are validated, tables are tested, dashboards are monitored, and event tracking is verified. The checks produce failures, and then people take over: investigate which table broke and why, work out which team owns the source, file the ticket, and follow it up. Meanwhile, verifying that analytics events fire correctly across the client, ingestion, and semantic layers is manual work that scales with the number of user journeys.
This is the area where our team has the most direct experience, and the patterns below are the ones we reach for. Both agents are bounded, evidence-first, and keep a person in the loop until the measured accuracy earns more autonomy.
Agent one: failure triage and routing
- Failed testfrom the results table
- Investigateread-only queries against source tables
- Classify causedata, pipeline, definition, test bug
- Find ownercode: ownership map, open tickets
- Draft ticketevidence, impact, priority
- Reviewerapproves until trust is earned
- Ticket and trailfiled, linked, logged
- Read the results table, not the dashboard. The agent queries test results directly, with the test definition and history, so it sees the same failure an analyst would.
- Investigate with narrow, read-only tools. It queries the source tables to check whether the failure reflects bad data, a late pipeline, a changed definition, or a broken test. Queries are scoped and capped, and the results carry row counts and examples, not full tables.
- Route in code. The owning team comes from an ownership map. A lookup does not need a model. The same step checks for an existing open ticket, so one root cause produces one ticket.
- Draft with evidence. The ticket states what failed, what the agent checked, what it found, the suspected cause, and the downstream impact. The owner can act without re-investigating.
Agent two: analytics test journeys from an instruction
- Instructionplain language
- Plansteps and expected events
- Drive clientbrowser or app, with tools
- Capturenetwork events, payloads
- Validatecode: schema, values, order
- Tracethrough ingestion and semantic layers
- Results and audit tablesevidence for every run
An instruction might read: visit the homepage, scroll to the end, play the first video, and verify that the expected pageview, scroll, and video events fire with the right attributes. The agent plans the journey, drives the client, captures the events, and validates them. Validation of schemas, values, and ordering is code. The agent's role is to run the journey and to read what comes back when something differs. It then follows the data through each backend layer and writes the result with the evidence.
The value is reach. Describing a journey takes a sentence, where scripting and maintaining a test for it takes engineering time. Because every run writes to results and audit tables, a pass is evidence, not just a green check.
The trust ladder
- Advise. The agent drafts. A person files and routes every ticket.
- Approve. The agent proposes a routed ticket, and a person approves with one action.
- Act on the safe set. For failure types where measured routing accuracy is high and the cost of a wrong route is small, the agent files directly. A person reviews a sample.
- Widen. Add types as the evaluation supports it. Pull a type back down if its accuracy slips.
Each rung is a decision with a number behind it, from the evaluation described in the evals article.
How to evaluate it
- Routing accuracy. On closed tickets, did the agent pick the team that ended up fixing it? Code can grade this exactly, by failure type.
- Cause classification. Compare the agent's cause label with the resolver's.
- Duplicate suppression. How often does one root cause yield more than one ticket?
- Test journeys. Seed known tracking defects and measure whether the agent finds each one, and how often it reports a defect that is not there.
- Resolution time. Time from failure to the owner starting work. Compare before and after.
Agent tooling is easier to maintain when it is shared. The Model Context Protocol gives these tools one interface that several agents and applications can use, so the results table and the ticketing system are wrapped once and reused.
This is a reference design. It describes how we would build the system and cites Anthropic's documentation for platform facts. It is not a client case study, and the figures it uses are list prices or worked examples, not results.
Have a process like this?
Bring it to a 45-minute working session.