Reviews, support tickets, app store comments, and social posts hold the earliest signal about what is wrong with a product. Most organizations read a sample. People triage what they can, priorities follow whoever shouted loudest, and a problem is visible in the data for weeks before anyone connects it. Coverage, not insight, is the constraint.
Classification at volume is a task language models do well and cheaply. The harder and more valuable parts are detecting what is new, and delivering the right evidence to the right team.
The design
- Feedbackreviews, tickets, comments
- Preparecode: dedupe, language, spam
- Classifysmall model, structured output
- Groupcluster new and unmatched
- Summarizeissue brief with examples
- Ownerconfirms and acts
- Issue trackerlink, evidence, status
- Prepare in code. Remove duplicates and spam, detect language, attach product and version metadata. Cheap, deterministic, and it keeps model spend on real content.
- Classify with a small model. Each item gets a topic from your taxonomy, a sentiment, a severity, and the product area, returned as structured output so the result is always valid against the schema. This is the work Anthropic positions its smallest tier for: high-volume, latency-sensitive classification and extraction. On non-sensitive data, batch requests at half price suit a backlog.
- Group what does not fit. Items that match no category confidently, or that cluster around a new term after a release, go to a separate path. A mid-tier model summarizes each new cluster with representative examples. This is how an emerging defect shows up as a brief before it shows up as a pile.
- Route with evidence. A confirmed issue creates a ticket for the owning team with the count, the trend, the affected versions, and representative quotes. Owners should not need to open a dashboard to understand it.
The taxonomy is a product
Classification quality depends more on the taxonomy than on the model. Categories that overlap produce disagreement between people as well as models. Give the taxonomy an owner, a change process, and a version. When categories change, re-run the evaluation set against the new version, and keep the old labels so trends stay comparable.
Where people stay involved
- High-severity items. Safety, legal, and regulatory mentions go to a person at once, with the model's classification as a hint and not a decision.
- Taxonomy changes. A person accepts a new category before it goes live.
- Sampling. A rotating sample of classified items is reviewed every week. The review keeps the evaluation honest after launch.
How to evaluate it
- Stratified labeled sample. Label a sample that includes rare categories and ambiguous items, not only the common ones. A random sample hides weakness on categories that matter most.
- Precision and recall per category. Report by category. Overall accuracy flatters a system that is wrong where it counts.
- Agreement among people. Measure how often two reviewers agree on the same items. The system cannot beat that ceiling in a meaningful way, and a low number points to the taxonomy.
- Detection lead time. On past incidents, how early would the new-issue path have raised it? Replay the history.
- Drift. Track the distribution of categories and confidence over time, and alert on shifts.
The business case is simple to state and should be measured: how much feedback now gets read, how quickly problems reach their owner, and how much analyst time moves from sorting to deciding.
This is a reference design. It describes how we would build the system and cites Anthropic's documentation for platform facts. It is not a client case study, and the figures it uses are list prices or worked examples, not results.
Have a process like this?
Bring it to a 45-minute working session.