Complaint triage that sends the uncertain cases to a person.

The question

Every complaint a bank receives has to be read, matched to the product and the rules it concerns, and sent to the right place. Can a system do that for the clear cases, show its evidence, and hand the unclear ones to a person? This project answers with the CFPB’s public database of US consumer complaints, sorted into ten product categories.

Cleaning before modelling

The database holds 13,906,793 complaints; 3,740,313 include the consumer’s own description. A third of the descriptions were exact duplicates: resubmissions and template complaints from law firms. Left in, the same text could land in both training and test and inflate every score, so duplicates were removed before splitting, leaving 2,474,632. The split is by time: training on complaints from before 2025, testing on 274,065 from mid-2025 onwards.

The categories are badly imbalanced. Credit reporting is 52.7% of the training data; debt management is 0.1%, just 1,846 complaints.

A baseline that looked better than it was

A logistic-regression baseline on TF-IDF features scored a macro-F1 of 0.754 on a sample of 100,000 complaints. Trained and tested on the full data, it scored 0.689. The sample had made it look competitive, so every later model is compared against 0.689.

Models

• Two transformers on one laptop GPU. RoBERTa and ModernBERT were fine-tuned on an 8 GB card. Tokenising the data once, up front, cut the estimated training time from 97 hours to 27.
• One change at a time. An ablation tested each training setting on its own; only a longer input of 384 tokens helped, and it was kept for both models.
• Distillation. A 7-billion-parameter teacher collapsed to a single class for most inputs, so a smaller model built for zero-shot classification, BART-large-MNLI, labelled 25,000 complaints with soft probabilities. Trained on the full data with those labels, RoBERTa rose from 0.730 (standard fine-tuning on 500,000 complaints) to 0.750.
• A stack. Logistic regression over both models’ probabilities reached 0.757.

Automatic, or a person

Two confidence checks come first: if the first model is less than 65% sure, or the combined model less than 55%, the complaint goes to a person. Above them, a Thompson-sampling bandit, trained on the outcomes of 1.8 million real complaints, picks which model’s answer to use, or a person. In offline tests it sent 8% of complaints to a person. Its average reward, 0.875, matched always using the stacked model, 0.876: its value is in choosing which cases to hand over, not in better answers.

The live app shows every step: the product, the confidence, the routing reason, and the CFPB examination procedures that apply, found with BM25 among 2,689 passages from 12 procedure documents.

Results

On 274,065 held-out complaints, the stacked model reaches a macro-F1 of 0.757, against 0.689 for the baseline. 13.8% of complaints are misclassified, and the errors are systematic: three pairs of categories account for 39.5% of them, led by debt-collection complaints labelled as credit reporting, because consumers describe the damage to their credit report. Debt management stays the hardest category, at about 35% F1.

Limits

• Some wrong answers come with very high confidence, above 0.99, so confidence alone cannot catch every error.
• The model cannot say a complaint is outside its ten products. An off-topic message still gets a product, sometimes with high confidence, and can be handled automatically.
• The bandit learned from past outcomes and a yes-or-no reward; it would need retraining as complaints change.
• Retrieval is keyword-based, to keep the free server’s start-up short, so it misses passages that use different words.
• The free server sleeps when idle; the app shows stored results while it wakes.