Building cellar a support agent that checks its own work

I built an AI customer support agent for a wine and spirits store called cellar. It can look up orders, cancel them, give refunds up to a limit, check whether something is in stock, and hand a conversation to a real person when it should. It runs entirely on my own computer, on two graphics cards and local models, and there is a live demo where you can sign in as a customer and try to break it.

I wanted a project that covered what AI engineer roles actually ask for: building agents, choosing between prompting, retrieval and fine-tuning, designing for tool use, building systems that improve with use, and shipping it into a real product. So instead of five small demos, I put all of it into one thing and measured every part of it.

How it works

The agent is built with LangGraph, with LlamaIndex for searching the store's policies and products, and Qwen models served by Ollama. I used a customer support dataset from Kaggle with 27,000 real support messages, and real product and sales data from Montgomery County's open data portal, so the agent talks about actual products with actual sales history. The Kaggle copy of the sales data only covered 10 months, which was too short for the demand forecast, so I pulled the full 24 months from the county directly.

Every message goes through the same steps. It masks card numbers before any model sees them and blocks prompt injection, works out what the customer is asking for, looks up the store's policies, takes actions with tools that are locked to the signed-in customer, and then checks its own reply before the customer sees it. The rule I kept coming back to was: code where an action must happen, and the model where judgment helps.

What I found

My first perfect score was a lie

I compared three ways of teaching the agent to understand customers, and one of them scored 100% on my test set. Instead of celebrating, I dug into it and found that the test questions were almost copies of the training questions. The dataset has several versions of each sentence (lowercase, no punctuation, typos), and my random split had put versions of the same sentence on both sides. After rebuilding the test so each sentence family stays on one side, and removing 3,615 training rows that were nearly identical to a test question, the honest score was 97.4%. That taught me to be the most suspicious of the results I like the most.

The agent said it cancelled orders it never touched

While testing, I caught the agent telling a customer their order was cancelled when it had not called the cancel tool at all. That is the worst thing a support agent can do, because the customer believes it and still gets charged. Adding more instructions to the prompt did not fix it, so I built a check that compares everything the reply claims with what actually happened in the database. If they don't match, the agent gets one more try, and if it still can't do it, the customer is handed to a person and told honestly that nothing changed yet. One detail surprised me: the model ignored my retry note when it was a system message, but followed it reliably when it was sent as a user message with only the claimed tool available.

Fine-tuning beat prompting the same model

I fine-tuned a small 3B model with QLoRA on my RTX 3060, which took about 12 minutes, and compared it with prompting the exact same model. Prompting got 80.7% of customer intents right. The fine-tuned version got 98.1%, and it was faster too, around 50ms instead of 127ms, because it no longer needs a long prompt listing every intent. Getting it into production was its own lesson: importing the model into Ollama directly produced a model that only answered "????", even though the same weights worked perfectly in PyTorch. Converting it with llama.cpp first fixed it.

Where a model runs changes its answers

A number in my results didn't match the day before, so I ran it again three times. The fine-tuned model scored 98.1% when it ran on the GPU and 99.6% when Ollama pushed it onto the CPU because the GPU was full. Both were exactly repeatable. The CPU and GPU round numbers slightly differently, and that is enough to flip a few borderline answers. I report the GPU numbers, since that is how the demo actually runs.

My own questions found a gap

I wrote 40 product questions myself, the way a real customer types, with slang, typos, budgets and occasions. The fine-tuned model got 97.5% of them, retrieval got 95%, and prompting got 85%. The interesting part was the mistakes. Almost all of them were questions about buying in bulk, like "need 2 cases of champagne for a company party, can you do that?", which are about placing an order as much as asking about a product. The test didn't just find model errors, it found a place where my labels themselves were unclear.

The agent copied an example from its own instructions

When a customer asked about "my last order" without a number, the agent looked up order ORD-12345, which doesn't exist, and then told the customer the number they gave was wrong. It had copied the example order number from its own tool description. I removed the example and made it look up the customer's recent orders first.

The live demo cancelled the wrong order

After deploying, I tested the live site as a customer with several open orders and asked it to cancel my latest one. It cancelled the second newest. The tool returned orders newest first but never said so, and the model misread the list. Now the tool names the most recent order directly, and the confirmation card shows when the order was placed, so a wrong pick is visible before anything happens. My test scenarios had missed this case, so I added it to them.

What I learnt

The biggest lesson was to measure everything before I believe it. Every number on the project comes from a test that checks the store's database, not from reading a reply and deciding it looks right, and that is how I found most of the problems above. The second lesson was that the important rules belong in code. Refund limits, the 21+ age rule, asking before acting and checking claims all live in code that the model cannot talk its way around. The third was that real use finds what tests miss, so a project is not done when the tests pass, it is done when real people have tried it and what they found is fixed.

I also built this with Claude Code as my pair programmer, and that taught me something too. It made me fast, but the useful skill was directing it, making the technical calls, and checking its work instead of just accepting it. Several of the findings above started with me looking at a result and asking whether it was actually true.

Where it stands

The agent passes 79 of 81 test conversations, including multi-turn, adversarial and wine-knowledge cases, and makes the right tool call 98.8% of the time. The two failures are written up in the README: a message with two requests in it, and a reply that finds the right order but doesn't name it. Next I want to handle messages with more than one request and sort out the bulk-buying labels.

You can try the demo while my machine is on, and the code, evals and full write-up are on GitHub.