AI Chat Starter: the eval log

The made-up user

Written the way a buyer would put it. {ASSETS}, {ART} and {GODOT} stand for the folder with their files and the engine's path; their asset pack isn't published.

The user runs a small vet clinic and builds with an agent. They know what a chat assistant is but don't read code. They have no AI API key yet (the eval runs in demo mode, with the fake model; nothing is spent).

• Product: Pawpal, a pet-care Q&A assistant for Green Hill Vet Clinic. Pet owners ask questions; Pawpal answers from the clinic's care sheets, with sources.
• Documents: the clinic's care sheets (Markdown) in assets/care-sheets/, and a vaccination schedule they also have as an HTML page.
• Voice: warm, plain, short. Never diagnoses; for anything urgent it tells people to call the clinic.
• Business: free for clinic clients (small allowance), and a paid "Plus" plan for breeders and shelters.
• Brand: warm orange and cream, a paw-print logo (they'd like one drawn).

First message

Hi! I want to turn this template into "Pawpal", a pet-care assistant for our vet clinic (Green Hill Vet Clinic). It should answer pet owners from our care sheets, which are in {ASSETS}/care-sheets. Can you take a look and get us started?

Interview answers

• Purpose: pet owners' questions about feeding, vaccines, grooming and when to call the vet. Only from the care sheets; say so when the sheets don't cover it.
• Voice: friendly, plain words, at most a few short paragraphs. Never diagnose. Urgent signs (not breathing, bleeding, poisoning) → "Call the clinic now: 555-0142".
• Provider: Claude is fine (they'll get a key later). Keep demo mode working.
• Assistants: just one, "Pawpal" (documents). Drop General.
• Plans: Free (20,000 tokens a month) and Plus ($15 a month, 1,000,000 tokens). Document limits can stay as they are. No Team.
• Brand: orange #c2410c and cream #fff7ed, keep dark mode. A paw-print mark.
• Hosting: later.

The requests

1. Make it Pawpal: our name, a paw-print logo, warm orange and cream, our words on the landing page, and one assistant called "Pawpal" that only answers from our care sheets.
2. Write Pawpal's instructions: friendly and short, never diagnose, and for urgent signs tell people to call the clinic at 555-0142. Add a few checks we can run later with a real key to make sure it behaves.
3. Our vaccination schedule is an HTML page ({ASSETS}/vaccines.html). Let us upload HTML files too, and make sure answers about vaccines cite it.
4. Plans: Free gets 20,000 tokens a month, Plus is $15 a month with 1,000,000 tokens. No Team plan.
5. Add a tool so Pawpal can tell owners the clinic's opening hours and whether we're open right now (Mon–Sat 8:00–19:00, closed Sundays, clinic time zone America/Chicago).

The run record

As we wrote it on 2026-10-14, with local paths and internal references taken out. "I" is the person running the eval.

• Template: ai-chat-starter 1.0.0, fresh from the packed zip, with the Green Hill care sheets and vaccines.html from assets/.
• Agent: a Claude Sonnet 5.5 subagent with no CodePlate context, on Windows (Git Bash + PowerShell), Playwright through Edge, and no real API key (fake provider only). User turns were relayed one at a time from concept.md and requests.md.
• Result: interview + 5/5 requests passed. I re-ran npm run check independently at the end: all seven steps pass. Live behaviour with a real model is still unverified, by design.

Where the template fell short (fixed before merge)

1. No shared knowledge base. "Answer from our documents" is the main use case. Documents now have a scope set in config:
   • user: per-user uploads.
   • shared: an admin-managed library plus a knowledge/ folder indexed at startup.
   • both.

The sample uses both.

2. The fake embedder was too weak for demo content without tuning. It's now a better offline scorer, and minScore tuning is documented in the RAG recipe.
3. eval:live hit the rate limit. Eval runs no longer count against the app's rate limit. --fake now validates the case file and plumbing and marks behaviour assertions as skipped rather than failed.
4. The brief now says to run the full npm run check after every change.
5. HTML is a built-in document type.
6. Billing layout follows the plan count. Recipes refer to config/plans.ts instead of quoting numbers. The plan chosen on the pricing page carries through sign-up to checkout.
7. Markdown lists in the document view render as lists.

← AI Chat Starter · Every eval log