Test records

Agent evals

To test a template, we give it to a coding agent in a new session with no CodePlate context, along with a made-up user's brief and about five customization requests, sent one at a time. After each request the template's checks have to pass and the result has to look right. When the agent got stuck, we changed the template and left the eval as it was.

An eval is a scripted scenario that we run ourselves, at most once per template version. So far 15 of the 18 templates have had a run, one each. It is not a benchmark or an independent review, and one run can't tell you what your agent will do with the same requests. Each log is the record as we wrote it, including requests that only partly passed and the template fixes the run caused. We removed local file paths and internal references.