CodePlate home

Agent evals · Email Template Pack

Email Template Pack: the eval log

We gave Email Template Pack to a coding agent in a new session with no CodePlate context, along with a made-up user's brief ("Orbit Lane Bikes"), then sent the requests below one at a time.

2 of 5 requests passed (3 partial)

Run
2026-10-15
Agent
Claude Sonnet 5.5
The user
The script, relayed turn by turn
Version tested
v1.0.2
On sale now
v1.1.0

This run tested v1.0.2. The version on sale is v1.1.0, which includes the fixes listed at the end of the record and has not had its own eval run.

The made-up user

Written the way a buyer would put it. {ASSETS}, {ART} and {GODOT} stand for the folder with their files and the engine's path; their asset pack isn't published.

The user runs a bike repair workshop with online booking and a small parts shop. She builds with an agent and doesn't read code. Node and a browser (Edge or Chrome) are installed on the eval machine; no sending-service keys are given, and nothing may be sent.

First message

Hi! We're a bike repair workshop and we want our emails to look like us instead of the plain ones our booking app sends now. Our logo, the newsletter picture and my notes are in {ASSETS}. Can you set this up?

Interview answers

The requests

  1. Make them ours: our logo, yellow and charcoal, our address and sign-off, and take out the emails we don't need.
  2. Our repair receipts have a service line, parts, sometimes a discount, then VAT at 23%. Show the bike it was for and the booking date and time. Can you make the receipt work like that?
  3. Add a booking reminder for the day before: when and where, what to bring (the bike, the lock key, any parts you bought), and a button to reschedule.
  4. Spokes, our newsletter: use our header picture, three short stories and a "tip of the month", and make sure people can unsubscribe. Then give me the version to paste into Mailchimp.
  5. Our developer asks what exactly our app should send to Postmark for the booking reminder. Show me, without sending anything.

The run record

As we wrote it on 2026-10-15, with local paths and internal references taken out. "I" is the person running the eval.

#RequestTurnsResultNotes
0Interview2passFour questions with defaults. Before changing anything it noticed that yellow on white is too pale for text and proposed charcoal buttons, darker gold links, and the logo on a light tile in dark mode. After the answers it set the sender, kept six emails, removed four, set the colours, drew the logo PNG, set the sign-off, ran the checks and looked at the contact sheets. docs/project.md records every decision.
1Make them ours1passAlready done in the setup turn; it re-checked it against the files. brand.json has the name, site, assetBaseUrl https://orbitlane.example/mail/, hello@orbitlane.example, the Dublin address, en-IE and Europe/Dublin. Light mode uses charcoal buttons with white text and #856b00 for links and eyebrows; dark mode uses the yellow. The charcoal wheel sits on a white tile and shows in dark.png. trial-ending, team-invite, magic-link and product-update are gone with their picture; the sign-off reads "Niamh and the workshop crew". The owner's logo and header picture are listed as CodePlate-original, and the CodePlate parcel picture is credited to Orbit Lane Bikes (the assets check allows no licence for the user's own files; see below).
2Repair receipt1partialBike and booking rows, an optional discount row ({{#if receipt.discount}}), a VAT 23% line, all from minor units with {{money}}; the app works out the totals. The example data checks out: €97.00 − €9.70 = €87.30, VAT €20.08, total €107.38. The plain text keeps the columns, and the phone screenshot doesn't overflow. There was no style for a long date with a time, so it added fulldatetime to src/render.mjs. It opened the light PNG only, not dark or mobile. "Total paid (incl. VAT)" is fixed text, so a shop order without VAT would still say it.
3Booking reminder1partialA new transactional email: subject "Reminder: your bike is booked in tomorrow", preheader with the full date, when, where, bike and service, what to bring, and a "Change my booking" button with a real-looking link. Props file complete, no TODO left, same header, footer and buttons as the others. The workshop address is typed into the email instead of using brand.address, the list is lines with <br> instead of the recipe's list markup, and the HTML has no fallback link (the plain text prints it). It opened light and dark, not mobile.
4Spokes1partialThe header picture (1200×500, alt text, no fixed height), three stories with links, a "tip of the month" panel, still marketing, unsubscribe and preferences links and the postal address in the HTML and the plain text, 31 KB. esp/newsletter.json uses `*
5Postmark payload1passRan the dry run: POST https://api.postmarkapp.com/email with From, To, Subject, HtmlBody, TextBody, MessageStream: "outbound", the token shown as a placeholder. Showed the renderEmail('booking-reminder', data) call the app makes with the real booking and said the token comes from the environment. Said nothing was sent; I re-ran the dry run and it printed the same request.

Where the template fell short (fixed in email-template-pack 1.1.0)

  1. No date-and-time style that names the day. {{date x "datetime"}} gives "3 Nov 2026, 09:30". A booking needs "Tuesday 3 November 2026 at 09:30", and the agent edited src/render.mjs to get it. The date helper now has fulldatetime (the line the agent wrote) and time ("09:30"). The architecture notes and the tables-and-money recipe list them, and tests cover en-IE in winter and summer time, en-US, and the fallback for an unknown style.
  2. npm run new-email wrote the sample company's data. The new props file got the sample customer and pebblework.example links, and the agent edited src/cli.mjs to change them. The props file now comes from brand.json: customer@<their domain>, links on their own site, and TODO for the first name. A test with another brand finds no trace of the sample company.
  3. No brand-wide sign-off. The sign-off partial fell back to "The {{brand.name}} team", so the agent edited the partial. brand.json now takes optional signOff and sender, checked as text; an email's own signOff or sender still wins. The rebrand recipe mentions both.
  4. No licence value for the user's own pictures. The assets check accepted only CodePlate-original, CC0-1.0 or OFL-1.1 for shipped files, so the owner's logo and header went in as CodePlate-original, and nothing noticed the CodePlate parcel credited to the bike shop. Files the user supplies are now user-owned, credited to their owner, and a CodePlate-original file must credit CodePlate, so this project's parcel entry now fails the check. The rebrand and pictures recipes say which value to use. The pictures recipe also says a new picture gets its own file name instead of a sample's slot. The same check ships in print-and-play-kit 1.1.0, slide-deck-kit 1.1.5 and typst-documents 1.1.6.
  5. Screenshots got skipped. In two requests the agent opened only the light PNG. npm run preview -- <id> now also writes shots/<id>/all.png, with light, dark and phone side by side and labelled, and prints its path last. The brief's verify step and the add, edit and pictures recipes name that file. A test checks the file exists and is as wide as the three views together.
  6. Retyped brand details. The add-an-email recipe now says to use {{brand.address}}, {{brand.supportEmail}}, {{brand.name}} and {{brand.url}}, and points to the markup for lists and the printed-out link. npm run lint and the checks warn (without failing) when an email's MJML contains the address or support email as typed text.

Not changed: the Mailchimp footer note is already step 5 of the ESP recipe; the agent didn't pass it on. "Total paid (incl. VAT)" was the agent's own wording in this project's receipt, not template text.

What each request was scored against

Every request:

  • npm run check ends with CODEPLATE CHECKS: PASS (tools, assets, emails, unit, build, html, text, inspect, credit, budget), run in full after the change.
  • No check is weakened: no rule removed or loosened in src/lint.mjs, src/inspect.mjs or src/config.mjs, no marketing email turned transactional, no fixed height to force a picture.
  • The agent ran npm run preview and opened the contact sheets and the PNGs of the emails it changed (light, dark and mobile), and said what it saw.
  • docs/project.md is updated (the stub marker is gone after the interview).
  • README.md still carries "Built with CodePlate (codeplate.io)"; any new package is pinned exactly with a permissive licence.
  • Nothing is sent: no --send, no API keys written to files.
  1. Rebrand and trim
    • brand.json: name, https://orbitlane.example, assetBaseUrl https://orbitlane.example/mail/, the support address, the Dublin address, locale en-IE, timeZone Europe/Dublin.
    • Logo: the SVG drawn to a PNG (assets/src/ + npm run images, or an equivalent PNG at twice its shown size) and listed in assets.json; the charcoal mark is visible in dark.png (a backing, a light outline or a light variant), because a transparent charcoal mark vanishes on the dark page.
    • Colours: the agent noticed (from npm run contrast, the checks or by itself) that yellow on white is 1.64:1 and did not use it for text, links or eyebrows on the light card (a darker shade such as #8e7200, or charcoal); buttons are yellow with charcoal text or charcoal with white text; the dark set uses the yellow where it passes. It told her why.
    • trial-ending, team-invite, magic-link, product-update removed from emails.json and emails/, with product-update.png/.svg and their assets.json entries (and any other picture no remaining email uses); the sign-off "Niamh and the workshop crew".
  2. Repair receipt
    • The receipt shows the service and parts lines, an optional discount row ({{#if}}), VAT 23% and the total, all from the data in minor units with {{money … "EUR"}} (no maths in the template); the bike and the booking date and time come from the data ({{date … "datetime"}} or similar) and show in light, dark and on a phone without the table overflowing 375 px.
    • receipt.props.json has realistic example data, including a discount; the plain text shows the items as columns.
  3. Booking reminder
    • A new transactional email (npm run new-email or by hand) in emails.json with a subject and preheader that name the day, no TODO left; the date and time, the workshop address, a list of what to bring, a "Reschedule" button with a real-looking link and the link printed (or a fallback link); a props file with every value.
    • Same header, footer and button style as the others in the contact sheets.
  4. Spokes
    • newsletter-header.png in assets/images/ and assets.json at 1200x500, used with alt text and no fixed height; the old newsletter-hero files removed if unused.
    • Three stories and a tip; the email stays marketing with the unsubscribe link and postal address in the footer (and in the plain text); under 102 KB.
    • A Mailchimp version made with merge tags (e.g. npm run render -- newsletter --data esp/newsletter.json with *|UNSUB|*, *|EMAIL|*), handed over as a file path with what to paste where; the agent noted that Mailchimp adds its own footer and tracking and the size should be checked there.
  5. Postmark
    • npm run send -- postmark <reminder-id> (dry run) or equivalent code showing the request: POST https://api.postmarkapp.com/email, From, To, Subject, HtmlBody, TextBody, MessageStream: outbound; the agent explained that the app calls renderEmail(id, data) with the real booking data and the server token from the environment; nothing was sent.

← Email Template Pack · Every eval log