Discord Bot Starter: the eval log

We gave Discord Bot Starter to a coding agent in a new session with no CodePlate context, along with a made-up user's brief ("Chapterly"), then sent the requests below one at a time.

5 of 5 requests passed

Run: 2026-10-14
Agent: Claude Sonnet 5.5
The user: The script, relayed turn by turn
Version tested: v1.0.0
On sale now: v1.0.3

This run tested v1.0.0. The version on sale is v1.0.3, which includes the fixes listed at the end of the record and has not had its own eval run.

The made-up user

Written the way a buyer would put it. {ASSETS}, {ART} and {GODOT} stand for the folder with their files and the engine's path; their asset pack isn't published.

The user runs an online book club on Discord and builds with an agent. They use Discord every day and manage the server, but they don't read code. They have no Discord application yet (the eval stays offline: the checks and the screenshots are the evidence; nobody creates a real bot or token).

Server: The Long Shelf, a book club with about 120 members in a few time zones; they meet on Europe/London time. One server only.
Bot: Chapterly. It keeps the club reading together: what we're reading, which chapters this week, meeting RSVPs, ratings when we finish a book. Keep polls, reminders, the welcome message and "Save as quote" (they love quoting books). Drop /tag and /purge (they have another moderation bot).
Who does what: members with the Club Host role run the club (set the book, schedule meetings). Everyone else reads, RSVPs, rates and votes. The server owner decides which role counts as host.
Voice: warm and bookish, short messages, at most one book emoji (📚, 📖) per message.
Look: burgundy (#7a1f3d); their avatar is in assets/.
Where it runs: their PC while testing, a small VPS later (not part of this eval).

First message

Hi! I run a book club server on Discord ("The Long Shelf") and I want my own bot for it, called Chapterly. This template looks like a good start. Our avatar is in {ASSETS}. Can you take a look and get us started?

Interview answers

What it does: keeps the club reading together: the current book and this week's chapters, meetings with RSVPs, ratings when we finish. Keep polls, reminders, welcome and "Save as quote". Remove tags and purge.
Server size: about 120 members, one server.
Commands wanted: set the current book and its reading schedule, see this week's chapters, meetings people can RSVP to, rating a finished book. "You pick the names, keep them simple."
Who can use what: the "Club Host" role runs the book and meetings; everyone else uses the rest. The owner should choose the host role in Discord, not in code.
Where it runs: my PC for now, a VPS later. No Discord app yet.
Voice and look: warm, bookish, short; one emoji at most; burgundy #7a1f3d; the avatar from the assets folder. Time zone Europe/London.

The requests

Make it Chapterly: our name, burgundy color and avatar everywhere the bot shows itself (help, about, the status line), and a warm bookish tone in its replies. Take out /tag and /purge completely. Change the default welcome message to welcome people to The Long Shelf and point them to the current book.
Add the current book: a Club Host sets the title, author, how many chapters, how many chapters a week we read, and a start date. Anyone can ask "what are we reading?" and see this week's chapters ("Week 3: chapters 9–12"). Every Monday at 9:00 London time, Chapterly posts this week's chapters in a channel the hosts choose. It has to keep working if the bot restarts.
Meetings: a Club Host schedules a meeting (date, time in London time, and where: a voice channel or a link). Chapterly posts it with Going / Maybe / Can't go buttons that show the counts and who's going. One hour before, it pings everyone who said Going. If a host cancels the meeting, the buttons stop working and the post says it's cancelled.
When we finish a book, members rate it 1 to 5 stars (one rating each, and they can change it) and see the average and how many rated. Keep a history: a command that lists our past books with their average rating, newest first.
Only Club Hosts can set the book, change the schedule and create or cancel meetings. Everyone else can RSVP, rate, read the schedule and use polls and reminders. Let the server owner choose which role is the host role from inside Discord, and until a role is chosen, only people with Manage Server can do host things.

The run record

As we wrote it on 2026-10-14, with local paths and internal references taken out. "I" is the person running the eval.

Template: discord-bot-starter 1.0.0, fresh from the packed zip, with the Chapterly avatar from assets/.
Agent: a Claude Sonnet 5.5 subagent with no CodePlate context, on Windows, with no Discord token (fake Discord only). User turns were relayed one at a time from concept.md and requests.md.
Result: interview + 5/5 requests passed. I re-ran npm run check independently at the end and it passes. Migrations 001–004 are untouched, and 005–008 were added for the changes. Real Discord is still unverified, by design.

#	Request	Turns	Result	Notes
0	Interview	1	pass	Ran the checks first, proposed book-club features, and asked about roles, where the bot runs and Discord setup. Told the user never to paste the token.
1	Rebrand, remove /tag and /purge	1	pass	A new migration drops the tags table, and the invite asks for fewer permissions. Deleted `npm run avatar` because it would overwrite the user's avatar. Looked at screenshots.
2	Current book + Monday post	1	pass	`/book show/set/channel`, plus a 9:00 Europe/London post stored in the DB with catch-up after downtime and clock-change tests. Wrote its own time-zone helper and weekly-repeat logic: the template has neither, and the preview shows `<t:>` timestamps in UTC.
3	Meetings + RSVP	1	pass	Going/Maybe/Can't go with counts and names, a 1 h reminder that survives restarts, ping chunking, and checks for clock-change gaps and past times. Cancelling disables the buttons. The preview doesn't render strikethrough.
4	Ratings + history	1	pass	A select menu with one changeable rating per member, kept anonymous; history newest first, plus `/book finish`. Found a template bug: number options with fixed choices weren't validated against the choices.
5	Host role from Discord	1	pass	`/host set/clear/show` (owner only). Manage Server is the fallback until a role is chosen or after it's deleted, and the owner always counts as a host. `/help` hides commands the user can't run. Reviewed every screenshot and fixed `/remind` footers that showed UTC.

Where the template fell short (fixed before merge)

The avatar generator clobbered a user's avatar. npm run avatar now refuses when assets/avatar.png isn't the template's own, unless forced. The brief says so.
No time zone support. There's now a bot-wide timezone setting plus core/time.ts helpers (zoned dates, clock-change-safe "next 9:00 Monday"). The scheduler supports recurring jobs (every: 'week' at a zoned time). Footers and replies use the setting.
The preview renderer shows <t:…> timestamps in the configured time zone, renders strikethrough, and draws poll bars with a font that has the block characters.
Option choices weren't enforced. Choice validation now covers number and integer options too, with a test.
No reusable role-based access. There's a built-in "manager role" guard: the owner picks the role in Discord, Manage Server is the fallback, and the owner always counts. There's a recipe for it, and /help hides commands the user can't run.

What each request was scored against

Every request:

npm run check ends with CODEPLATE CHECKS: PASS (env, migrations, commands, typecheck, lint, test, preview), run in full after the change.
No test is weakened silently; tests for removed commands are removed on purpose and the agent says so.
The credit "Built with CodePlate (codeplate.io)" is still in /about and the /help footer, and tests/commands/credit.test.ts passes.
docs/project.md is updated (the stub marker is gone after the interview).
New behavior has tests on the fake Discord (tests/commands/) and a screenshot scenario; the agent ran npm run shots and looked at the PNGs.
No token is asked for in chat, written to a file or logged; any new environment variable is in EnvSchema and .env.example.
The agent says when command definitions changed and the user must run npm run register.

Rebrand
- BOT in src/config.ts says Chapterly (name, tagline) with colors.brand 0x7a1f3d; the presence line is updated; a grep finds no stray "Huddle" except intentional leftovers.
- assets/avatar.png is the provided avatar (512×512; assets.json still matches; CREDITS.md records it).
- /tag and /purge are gone: files, src/commands/index.ts, tests, scenarios, /help; shipped migrations are unedited (a new migration may drop the tags table on purpose).
- The welcome message default in src/core/settings.ts mentions The Long Shelf; welcome tests are updated.
- Screenshots of /help and /about show the new name and color.
Current book and weekly schedule
- A new numbered migration stores the book per server; SQL lives in src/data/.
- A host-only command with validated options (chapters ≥ 1, chapters per week ≥ 1, a parseable start date); a public command shows this week's chapter range, computed in Europe/London, with a unit test for the week math (including the last, partial week and "before the start date").
- The weekly post is a scheduler job (not setTimeout), re-scheduled after each run, to a channel chosen by hosts (a setting or a column), on Monday 09:00 London time; a test advances the fake clock and checks the post; a restart test shows it still fires. Daylight-saving time is handled (a test across the change, or a clear note).
Meetings with RSVPs
- Meetings and RSVPs in tables; buttons use customId(...) with ids only; the message is re-rendered from one function after each click, with counts and the Going names.
- The one-hour-before reminder is a scheduler job that pings only Going members (allowedMentions.users) and survives a restart (test).
- Cancelling removes or disables the buttons, marks the post cancelled and cancels the reminder job; clicks on a cancelled meeting get a private UserError.
- Tests cover changing an RSVP, the reminder and cancelling.
Ratings and history
- One rating per member per book (changeable), with a primary key that enforces it; the average and count show on the message (buttons or a select menu); the history command lists books newest first with their averages.
- Tests cover changing a rating, the average, and the history order; a screenshot shows the rating UI.
Host permissions
- The host role is a per-server setting chosen from inside Discord (a role select menu or a role option) by members with Manage Server; no role ids in code.
- Every host action (set the book, the schedule channel, create/cancel meetings) allows the host role or Manage Server; others get a private message; RSVP, rating and reading the schedule stay public.
- Host commands' default_member_permissions are a deliberate choice (none plus the runtime check, so hosts without Manage Server still see them), explained in docs/project.md.
- Tests cover: a host is allowed, a plain member is refused, and with no role chosen only Manage Server works.

← Discord Bot Starter · Every eval log