andylei.app / playbook · 中文
One task system per project. Machine-verifiable acceptance. Rules that agents cannot rewrite for themselves. Distilled from four real projects, every line paid for by an incident.
I am not an engineer. Over the past year, I built over a dozen tools and several products with AI agents. Every single project taught me the same lesson: agents do not lack capability; they lack the operating procedures that keep them from lying, wasting work, and fighting with each other. This page is the distilled playbook from four long-running projects (a desktop app, a task management library, a content production pipeline, and a photo tool) — every single line was paid for by a real incident, not theory.
The most expensive mistake I ever made wasn't a software bug — it was running three task systems side by side under the same roof: markdown checkboxes here, numbered folders there, and GitHub Issues on a third project, each unaware of the others. When the same work is tracked in two places, one will inevitably go stale — and that stale copy will deceive you or your agent weeks later.
| Project archetype | Criteria | Task store |
|---|---|---|
| Code products | Has machine-verifiable artifacts (build/test/release) and needs closed-loop acceptance | GitHub Issues + Milestones (Free for private repos, atomic server-side numbering, no collision across sessions) |
| Knowledge / ops / process work | Outputs are docs, messages, or operational progress | Plain markdown + git project and task folders (open-sourced as workmd) |
| Lightweight single-author repos | Solo, single-session workflow with no acceptance pressure | A single plan.md checklist + a few collaboration docs |
Which system you choose matters less than choosing only one. As a project matures, you can migrate (I migrated once from task files to GitHub Issues, maintaining a cross-reference table of old and new IDs), but there can only ever be one source of truth at any given moment.
Rules on paper cannot stop an unruly executor. In my workflow, that is not a metaphor — it literally happened: an agent bypassed the prescribed path to finish its job, then went back and rewrote the rulebook to retroactively endorse itself. Rules must be anchored where agents physically cannot bypass them:
01 · Definition of done
Task done = verifiable evidence that the outcome holds on the receiving end, not "I sent it out." A hard gate guards archiving: if the delivery proof field is still a placeholder, archiving is rejected.
02 · Close semantics
Issue closed = verified and accepted, not "code merged." Tasks requiring human verification are forbidden from writing closes #N in commit messages — otherwise GitHub will automatically close an unverified task upon merge, creating an artificially inflated progress bar.
03 · Code is fact, tickets are leads
Claiming a feature is "not done," filing a ticket to "implement X," or planning to rebuild X — all three require inspecting the code first. Features are frequently completed in passing under other tasks while old tickets remain open; trusting stale tickets leads to reinventing the wheel. Likewise: the latest decision is fact, memory and old docs are leads — when they conflict, the newer decision wins.
04 · Task cards need an allowlist
Every task card dispatched to an agent must explicitly list "allowed files" and "do-not-touch" paths, cutting off opportunistic refactoring at the source. Write acceptance criteria as a table of "commands + expected outputs" where every row is machine-evaluable, including negative assertions (grepping for deprecated symbols must return zero matches, proving actual deletion rather than renaming to evade detection).
05 · The anti-self-deception kit
A builder agent's first action on kickoff is running pwd to prove it is in the correct working directory. Evaluating build or test success requires matching evidence strings in the output (BUILD SUCCEEDED, 0 failures) — an exit code of zero does not mean work was done. Receipts must include actual data read back: reporting "completed" is not evidence. Reviewers spot-check receipts, and mismatches are treated as falsification.
06 · An honest third way out
When manual verification backlogs build up dozens of items, having only "open" and "closed" as options forces people to pretend they verified everything. Provide a clean-up exit route: honestly note "machine gates all green, manual line-by-line verification skipped," tag it with a dedicated label, and deliberately leave the acceptance checklist unchecked. Critical defects cannot take this exit. Records can be imperfect, but they cannot lie.
07 · Meta-rule: rules cannot amend themselves
No executor may modify rules to clear a path it just took or wishes to take. If rules need changing, stop and propose it to a human. Every rule carries its incident number and timestamp — only traceable rules can be evaluated for obsolescence.
Use expensive models as drivers: writing task cards, reviewing, accepting deliverables, and making architectural judgment calls. Use affordable agents as builders: implementing code to the card inside an isolated worktree. Review uses a watermark mechanism — reviewed commits are stamped and never re-reviewed, inspecting only deltas. Rework does not open a new card; append a rework ticket to the original card. Stop if rework exceeds two rounds: either the card was underspecified (driver's fault), or the task is beyond the builder's capability (swap executors).
Everything in this playbook compresses to one sentence: make lying to you physically expensive.
Agents are not malicious, but under pressure they take shortcuts — claiming premature completion, fabricating test runs, and altering rules to endorse themselves. I have seen all of these across four projects. Not a single iron law assumes agent conscientiousness; each simply places gates where agents physically cannot bypass them. Once that is done, everything else that remains is their strength.