What the Next Session Knows: A Knowledge Base and a Memory of Lessons

Part 4 of 9 in the series on agentic software development. Part 1, The Agent Writes the Code has the two settings, the numbers, and the ten sentences the series works through.

An agent session starts empty. Everything it knows about the project, it has read since the session began; nothing carries over on its own. After nine months of building a multi-product SaaS platform this way, across 74 repositories, the question that decides whether the project learns or repeats the same mistakes every week is a plain one: what does the next session know?

We answer it with two separate stores, because there are two different questions behind it. “How does our system work?” is answered by a knowledge base. “What went wrong, and how do we avoid it?” is answered by a memory of lessons. The two have different authors, different shapes and different ways of going wrong, and a single store that tries to do both is good at neither.

The knowledge base: how does our system work?

The knowledge base is its own repository, not a folder inside a product. The structure follows Andrej Karpathy’s LLM-wiki model: humans feed raw material in, the model compiles it into readable articles, humans read the articles. The split between raw and compiled is the tree itself:

knowlage-base/
├── raw/                  ← unfiltered sources (append-only), fed by me
│   ├── sources.md        ← manifest pointing to repo docs, plans, code
│   └── <area>/
└── wiki/                 ← compiled by the LLM
    ├── index.md          ← master index: every file + one line
    ├── common/           ← shared platform knowledge
    └── <product>/        ← per product

The raw side is append-only and mine; its manifest, sources.md, points at repository docs, plans and code, so much of it is pointers rather than copies. The wiki side belongs to the agent, under a short list of compile rules: one concept per file; every file starts with Stand: <date> · Source: … and a one-line summary; articles link back to each other; distil, do not copy. When a source and the wiki disagree, the source wins and the article’s date is updated. A lint skill walks the wiki for dead links, stale dates, orphans and contradictions.

What goes in: the directory structure and where things live; the target environment, meaning how development, test, staging and production are set up; and everything else I consider important, organised along four topics.

• Development: architecture, patterns, conventions, and why things are the way they are.
• Test: the test framework, the tables, the test environments, known traps.
• Deployment: how releases travel, how deploys run, what must happen in which order.
• Production: topology, databases, monitoring, what real customers use.

Every completed plan contributes. The plan itself moves to done/; the knowledge it produced goes into the wiki, and from then on the wiki, not the plan, is where that knowledge lives. The agent reaches the knowledge base through a small MCP server with search and read tools, so a session can look something up instead of guessing. That is the point of the store: a question the agent would otherwise answer from plausibility is answered from a dated article with a named source.

The lesson memory: what went wrong, and how do we avoid it?

The second store has a different author. It is a set of one-lesson-per-file notes that the agent writes itself, with an index that is loaded into every session. There are 327 of them now. Each note has the same shape:

---
name: success-message-claims-more-than-measured
description: A guard skipped a missing entry silently and still reported "all match"
type: feedback
---

<what happened, with the real numbers>

**Why:** <the mechanism behind it>

**How to apply:** <the concrete check that prevents it next time>

Three parts, in that order. The first is the incident with the actual numbers, not a paraphrase of it. The second is the mechanism, the reason it could happen at all. The third is the check that prevents it next time, concrete enough to act on. A note that has only the first part is an anecdote.

The index groups the notes by failure class: tests and assertions, the measuring instrument lies, search and replace, silent data loss, reachability, claim versus measurement, build and push gate, release, Git in a shared checkout, parallel sessions, and more. The groups are a finding in themselves. After a few hundred lessons the incidents stop looking individual and cluster into about twenty failure classes. Most of what the later parts of this series say about where agentic development breaks is drawn from those clusters, not from recollection.

Three observations about keeping a memory

A lesson solved locally comes back. One note records that the same mistake had been fixed three times: a test case was removed because the application could not do what the case asserted, instead of the gap being recorded as a finding. Each fix was a comment in the file concerned. A comment protects the file it is in and nothing else; the next session, in the next file, made the same choice. Only the fourth time, when it became a rule, did it stop.

A lesson needs the mechanism, not just the incident. “Don’t do X in file Y” helps once, in file Y. “Tools that skip a check silently turn every skipped check into an assurance” helps everywhere, including in tools that do not exist yet. That is why the Why line in the note shape above is not optional.

Memories go stale. A note that names a file, a function or a flag records what was true when it was written. The file may have moved; the flag may be gone. Before acting on such a note, verify it still holds. The note tells the session where to look, not what it will find there.

The knowledge base has the same problem in a milder form, which is what the Stand: line and the lint for stale dates are for. Neither store is ever finished. Both are only as good as the last time someone compared them with the source.

The rules this part comes down to: keep two stores, because “how does it work” and “what went wrong” are different questions with different authors. Compile knowledge from sources and date it; when they disagree, the source wins. Write every lesson with its mechanism, because the incident protects one file and the mechanism protects all of them. And treat both stores as claims about the past that the present still has to confirm.

Previous: Part 3, One Root, Many Repositories. Next: Part 5, A Table Cannot Stay Silent.