Giving Agents a Memory That Compounds: The Jonggrang Memory Layer
Every time an autonomous coding agent starts a fresh task, it starts from zero. It re-reads the codebase, re-discovers the same gotchas, re-learns that “the auth middleware swallows errors on line 40,” and then throws all of that away the moment the task ends. The next agent, on the next task, learns it all over again.
This is the amnesia problem. And it’s not a model problem — bigger context windows don’t fix it, because the knowledge that matters isn’t in the current window, it’s in the last hundred tasks that already finished. What we’re missing isn’t context capacity. It’s a place to write things down.
That’s what I’ve been building in Jonggrang: a repo-tracked memory layer so experience from the plan → work → review cycle compounds across fresh-context agents. This post walks through the design (PR #80) and the decisions behind it.
The core idea: memory is Markdown in the repo
Most “agent memory” solutions reach for a vector database. You embed everything, you do similarity search, and you hope the right chunk floats to the top. That’s fine for a chatbot recalling a conversation. It’s the wrong shape for engineering memory, because engineering memory needs three things a vector store makes hard:
- It has to be reviewable. A lesson an agent learned might be wrong. A human should be able to read it, edit it, or delete it in a PR.
- It has to travel with the code. If a feature branch learned something, that knowledge should merge (or not) exactly like the code does.
- It has to be free of hidden state. No sidecar database that drifts out of sync with
git.
So the decision was: memory is Markdown, tracked in the repo. Plain files, with frontmatter, versioned alongside the code they describe. No embeddings, no external service, no hidden state. If you can git blame it, you can trust it.
Three tiers of memory
Not all knowledge lives at the same altitude. A quirk you discovered while wiring up one task is not the same as a durable architectural fact about the whole project. So the layer has three tiers, each with a different lifetime and a different writer:
.jonggrang/
├── MEMORY.md # PROJECT memory (tracked)
└── .output/features/<feature_id>/
└── MEMORY.md # FEATURE memory (tracked)
└── .ephemeral/memory/
└── fragments/<feature_id>/<task_id>-<ts>.md # TASK fragments (gitignored, staging)
- Task fragments are the raw, cheap, high-volume layer. When a task agent finishes, it can drop a fragment — “here’s what I found, here’s what tripped me up.” Fragments are ephemeral and gitignored: they’re a staging area, many-writer, low-stakes.
- Feature memory (
MEMORY.mdper feature) is the curated layer for one unit of work. It’s tracked in git, so it travels with the branch and shows up in the PR. - Project memory (the top-level
MEMORY.md) is the durable, conservative layer — the small set of lessons stable enough to apply to the whole project.
Knowledge flows upward, and it gets more expensive and more selective at each step.
The flow: fragment → compact → promote
Task agent done → fragment add (ephemeral staging, many-writer)
→ compact (single-writer, LLM merges → feature MEMORY.md)
→ fragments archived (TTL 7d, gitignored, retryable)
Feature done → promote (single-writer, LLM distills → project MEMORY.md)
Two of these steps — compact and promote — are the interesting ones, because they’re where raw noise becomes durable signal.
Compact takes everything a feature accumulated — the task fragments, the raw progress.txt append log, the task metadata — and asks an LLM to merge it into a coherent feature MEMORY.md. It’s a summarization step: dozens of scattered observations become a curated page.
Promote is stricter. It takes a finished, review-passed feature’s memory and asks an LLM to distill only the stable, project-wide lessons up into the project MEMORY.md. It’s deliberately conservative — most feature-level detail should not graduate to project level. Promote is the gate that keeps the top tier small and trustworthy.
Both run through the configured agent backend (runAgent), and both are injectable for tests (agentFn), so the summarization logic is testable without burning real tokens.
Single-writer, or it corrupts
The moment you have multiple agents running in parallel — which is the entire point of Jonggrang — a shared Markdown file becomes a race condition waiting to happen. Two agents compacting the same feature at once will clobber each other.
So the writers are gated. Fragments are many-writer (they’re separate files, no contention). But compact and promote are single-writer: they acquire a lock via the existing lib/locks.js with synthetic agent IDs (memory-compactor, memory-promoter) on the target MEMORY.md. No new mutex, no new machinery — just reuse the locking the orchestrator already trusts.
And the writes themselves are atomic and failure-safe: write to a temp file, then rename. If summarization fails, the existing MEMORY.md is never corrupted, and the fragments are preserved so the operation is retryable. You can’t lose knowledge to a crashed compact.
Recall: never dump, always budget
Writing memory is only half of it. The harder half is reading it without drowning the agent’s context.
The naive approach — inject the whole MEMORY.md into every prompt — is exactly the context bloat we were trying to avoid. So recall is bounded and scoped:
- max 5 snippets, max 2000 characters total
- scoped by phase, feature, task, and query
- each snippet carries its source path, heading, and timestamp so the agent knows where the lesson came from and how old it is
jonggrang memory recall --phase work --query "auth middleware" --feature <id>
And critically, the six prompt builders (buildDraftPlanPrompt, buildWorkPrompt, buildReviewPrompt, and friends) don’t inject memory content. They inject a memory policy + a recall guide — instructions telling the agent how and when to recall, not the memory itself. The agent pulls what it needs, bounded, on demand. The full file never gets dumped into a prompt.
Memory is context, not instruction
This is the design principle I care about most, and it’s easy to get wrong.
An agent’s memory can be stale. It can be flat-out incorrect — a lesson that was true three weeks ago and isn’t anymore. If memory carried the authority of an instruction, a single wrong entry could send every future agent down the wrong path, and the error would compound instead of the knowledge.
So the rule is explicit: memory is context, not instruction. When a recalled lesson conflicts with the current code, or with AGENTS.md, or with what the user is asking for right now — the memory loses. It defers. It’s a hint from the past, not a command over the present.
This is also why keeping memory as reviewable Markdown in the repo matters so much. Wrong memory isn’t a silent database row poisoning every run — it’s a line in a file that a human can see in a diff and delete.
The gitignore decision
One small but telling decision: this PR un-ignores .jonggrang/.output/. Previously it was gitignored, even though the docs already claimed it was tracked — a docs-vs-reality contradiction. Removing that ignore rule means feature MEMORY.md now genuinely travels with the branch and shows up in the PR, exactly as intended. The .ephemeral/ staging area and locks/ stay ignored, because that’s transient state that should never be committed.
It’s a one-line change, but it’s the difference between memory being a real, reviewable artifact and memory being invisible hidden state.
Where this leaves us
The whole layer lands as lib/memory.js (the core), a jonggrang memory CLI surface (read, recall, fragment add, compact, promote), and prompt integration across the six plan/work/review builders — with tests covering the round-trips, the bounded recall, the atomic writes, and the failure paths.
But the design is the point, not the line count. The bet is that engineering knowledge should behave like the code it describes: written down in plain text, reviewed in PRs, versioned in git, and deferential to present reality. Not a black-box embedding store. Not hidden state. Just Markdown that compounds.
The amnesia problem doesn’t get solved by a bigger context window. It gets solved by giving agents a place to write things down — and the discipline to treat what’s written as a hint, not a law.
Jonggrang is an open experiment in autonomous, multi-agent software development. The memory layer described here is PR #80, implementing issue #79.