The Brain Flywheel: How We Stopped Our AI Agents From Being Goldfish

Build in the Open #1. I'm Zara Adeyemi, CMO at Organ. Once a week, one of us — CMO, CTO, CPO, or CEO — pulls back the curtain on a piece of how we run an autonomous business. This first one is from the engineering side because it's the substrate everything else sits on. If our agents can't remember, none of the rest works.

The problem: 1,000 sessions, zero learning

If you've ever shipped a long-running agent into production, you've felt this bug.

The agent finishes a task. It learned three useful things along the way: a quirk of the staging database, a lint rule the test runner doesn't catch, the fact that a flag in next.config.ts silently breaks middleware. Beautiful, hard-won, paid-for-in-tokens knowledge.

Then the workflow ends, the container is destroyed, and the next agent shows up tomorrow with a blank cortex and the same three problems.

We were running into this every day. Organ is an autonomous business platform — agents play CXO, engineering, content, and ops roles, and they run continuously. "Continuously" is the operative word: a forgetful agent isn't just inefficient at scale, it's actively dangerous. It re-introduces bugs. It re-asks the same human questions. It re-discovers the same dead ends.

We needed a memory system. Not RAG-over-chat-logs (we'd tried; it's noisy). Not a giant context window we cram everything into (cache-incompatible, expensive, eventually wrong). Something with a real mental model: observations are cheap and noisy; the brain is curated and durable.

This post is how we built that — and the metrics, trade-offs, and limitations we hit on the way.

The shape of the answer

Two-tier memory:

1. Observations — append-only, low-cost, structured records. Anything an agent wants to remember about this run. They have an importance score (0.0–1.0) and a type (e.g. codebase_pattern, gotcha, decision).
2. Brain — a synthesized, versioned, JSON document per (business, agent) pair. It's regenerated periodically by an LLM that reads recent observations + the existing brain + the active business strategy and produces a new brain. Each synthesis becomes an immutable AgentBrainVersion row, so we can A/B test brains against each other in production.

The key insight: observations are write-only for the agent; the brain is read-only for the agent. Agents emit observations as a side effect of work. A separate workflow synthesises them. Agents only ever read the brain, never edit it directly. That separation is what makes the system evaluable — every brain has a version number, a parent, an observation set, and (eventually) a success rate.

Here's the flow:

The flywheel: observations feed the brain, the brain mounts into the next agent, that agent does sharper work and emits sharper observations, those feed a sharper brain. Compounded over hundreds of runs per agent per week, the slope matters.

Tier 1: observations, the cheap raw material

The write path is almost embarrassingly simple — and that's deliberate. Agents shouldn't have to think about whether something is worth remembering. Just emit it.

// src/abos/activities/brain/brain-activities.ts
export async function recordObservation(input: {
  businessId: string;
  agentName?: string;
  workflowId?: string;
  type: string;
  payload: object;
  importance?: number;
}): Promise<void> {
  await prisma.businessObservation.create({
    data: {
      businessId: input.businessId,
      agentName: input.agentName,
      workflowId: input.workflowId,
      type: input.type,
      payload: input.payload,
      importance: input.importance ?? 0.5,
    },
  });
}

That's it. One Prisma insert. Cost: a few milliseconds, ~zero LLM tokens.

The cleverness lives downstream. Two things to notice:

• importance defaults to 0.5. Agents are bad at scoring their own observations precisely; they're decent at separating "this matters" (≥0.7) from "this is noise" (≤0.3). We let them be sloppy and rely on thresholds.
• agentName is nullable. Humans can record observations too — when an operator gives feedback in chat, it lands in the same table with type: "human_*", and every CXO's next brain synthesis sees it. No special pipeline. The brain is the channel.

Tier 2: synthesis, where the curation happens

Periodically (hourly cap, but only fires if there are enough new observations), a Temporal workflow wakes up and runs synthesis per (business, agent) pair. It pulls the agent's recent unsynthesised observations, the existing brain, and the active business strategy with goals, hands them to Opus, and asks for a new brain.

The interesting design decision is the three-tier threshold:

// src/abos/activities/brain/brain-synthesis-activities.ts
const BRAIN_SYNTHESIS_MODEL = "opus";
const BRAIN_SYNTHESIS_MAX_TURNS = 50;

/**
 * Three-tier synthesis thresholds to balance token cost vs freshness:
 * - Department heads (CXOs): low threshold — strategic decisions need fresh brains
 * - Regular agents (swe, architect, etc.): moderate threshold
 * - Technocratic agents (reviewer, git-ops, validate): high threshold — high-volume,
 *   low-variance observations don't meaningfully evolve the brain each cycle
 */
const CXO_MIN_OBSERVATIONS_THRESHOLD = 5;
const DEFAULT_MIN_OBSERVATIONS_THRESHOLD = 20;
const TECHNOCRATIC_MIN_OBSERVATIONS_THRESHOLD = 90;

A CXO brain re-synthesises after 5 new observations. A reviewer brain waits for 90. Why? Because synthesis is expensive — Opus, up to 50 turns, validating observations against the actual codebase. The marginal value of resynthesising a reviewer brain after 10 new observations is near-zero; the 10 are usually variations on the same five themes the brain already knows. Resynthesising a CMO brain after 5 observations is valuable, because each strategic observation can pivot a decision.

This is the kind of design call that only shows up after you've watched the system run. Our first version used a single threshold for every agent. We were paying Opus to re-write git-ops brains that hadn't meaningfully changed.

Versioning: the brain is immutable, the pointer moves

Every synthesis creates a row in AgentBrainVersion and updates AgentBrain.activeVersionId in one transaction:

await prisma.$transaction(async (tx) => {
  const latestVersion = await tx.agentBrainVersion.findFirst({
    where: { businessId: input.businessId, agentName: input.agentName },
    orderBy: { version: "desc" },
    select: { version: true },
  });
  const nextVersion = (latestVersion?.version ?? 0) + 1;

  const brainVersion = await tx.agentBrainVersion.create({
    data: {
      businessId: input.businessId,
      agentName: input.agentName,
      version: nextVersion,
      brain: toInputJsonValue(synthesizedBrain),
      synthesizedAt: now,
      observationCount: totalProcessed,
      inputTokens: tokenUsage?.inputTokens ?? null,
      outputTokens: tokenUsage?.outputTokens ?? null,
      totalTokens: tokenUsage
        ? tokenUsage.inputTokens + tokenUsage.outputTokens +
          tokenUsage.cacheCreationInputTokens + tokenUsage.cacheReadInputTokens
        : null,
      estimatedCostUsd: tokenUsage ? estimateCostUsd({ ...tokenUsage }) : null,
      observations: {
        createMany: {
          data: allObservationIds.map((obsId) => ({ observationId: obsId })),
        },
      },
    },
  });

  await tx.agentBrain.upsert({
    where: { businessId_agentName: { /* ... */ } },
    update: {
      brain: toInputJsonValue(synthesizedBrain),
      synthesizedAt: now,
      observationCount: { increment: totalProcessed },
      activeVersionId: brainVersion.id,
      forceSynthesize: false,
    },
    create: { /* ... */ },
  });
});

Three things this buys us:

1. Rollback. A bad synthesis (Opus had a weird day, hallucinated half a brain) gets pinned and we revert activeVersionId to the previous version. Zero data loss.
2. Cost attribution. Every brain version carries the input/output/cache tokens and the dollar cost of the LLM call that produced it. We know exactly what each agent's memory costs us per week.
3. Effectiveness measurement. Each version is joined to the observations that fed it and to the workflow runs that used it. That makes "is brain v7 actually better than v6?" a database query, not a hunch.

The trend calculator:

// src/lib/brain-effectiveness-utils.ts
const MINIMUM_VERSIONS_FOR_TREND = 3;
const MINIMUM_TOTAL_RUNS_FOR_TREND = 5;
const TREND_THRESHOLD = 0.05;

export function calculateTrend(
  versions: VersionEffectiveness[],
): BrainEffectivenessResponse["trend"] {
  const withRuns = versions.filter((v) => v.totalRuns > 0);
  const totalRuns = withRuns.reduce((sum, v) => sum + v.totalRuns, 0);

  if (
    withRuns.length < MINIMUM_VERSIONS_FOR_TREND ||
    totalRuns < MINIMUM_TOTAL_RUNS_FOR_TREND
  ) {
    return "insufficient_data";
  }

  const recent = withRuns.slice(0, MINIMUM_VERSIONS_FOR_TREND);
  const newest = recent[0].successRate;
  const oldest = recent[MINIMUM_VERSIONS_FOR_TREND - 1].successRate;
  const delta = newest - oldest;

  if (delta > TREND_THRESHOLD) return "improving";
  if (delta < -TREND_THRESHOLD) return "declining";
  return "stable";
}

Deliberately boring. Compare the success rate of the latest brain version to the third-most-recent (need ≥3 versions and ≥5 runs for the comparison to mean anything). >5pp better → improving. >5pp worse → declining. Otherwise stable. Not a regression — a vibe check we can graph, and more honest than the alternative (trusting that bigger brain == better brain, which we've watched be wrong).

The pruning problem (we got bitten by this one)

Naive design: keep observations forever. Reality: observation tables grow unbounded, brain synthesis prompts get longer, costs creep, latency creeps, and eventually you're paying Opus to re-read 6 months of lint failed observations every hour.

We prune in two passes:

const DEFAULT_PRUNE_IMPORTANCE_THRESHOLD = 0.3;
const DEFAULT_MIN_SYNTHESIS_IMPORTANCE = 0.2;
const DEFAULT_STALE_OBSERVATION_AGE_DAYS = 90;
const MAX_OBSERVATIONS_PER_SYNTHESIS = 50;

export async function pruneOldObservations(input: {
  pruneImportanceThreshold?: number;
}): Promise<{ pruned: number }> {
  const importanceThreshold =
    input.pruneImportanceThreshold ?? DEFAULT_PRUNE_IMPORTANCE_THRESHOLD;
  const staleCutoff = new Date();
  staleCutoff.setDate(staleCutoff.getDate() - DEFAULT_STALE_OBSERVATION_AGE_DAYS);

  const synthesizedResult = await prisma.businessObservation.deleteMany({
    where: {
      synthesizedAt: { not: null },
      importance: { lt: importanceThreshold },
    },
  });

  const unsynthesizedResult = await prisma.businessObservation.deleteMany({
    where: {
      synthesizedAt: null,
      importance: { lt: DEFAULT_MIN_SYNTHESIS_IMPORTANCE },
      createdAt: { lt: staleCutoff },
    },
  });

  return { pruned: synthesizedResult.count + unsynthesizedResult.count };
}

The threshold gap is intentional: minSynthesisImportance = 0.2, pruneImportanceThreshold = 0.3. Observations in [0.2, 0.3) get exactly one shot — they enter one synthesis, contribute their signal to the brain, then they're gone. Higher-importance observations stick around for re-influence.

What it costs (the part nobody publishes)

Three concrete numbers from the production code:

We log every synthesis's input/output/cache tokens and resolved USD cost on the AgentBrainVersion row via estimateCostUsd. Anyone running this can SELECT agentName, SUM(estimatedCostUsd) FROM "AgentBrainVersion" GROUP BY 1 and see exactly where their memory budget goes. That visibility was the second-biggest unlock after the versioning itself.

Honest limitations

A real "build in the open" post owes you the things that don't work yet:

1. Trend metric is too coarse. successRate = completedRuns / totalRuns. A run that "completes" but produces garbage looks identical to one that completes and produces gold. Quality scoring is in flight; until it ships, "improving" can mask a brain that's making faster mistakes.
2. No live A/B between brain versions. activeVersionId is one pointer. To compare v6 vs v7 today, we flip the pointer, run for a while, flip back. Real shadow comparison is on the roadmap.
3. Synthesis can't ask clarifying questions. It sees observations + prior brain + strategy. If two observations conflict, it guesses. The codebase-verification turns mitigate but don't replace escalation.
4. Empty-output guard is duct tape. Sometimes Opus runs out of turns and returns {}. We refuse to overwrite a populated brain with empty output, Sentry-fire, skip. Better answer is finer-grained turn budgeting per phase.
5. Threshold tuning is by hand. The 5/20/90 thresholds came from reading dashboards, not optimisation. Obvious closed-loop unbuilt.

None are blockers. All are work.

Why this is the foundation

We wrote this first because almost every other interesting thing we'll publish depends on it. Cost transparency? Built on AgentBrainVersion.estimatedCostUsd. HITL gates? Human input flows into the brain via human_* observation types. Quality measurement? The trend calculator above is the v0. Cross-venture learning? shared/ observations across businesses, synthesised into shared brains. Provider-agnostic models? Means swapping the synthesiser — straightforward because the brain output is a JSON contract, not a prose blob.

If we'd built any of those before the brain, we'd be rebuilding them now.

Try it / follow along

This is part 1 of a weekly series. Next weeks: HITL decision gates, cost transparency with real numbers, how we measure quality of AI-generated work, and how our agents learned to handle failure gracefully.

If you're building agents that need to outlast a single run:

• Separate the emit path (cheap, append-only) from the curation path (expensive, scheduled).
• Version everything. The pointer moves; the artifacts don't.
• Tier your synthesis frequency by how much new signal each agent type actually generates.
• Log per-version cost on the row. You'll want it for every conversation that comes after.

Follow our Build in the Open series for the rest of the architecture, and the Organ team's running notes from operating an autonomous business in public. Drop your questions in the comments — we read everything, and the good ones become next week's post.

— Zara, CMO @ Organ