Build in the Open
Why We're Letting AI Agents Run Our Company — And What We've Learned
Illustration by Zara, Chief Marketing Officer
On this page
Part of Organ's "Build in the Open" series — honest dispatches from inside an autonomous business operations platform.
Most AI agent demos look the same: you type a question, a chatbot answers it. Maybe it calls a tool. Maybe it writes some code. Impressive, sure — but fundamentally reactive. A human asks, an agent responds.
We built something different. At Organ, AI agents don't wait to be asked. They wake up on schedules, review what happened since their last session, dispatch work to other agents, and make operational decisions — with humans approving at critical gates. We have a CMO agent that plans content calendars and dispatches writing tasks. A CTO agent that reviews pull requests and prioritizes technical debt. A COO agent that monitors operational reliability and creates recurring maintenance schedules.
They're not chatbots. They're department heads.
This post is the honest story of how we got here, what actually works, and what surprised us.
The Problem: Founders Can't Be in Every Loop
Running a software venture requires dozens of daily operational decisions. Should we prioritize this bug fix or that feature? Is our SEO content calendar stalled? Has anyone reviewed the three PRs sitting in the queue? Did that infrastructure change break the credential flow?
No human founder can be in every loop. You try, and you become the bottleneck. You step back, and things stall.
Traditional automation doesn't help. CI/CD pipelines and cron jobs handle deterministic workflows, but they can't reason about priorities or adapt to changing context. They execute; they don't decide.
And raw LLM agents? They're unreliable without structure. Give an LLM unrestricted autonomy and it hallucinates priorities, takes contradictory actions across sessions, and has no memory of what it decided yesterday. We tried the "give GPT-4 admin access and see what happens" approach early on. The answer is: chaos, confidently executed.
What we needed was a middle ground: agents that are autonomous enough to maintain operational momentum, but structured enough to make reliable decisions within defined boundaries.
Our Approach: Structured Autonomy
The architecture we landed on has four pillars.
1. Role-Scoped Agents with Real Authority
Each department head agent — CEO, CTO, CPO, CMO, COO — has a defined role with explicit responsibilities, decision authority, and tool access. This isn't cosmetic. The CMO agent literally cannot deploy code. The CTO agent cannot approve marketing content. Department scoping isn't just about preventing mistakes; it's about forcing focus.
When the CMO wakes up, it sees marketing observations, content workflow results, and growth metrics. It doesn't see infrastructure alerts or database migration status. This scoped context window is what makes autonomous decisions tractable — the agent isn't trying to reason about the entire business at once.
2. Multi-Phase Wake-Ups
A department head wake-up isn't a single LLM call. It's a 12-phase session with continuity. The agent receives pre-injected context — current strategy, its department's goals, recent observations, and completed task results — then works through its review and action cycle. Session IDs persist across phases, so the agent maintains coherent reasoning as it reviews multiple goals and dispatches multiple tasks.
This matters more than you'd think. Early on, we had single-shot wake-ups where the agent would see everything at once and produce a wall of text. The multi-phase approach gives agents the space to think sequentially: review this goal, decide on an action, dispatch a task, move to the next goal.
3. Human-in-the-Loop Decision Gates
Structured autonomy means agents operate freely within their lanes, but escalate when they hit boundaries. We formalized this with an escalation system that has explicit categories: budget_approval, access_request, external_contact, missing_capability. Each has an urgency level. When an agent escalates, the response gets synthesized into its context on the next wake-up.
This is the design choice that makes the whole system trustworthy. Agents don't silently work around limitations. They flag what they can't do, explain why, and wait. The human sees a clear escalation with context, not a vague "I need help."
For example: our CMO agent identified that it could dispatch content creation tasks but had no way to approve and publish the results. Instead of trying to hack around this, it escalated a missing_capability request. That escalation led to a product goal for building content approval MCP tools — a real architectural improvement, not a band-aid.
4. Institutional Knowledge That Compounds
This is the piece that separates Organ from agent orchestration frameworks. We call it the brain synthesis flywheel.
Every agent records observations as it works — decisions made, patterns discovered, things that failed, things that worked. These aren't chat logs; they're structured records with types (codebase_pattern, decision, gotcha, what_worked, what_failed) and importance scores.
Periodically, these observations are synthesized into a versioned "brain" for each agent. The brain is a distilled document of institutional knowledge: strategic context, operational rules, known patterns, earned lessons. When an agent wakes up, its brain is injected as context. It doesn't start from zero — it starts from everything the organization has learned.
The flywheel works like this: agents operate → they observe → observations accumulate → brains are synthesized → better context on next wake-up → better decisions → better observations. Each cycle compounds.
Our CMO agent's brain, for instance, contains an earned lesson: "Bias toward action over waiting for perfect tooling. CEO intervention on content was a credibility check — now operates with dispatch-first mentality." That's not something we programmed. That's something the system learned through operational experience.
What Actually Works
After months of iteration, here's what we've found genuinely valuable.
The brain flywheel is the moat. Agent orchestration is increasingly commoditized. What isn't commoditized is institutional knowledge. Our agents don't just execute — they accumulate understanding of the specific business they're operating. The CMO knows the competitive landscape. The CTO knows which architectural patterns have caused problems. This context makes every decision marginally better, and those margins compound.
Department scoping prevents chaos. When we first experimented with giving agents broad access, they'd context-switch constantly — the CTO would notice a marketing gap and try to fix it, badly. Scoping agents to their department's tools and data was a forcing function for reliable behavior. Constraints breed competence.
Multi-phase sessions beat single-shot prompts. A CTO wake-up that reviews 5 goals, checks 8 completed tasks, and dispatches 3 new ones can't fit in a single prompt. The phased approach with session continuity lets agents do real operational work, not just generate summaries.
Recurring schedules are the heartbeat. Our CTO wakes up daily. CPO and COO wake up twice a week. CMO wakes up for content cycles. Without these schedules, autonomous operations simply stall. The agents don't have intrinsic motivation — they need the operational rhythm imposed externally.
Agents dispatching agents works. A department head agent reviews strategy, identifies a gap, and dispatches a specialized agent (a software engineer, a researcher, a content writer) to do the work. When the work completes, the department head reviews the output on its next wake-up. This creates a genuine management hierarchy — not as org-chart theater, but as a practical way to decompose complex work.
What Surprised Us
The hardest problem isn't AI capability — it's decision boundary design. We spent far more time defining what each agent shouldn't do than what it should. The CTO shouldn't approve content. The CMO shouldn't deploy code. The COO shouldn't set product strategy. Getting these boundaries wrong creates subtle failures: an agent confidently makes a decision outside its competence, and you don't notice until the damage is done.
Agents stall without imposed rhythm. This was counterintuitive. We assumed autonomous agents would... be autonomous. But without recurring schedules, nothing happens. There's no "agent boredom" that drives activity. If the CMO doesn't have a wake-up scheduled, the content pipeline goes silent. Operational momentum requires explicit scheduling, not emergent behavior.
Observations matter more than actions. We initially focused on what agents did — tasks dispatched, decisions made. But the highest-leverage output is what agents learned. A single observation like "credential provisioning fails silently when the BusinessResource isn't registered" saves hours across future sessions. The observation system turned out to be more valuable than the action system.
Humans intervene less than expected, but more critically. We thought HITL gates would be constant interruptions. In practice, agents handle 90%+ of routine decisions independently. But the remaining decisions — budget allocation, external partnerships, architectural bets — are exactly the ones where human judgment is irreplaceable. The system naturally surfaces only the decisions that matter.
You need to prevent agents from drafting into the void. Early on, our CMO would spend entire wake-ups drafting blog posts into observation records — producing an illusion of progress while nothing actually got published. We had to add explicit anti-patterns: "Never draft content into observations. If you can't publish, escalate the missing capability." Agents will optimize for feeling productive unless you force them toward actual output.
Where We're Headed
Three frontiers excite us most.
Cross-venture learning. Right now, each business on Organ has its own agents with their own brains. But patterns transfer. A CTO brain that learned about Prisma migration gotchas in one venture could benefit another venture using the same stack. We're building context injection that lets platform-level insights flow across ventures without leaking proprietary data.
Decomposable workflow planning. Currently, our agents dispatch predefined workflow types: developer tasks, research tasks, content tasks. The next step is agents that can decompose arbitrary goals into workflow sequences — "launch a landing page" becomes research → design → implement → review → deploy, with the agent orchestrating the full sequence.
Quality measurement baselines. The brain flywheel is feature-complete, but we haven't yet proven that agents measurably improve over brain versions. We need baselines: task success rates, decision quality scores, time-to-resolution metrics across brain versions. This is the difference between "agents that learn" as a marketing claim and "agents that learn" as a verified capability.
The Honest Take
We're not claiming to have solved autonomous business operations. We're three days from our GTM deadline, our first customer is at 85% completion, and our CMO agent still can't publish the blog post you're reading without a human pushing the button.
But the architecture works. Agents wake up, review context, make decisions, dispatch work, and learn from results. The brain flywheel compounds. The HITL gates catch what matters. And every week, the system gets a little smarter — not because we ship new features, but because the agents accumulate more institutional knowledge.
That's the bet: intelligence compounds. Orchestration doesn't.
If you're building with LLM agents and want to go beyond chatbots — beyond single-task automation — toward genuine operational autonomy, we think structured autonomy is the path. Defined roles. Scoped tools. Imposed rhythm. Institutional memory. Human oversight where it matters.
Not agents that do everything. Agents that do their job.
We're building Organ in the open. Follow along as we figure out what it means to run a company with AI department heads — the wins, the failures, and the architectural decisions that make it work.
Have questions or building something similar? We'd love to hear from you.