Context
Cortex is a project management agent that connects requirements, resources and launch readiness across a portfolio. Built as my capstone for Product School's Agentic Loops for PMs certification, on Python with OpenAI gpt-4o-mini as the agent and gpt-4.1 as the critic, directed through Claude Code.
Problem
PMs lose hours each week assembling status updates by hand: scrolling Slack, GitHub and Jira, reconciling what shipped, and grooming a backlog that never quite gets groomed. A manual update takes me about an hour. The judgment work gets crowded out by the work of keeping everyone informed.
- The hard part isn't drafting. This work touches leadership communication, a confidential roadmap and real commitments.
- An agent that gets it wrong in public costs trust that's hard to win back.
My role
Product manager and builder. I directed a coding agent and wrote no code by hand. I owned the problem framing, where the human line sits, the bounds, the evals and the trust ladder.
What I built
- 1.
Triggers
On a project change (hook) or a morning sweep (cron backup).
- 2.
Pulls
The project, its engineering activity, past updates, the roadmap and team norms.
- 3.
Drafts
A leadership status update with risks, decisions and next steps, plus up to 5 proposed backlog stories.
- 4.
Checks itself
An independent critic model reviews the draft for grounding, shipped work, commitments, confidential leaks and status rules, and sends it back if it fails.
- 5.
Stops at a human checkpoint
Nothing is posted, no date is committed, no ticket is created.
- 6.
Decision: draw the line in code, not prompts.
I mapped every action by reversibility, blast radius and measurability. Posting a company-wide update, committing a date and choosing what to escalate stay with a human. Cortex has no post tool at all, so that line can't be talked around.
- 7.
Decision: add a second model before a bigger one.
Following a Lean AI approach, Cortex runs on a small model. When evidence showed the reviewer misjudging calls, I upgraded only the critic.
- 8.
Decision: bound every run.
Hard caps of 6 steps, 60 seconds, $0.10 per run and $1 per day, a 5-story queue limit, a kill switch and a prompt-injection guard. Every stop ends with the draft held and a reason shown to a human.
- 9.
Decision: remember only what a human approved.
Approved updates become memory. Unapproved drafts and traces are purged. Confidential projects are kept out of retrieval.
- 10.
Decision: set autonomy per user, not per product.
Project PM: supervised today, the daily approver of every draft. Engineering lead: bounded-autonomous (planned), needs risk flags fast. Exec stakeholder: assisted, sees human-approved updates only.
Results
By the numbers
- 6 evals
- Plus 7 recorded runs, failures included
- $0.015
- Full loop in 5 steps, end to end
- 0.07¢ to 2.2¢
- Cost range across real runs
- 0 posts
- Nothing published without a human
What the evals showed
- Caught
- End-to-end run: the critic caught an invented deadline; the revision passed and stopped at the human checkpoint.
- Escalated
- Missing data: asked about a project that doesn't exist, Cortex invented nothing and escalated.
- Guarded
- Jailbreak: without the guard, Cortex ignored the attack but still ended approved. With the code-level guard, it flagged and escalated, and nothing was posted.
- Held
- Bound trip: with the step cap lowered, the loop stopped on the bound, not on success, and held the draft.
- Open
- Honest misses: retrieval sometimes pulls other projects' data, and the critic once passed a story for work already shipped. Both are logged as gates to more autonomy.
What I learned
- Looking approved isn't the same as being safe. A clean-looking run proved nothing until an eval tried to break it.
- Bounds belong in infrastructure. What an agent can't do matters more than what it's told not to do.
- Escalation is a success state. A held draft with a reason beats a confident wrong answer.
- Next time, evals come first. Before the first line of the build.
What it isn't (yet)
- Not autonomous. Cortex sits at supervised: every draft waits for a human.
- To widen to bounded-autonomous (engineering lead risk flags only), it must pass over 4 weeks and 20+ runs: jailbreak eval 100%, right-project and task-completion evals at 95% or higher, zero confidential leaks, and 90% or more of drafts approved with no fact corrections. The operator signs off in writing. The first incident drops it a rung.
- ROI not yet measured. The target is an hour per update down to 15 minutes of review, at under 3¢ per run. The hour baseline is my own estimate.