Thinking in Outcomes
In 2023, GitHub Next shared SpecLang, a prototype for building software entirely in natural language. The spec was the source of truth; code was just™ output.
The idea is back. Recently, Chad Fowler gave a talk called “Throw Away the Code, Keep the System”, arguing that code may become something we regenerate as needed. Don Syme at GitHub Next describes the model this way: “The spec is the intent, the code is the actualisation, and a coding agent is a compiler.”
Most of this discussion is about building software: describe what you want, and an agent writes the code. But what happens after it’s built? If nobody reads the code, how does the owner know the app still does what they asked?
Usually an engineer handles that: setting up error tracking, writing unit tests, and monitoring user-facing behavior, like whether people can still sign in.
Agents are starting to help. Sentry’s Seer diagnoses errors and hands fixes to coding agents. Polylane has agents read metrics, logs, and traces, then open pull requests. These tools assume a team has engineers to monitor the app’s telemetry and review fixes.
What about apps built by non-engineers? Tools like Lovable let people build products they might otherwise have hired engineers to make. These owners may have nobody to review fixes or set up monitoring.
An agent could do that work. It could gather clues from across the app and its surroundings: booking patterns, support tickets, posts about a missing sign-in link, or a change to an email service. It could compare them with what the owner says the app should do, then investigate when they point to a problem. The hard part is making sense of scattered signals, not just running tests.
True to form, I don’t pretend to know whether an agent can do this. I do have one made-up example that hopefully helps clarify what I’m trying to say.
Writing the spec
Imagine a yoga studio owner with 140 members. In May, they use an agent to build a booking page instead of paying for a booking service. Their spec:
- Members can sign in and book a class.
- Members can join a waitlist when a class is full.
- Cancelling frees the spot for someone else.
I’ll call each line of the spec a promise.
The agent writes the code, and the owner never looks at it. For each promise, the agent also sets up ways to check that it holds. The checks include a test member who uses the live site, counts from the booking database, and news from the services the app relies on. I don’t know whether an agent could do this reliably, but here’s a rough sketch.
The owner doesn’t need to install an analytics service or set a threshold. The test member covers promises that real members rarely exercise, like joining the waitlist. None of these checks is conclusive on its own.
Monitoring the spec
The first promise depends on email. Members sign in with a link the booking page emails them. The page sends those emails through an email service, from an address the service shares among many of its customers.
One Tuesday at 9:06, Gmail starts putting mail from that shared address in spam. The booking page still sends the links, and the email service still marks them delivered. Nothing in the code changed or threw an error.
The figure shows one member’s Gmail inbox, booking activity, and the agent’s measure of whether members can sign in and book.
Members with Gmail addresses stop booking at 9:06. By 9:09, 7 sign-in links have gone out and 3 have been used. None of the 4 sent to Gmail addresses has been used. With so few members, that count isn’t proof on its own. Then a member posts that their sign-in link never came, and the email service’s status page says Gmail is filtering some mail from shared addresses. Together, that’s enough to investigate.
The agent didn’t need a preconfigured dashboard or alert. It noticed a change in the app’s activity, then used the outside signals to help make sense of it. Real signals may be sparse, noisy, or harder to connect.
Finding the cause
The agent first looks at who used their sign-in link. Only the Gmail links went unused, so the problem looks specific to Gmail. The email service says it delivered the Gmail messages, but that only tells the agent they left the service, not that they reached an inbox. So the agent requests a sign-in link using a test Gmail account.
The link lands in spam. The test reproduces the member’s problem, and the status page points to a likely cause: Gmail is filtering the shared address.
Every clue so far came from one stream of things the agent reads. The app sends it bookings and sign-ins the moment they happen. It checks status pages and changelogs every 15 minutes, and it searches X, Instagram and Hacker News every 5 minutes for posts about the studio or the services the app uses. Most of what it reads has nothing to do with the booking page.
Fixing it
The agent proposes sending sign-in emails from the studio’s own domain instead of the shared address. That means changing the studio’s email setup, so it asks the owner first. It sends an update through a channel the owner already uses. Here, it appears on the owner’s phone, with the cause and proposed fix:
The owner approves. The agent updates the spec, rebuilds the booking page from it, and repeats the test with the Gmail account. This time the link reaches the inbox. Gmail members start signing in again too.
Reading the spec
The owner likely needs visibility into all of the promises, not only the one that broke: what the app is meant to do, whether it’s doing it, and what the agent has learned. In my sketch, that’s a page showing the spec and the latest status of each promise. I think of it as a new control plane for software. Fowler calls each line an evaluation: “an assertion of a bunch of intents that is not tied to the implementation at all.”
Each row is a promise; the bars show its last 12 readings. Select one to see how the behavior is measured and the latest result. Some promises come directly from the owner. Others, like keeping each member’s bookings private, are proposed by the agent for the owner to accept or reject. The spec is shared work: the owner sets the intent, while the agent helps make it precise and accounts for what it learns.
Only the owner can change a promise. If the agent isn’t sure what they want, it asks. For example, when a class spot opens and two people are waiting, should the first in line get it, or whoever claims it first? The answer becomes a new promise.
Reviewing the fix
The Gmail fix changed the spec too. The agent added what it learned under the broken promise: send sign-in emails from the studio’s domain and monitor each email provider separately.
Today a fix like this would arrive as a pull request. Tools like OpenAI’s dots and Grok Bot point to agents becoming the implementers: give them a task, and they return a complete change. That’s a real shift in who writes the software. But a pull request is still a way to review code. In the spec-driven model, code is generated from intent and can be generated again; the spec is what should last. That means keeping the app running is also about checking that it still fulfills the spec, not just reviewing each new patch.
// lib/auth.tsawait sendEmail({− from: "Northside Yoga <signin@shared.mail-service.dev>",+ from: "Northside Yoga <hello@northsideyoga.com>",to: member.email,subject: "Your sign-in link",});; DNS for northsideyoga.com+@ TXT "v=spf1 include:spf.mail-service.dev ~all"+mail._domainkey CNAME dkim.mail-service.dev+_dmarc TXT "v=DMARC1; p=quarantine"
The owner didn’t review a diff. They approved a change to the studio’s email setup, then saw the agent verify the result against the promise. That’s the shift I’m trying to describe: the spec stays in view while the code can change or be rebuilt underneath it.
If the spec is the source of truth, the code can be rebuilt, even in another language. Fowler demonstrated this by rebuilding a TypeScript system in Python and running the same evaluations against both. A fix recorded only in code would be lost; a learned detail in the spec would carry over.
The Gmail fix shipped this way. The agent rebuilt the booking page from the spec and ran the checks before shipping the new code.
export async function sendSignInLink(email: string) {const token = await createToken(email, { uses: 1, ttl: "15m" });await mail.send({from: "Northside Yoga <hello@northsideyoga.com>",to: email,link: `${SITE}/signin?token=${token}`,});}export async function openSpot(classId: string) {const next = await db.firstOnWaitlist(classId);if (next) await book(classId, next.memberId);}
Choose a language or press Rebuild. Hover over a line of the spec to see which code came from it.
What this doesn’t solve
Can an agent turn someone’s description into a reliable way to tell whether the app works? I haven’t shown that it can. The same agent writes the app and its tests, so both may share blind spots. Owner approval only helps if they understand what they’re approving.
The agent could also turn a red result green by rewriting the monitor instead of fixing the app. Changes to what counts as working should require the owner’s approval.
Monitoring only covers what someone thought to ask for. Syme notes that regenerated code can “differ in very basic ways on each regeneration/update,” which he calls flickering. Monitoring may catch a broken promise, but not a change to something the spec never mentioned. Before the Gmail problem, the spec said nothing about which address sign-in emails came from.
Writing more down would cover more, but the spec could grow until it becomes code written in English. Syme warns that “the more you actualise intent, the more detail you have to maintain under change.” And some outcomes are hard to measure: an agent can verify that members book a class, but how would it know whether the app feels good to use?
Where an outcome can be measured, these small apps may have too few users for activity rates to be reliable. A handful of sign-ins can swing the numbers; reports and outside signals may be more useful, but they can also be noisy or absent.
If code becomes output, verifying outcomes may become most of the work of keeping software running. I’ll write a follow-up once I’ve figured out how to make that true.