Edd Brisley
Published on 28 September 2026
An AI agent sent us a fix for a problem we didn't have yet
We can build more software than ever with AI. This is how we're making sure we can still look after all of it.
A few days ago, an AI agent opened a pull request for one of our apps.
Nothing had broken and no customer had reported a problem. In fact, the change that could cause the problem was still months away from impacting the app.
So what had happened?
The agent had spotted a change in Atlassian's developer documentation, worked out that the upcoming change would eventually break our app, Protected Custom Fields for Jira, found the relevant code that we'd need to update, and proposed a fix in the pull request.
The agent's pull request may be the answer to a problem that every team will need to address as AI tooling helps us build more, faster. Who’s going to look after everything we build?
Every app is a subscription you sell to yourself
AI is great at building apps now. But the big problem with building things is that you've got to maintain them.
With frontier models in the loop, the first version of your new app might take just days to create. But every app you release depends on platforms, APIs, and third-party services. That means releasing an app into the world comes with a hidden subscription, payable in the form of your future attention and ongoing maintenance.
Our R&D team has felt that pain directly. My team has three engineers responsible for developing new products, but we're also responsible for maintaining a portfolio of established apps. We've had sprints where a new product is progressing brilliantly, only for an unexpected change from a provider to pull us straight back into maintaining an existing app.
This is part of the deal when you release software into the world, but for an R&D team like ours, it's particularly painful. What we really want to do is develop new, interesting software in interesting ways, with the confidence that our existing products, and the customers using them, are being well looked after.
By the time something breaks, you’re already too late
What makes the software maintenance problem so frustrating is that many of the fires we have to put out are announced well in advance. Atlassian, for example, might publish a deprecation notice months before switching a feature off.
The problem is that in most small teams, nobody's sole job is to read every changelog for every dependency of every app, every day, then cross-reference those changes against every codebase in your company.
So teams often discover these changes through a failing request in the logs or a confused customer in a support ticket. Then it's an investigation: talk to the customer, work out what broke and why, create a fix, test it... A small, announced change leads to an unplanned day of frantic firefighting, months after it could have been handled calmly, had you known about it.
My colleague and our Head of Engineering, Paul, had been circling this problem for a long time. His observation was the one that stuck: everyone is excited about how fast AI can make apps (us included!), but nobody is talking enough about who looks after them all.
So we built an agent to look after our software for us: our changelog agent watches the dependencies our apps rely on, spots breaking changes before they impact us, and suggests how and when to fix them.
What the changelog agent does
Every morning, on a schedule, the changelog agent checks whether anything our apps rely on, such as libraries, dependencies, or platform APIs, has changed since the last check.
When the agent finds a breaking change or a deprecation notice, the next question is: does this actually affect us? One of our apps might use the affected feature; the others might not. The agent scans our app's codebase in the same way an engineer would, except it does it every day, for every app, without being asked.
If the change is relevant, the agent proposes a fix:
Upstream changelog → change detected → repository relevance check → proposed fix → human review.
The last step only works if reviewing the agent's proposal requires less work than finding and fixing these problems ourselves. Otherwise, we've just built ourselves an agent that generates noise and distracts us from more important things.
The agent needs context the code alone can't provide
When a new engineer joins the team, you don't just give them access to your codebase. You spend time telling them what matters and why it matters. The agent needs the same kind of induction.
For the agent's judgements to be any good, it needs to know something no codebase can tell it: what we actually care about.
So, before the changelog agent looks after one of our apps, it interviews you:
- What is this app?
- Who uses it?
- What key things does it depend on?
- How much risk can you tolerate?
- How closely do you want different areas of the codebase monitored?
The answers are saved in a markdown file that the agent reads before getting to work every day.
This matters because not every app deserves the same level of vigilance. A widely used customer-facing app warrants close monitoring. On the other hand, a small internal tool with a few users needs less attention.
You can and should set the level of scrutiny for each app to match the risk of things going wrong in that app.
Back to that pull request
So what happened with the pull request from the introduction?
The changelog agent's claim was specific: an upcoming change on Atlassian's side meant that a simple but important check that our app performs—verifying that a user is who they say they are—would eventually start failing for some customers. The agent found where that check happens in our codebase and proposed a clean, focused patch.
Reading it, my first reaction was: interesting. My second was: is that actually true?
AI can sound more confident than the evidence supports, so the question needs asking. And the main difference now is that I actually have time to ask it.
The agent has given me a specific claim about a future problem, with a recommended fix attached. Because the change won't affect our app for a couple of months, I also have time to deal with it properly.
I'll do what I would do with any pull request from any engineer: verify before trusting. I'll read the documentation, check the proposed fix against our codebase, and decide whether to merge it.
At the time of writing, that verification is next on my list. But the good thing is, there's no last-minute emergency forcing my hand, and our sprint doesn't need to get derailed.
Where is the ROI in all of this?
Our agent is already running against our real repositories and has already opened pull requests in response to upstream changes. But has it saved us anything? The key thing in software engineering isn't how much time something took today, but how much time it saves you in the future. So, the honest answer depends on when you count.
Reviewing a proposed fix from the agent costs us a small amount of time today, but the return arrives later in the form of an incident that never happens.
The alternative to that is discovering the change through a failure that impacts customers, then our team racing through investigation, customer communication, and rushing out a quick fix. That costs far more in time and attention, as well as putting our reputation on the line.
One avoided incident might save a few hours. Across an entire portfolio of software, those savings add up: fewer disrupted sprints, fewer surprises in our support inbox, and more time to create value for customers in the form of improving existing apps, or shipping new ones.
Abundance needs owners
AI is making it easier, faster, and cheaper than ever to build software. But every time you release an app, it's a promise to keep it working.
So if you're shipping more software than ever but looking at your portfolio and starting to wonder, 'Who's actually keeping track of the APIs and dependencies underneath it all?', that's exactly the problem we built this agent for.
If AI means we're going to build more software than ever before, we'll need to get much better at looking after it. The same technology that helped us build all this software can help us keep it running.
How fast should you go?
It's worth asking before you speed anything up. Read more of our thinking on building well with AI.