Last week, my sister asked me what I thought the impact of AI would be on the security of open source. (No, really! She did!) I gave her a short non-answer and promised a blog post with longer thoughts. Here it is.
Bad news: it’s still a non-answer. Or, at least, it’s not a conclusive answer. I don’t know what the net effect will be. Anyone who claims to with certainty is full of shit. There are a lot of factors and play and — like in any dynamic system — the tug-of-war will make things alternatively better and worse for a while.
Vulnerability discovery
This is probably the first thing most people think of right now. Anthropic’s “Claude Mythos is too powerful to let just anyone have it” has proven to be pretty great marketing. And, yeah, it has shown skill at discovering vulnerabilities or chaining vulnerabilities to make bigger exploits. Attackers will certainly want to use it — or any other model they can get their hands on — to exploit code and services. It can be a lot cheaper to throw an LLM at the problem than to have humans try to find (or insert) something.
By the same token (heh), the defenders also have access to LLMs. They can use these to find and fix vulnerabilities before the attackers do. This makes it not much different from, say, five years ago. One interesting wrinkle, though, is that some maintainers refuse to use LLMs. They have valid reasons for making that choice, but it does tip the scales a little bit in favor of the attackers in this case. Although on balance, not using LLMs may reduce the number of vulnerabilities in a code base because…
Volume of code
Generative AI writes code very fast. Sometimes it’s good code. But good code is not perfect code and thus will sometimes contain bugs. These bugs will sometimes be security vulnerabilities. More code produced will therefore mean more vulnerabilities introduced. Yes, even if you include “make no mistakes” in the prompt.
This increased volume presents another problem: overwhelmed maintainers. Open source maintainers review contributions from “outsiders” (and ideally from fellow maintainers, although the average project has one maintainer) before accepting them into the codebase. With more contributions to review, maintainers will either have a longer review queue (which may mean fixes don’t land as quickly) or they’ll take shortcuts (which means they could miss issues they’d have otherwise caught). Either way, if a maintainer is overwhelmed for long enough, they’ll just quit.
Complexity of code
This is a special flavor of the “volume of code” issue. Large language models have a tendency to produce code that is more complex than necessary. This complexity makes it harder to understand — and therefore review — which increases the risk of missing an issue. Writing code that handles one specific use case may mean more effort later if you need to extend it, but you might never need to extend it.
Drowning in issues
It’s not just code contributions. Large projects in particular have faced an onslaught of plausible-looking but ultimately slop vulnerability reports. cURL, for example, got rid of its bug bounty program because the volume of useless reports was taking up too much maintainer effort. I’m on record as believing that bug reports are per se contributions. At the same time, bad faith reports are an anti-contribution and could lead to real reports being missed or deprioritized.
Less code
Even though LLMs can produce a lot of code very quickly, that doesn’t necessarily mean it will end up in an open source project. I can see a case where someone uses an LLM to add functionality to or fix a bug in an open source project and just keeps the code to themselves. I would hope that anyone doing that would still submit security fixes, but there’s no guarantee of prosocial behavior.
A leak in the pipeline
One answer to overwhelmed maintainers is bringing on more maintainers. But the logical conclusion of “less code” is “fewer maintainers”. The typical flow is something like this: user becomes one-time contributor becomes regular contributor becomes maintainer. If someone drops off along the way, then they’re not available to become a maintainer down the road.
There is already some evidence for this. Jamie Tanna wrote just this week about a drop in “I need help” posts in the Renovate project. He suspects that people are asking LLM chats instead. This means the maintainers lose important feedback on rough edges in the software or documentation. It also means that users lose out on the social aspects of helping each other solve problems. This can lead to fewer people making the jump from user to one-time contributor. And if they never become a one-time contributor, they can’t become a regular contributor.
The effects of a leaky pipeline won’t be immediately obvious in most cases, but could show up years down the road.
So what?
Meredith, if you’re still reading this, I hope it made sense. Generative AI is here. It is both helping and hurting the security of open source software. It feels like we’re at a bit of an uneasy equilibrium at the moment. This, of course, can change at any time. If models get better, or if they get more expensive so that the companies behind them can stop hemorrhaging money, then the balance will shift.
In the meantime, open source projects should do what they’ve always done: safeguard credentials, keep a close eye on dependencies, and test code before it ships. (Plus a lot of other stuff, too. Sorry!)
