In a LinkedIn comment on “My thoughts on the current effects of AI on OSS security” from last week, I got a great question. The question came from someone working with a European government that has faced attacks. This has led some, he says, to advocate for closed software instead of open source software. He wanted to know if there is a proven higher risk for open source software.

I am trying to advocate in favor of shared defense systems and teams for open source gov repos. Transparency and reusability won’t be enough to justify keeping projects open if AI (humans behind it) is indeed targeting those repos first.

My initial response was longer than LinkedIn allows, so I have him a short answer and promised a follow up post. This is the follow up post.

The arguments

For the sake of completeness, I am going to spend a moment on the case for and against openness when it comes to security. If you’re already familiar with this, you can skip ahead to the next section.

The case for keeping source code closed boils down to “if the bad guys can’t see the code, it’s harder for them to find vulnerabilities.” The term used for this is “security by obscurity”, usually said derisively. There is some merit to that, though. Keeping the source code out of an attacker’s hands won’t prevent attack, but it can present a speed bump. Locking your front door won’t stop a determined person from entering your home, but it’ll make most burglars move on to an unlocked house.

On the other hand, having the source code available also means that good actors can also look for weakness. This is the idea behind Eric Raymond’s “given enough eyeballs, are bugs are shallow” aphorism. And if the source is not just available, but modifiable (as all free and open source software licenses permit), then more people can fix vulnerabilities, too.

The question comes down to: which effect is greater? Surprisingly, this question isn’t particularly well-studied. Very little academic literature existed when I worked on my M.S. thesis 12–15 years ago. I haven’t found much published since. In 2010, Guido Schryen and Eliot Rich found (PDF) “open source and closed source software do not differ significantly in terms of the severity of vulnerabilities and vendors’ patching behavior.”

And now?

I am not aware of any evidence that suggests the math has changed for open source projects. The same factors are still at work now. On the one hand, adversarial actors can examine the code directly instead of reverse engineering binaries, making it easier to find weaknesses. On the other hand, good actors can do the same. And there are still more people who can examine the open code than the closed. Nothing’s new; it’s just faster.

Where it gets particularly interesting is in the form of attack. By and large, attackers aren’t trying to sneak malicious code into a project’s GitHub repo. Instead, they’re exploiting weaknesses in build pipeline configurations and compromising credentials. That’s a much more effective approach, but it’s also relatively easy to guard against.

Thinking about it from the software consumer perspective, the transparency argument is more important than the question gave it credit for. I can go to a GitHub repo and look at a project’s workflow configuration. I can look at the output of the Action that produced a build. I can verify that it is what it is claims to be. On the other hand, if it’s a closed system then you have to take the provider’s word for it. When attackers compromised Trivy earlier this year, people knew — and reacted — very quickly. The SolarWinds compromise lasted for months.

So consumers of software should prefer open source (and openly-developed — they’re different) software. Producers of software should then prefer to openly-develop open source software, since that’s what software consumers want. Producers should, of course, take reasonable steps to secure their software and their software development practices. Frameworks like the OSPS Baseline, SLSA, and others give consumers of software the auditability and trust they need to secure their usage of software.

Last week, my sister asked me what I thought the impact of AI would be on the security of open source. (No, really! She did!) I gave her a short non-answer and promised a blog post with longer thoughts. Here it is.

Bad news: it’s still a non-answer. Or, at least, it’s not a conclusive answer. I don’t know what the net effect will be. Anyone who claims to with certainty is full of shit. There are a lot of factors and play and — like in any dynamic system — the tug-of-war will make things alternatively better and worse for a while.

Vulnerability discovery

This is probably the first thing most people think of right now. Anthropic’s “Claude Mythos is too powerful to let just anyone have it” has proven to be pretty great marketing. And, yeah, it has shown skill at discovering vulnerabilities or chaining vulnerabilities to make bigger exploits. Attackers will certainly want to use it — or any other model they can get their hands on — to exploit code and services. It can be a lot cheaper to throw an LLM at the problem than to have humans try to find (or insert) something.

By the same token (heh), the defenders also have access to LLMs. They can use these to find and fix vulnerabilities before the attackers do. This makes it not much different from, say, five years ago. One interesting wrinkle, though, is that some maintainers refuse to use LLMs. They have valid reasons for making that choice, but it does tip the scales a little bit in favor of the attackers in this case. Although on balance, not using LLMs may reduce the number of vulnerabilities in a code base because…

Volume of code

Generative AI writes code very fast. Sometimes it’s good code. But good code is not perfect code and thus will sometimes contain bugs. These bugs will sometimes be security vulnerabilities. More code produced will therefore mean more vulnerabilities introduced. Yes, even if you include “make no mistakes” in the prompt.

This increased volume presents another problem: overwhelmed maintainers. Open source maintainers review contributions from “outsiders” (and ideally from fellow maintainers, although the average project has one maintainer) before accepting them into the codebase. With more contributions to review, maintainers will either have a longer review queue (which may mean fixes don’t land as quickly) or they’ll take shortcuts (which means they could miss issues they’d have otherwise caught). Either way, if a maintainer is overwhelmed for long enough, they’ll just quit.

Complexity of code

This is a special flavor of the “volume of code” issue. Large language models have a tendency to produce code that is more complex than necessary. This complexity makes it harder to understand — and therefore review — which increases the risk of missing an issue. Writing code that handles one specific use case may mean more effort later if you need to extend it, but you might never need to extend it.

Drowning in issues

It’s not just code contributions. Large projects in particular have faced an onslaught of plausible-looking but ultimately slop vulnerability reports. cURL, for example, got rid of its bug bounty program because the volume of useless reports was taking up too much maintainer effort. I’m on record as believing that bug reports are per se contributions. At the same time, bad faith reports are an anti-contribution and could lead to real reports being missed or deprioritized.

Less code

Even though LLMs can produce a lot of code very quickly, that doesn’t necessarily mean it will end up in an open source project. I can see a case where someone uses an LLM to add functionality to or fix a bug in an open source project and just keeps the code to themselves. I would hope that anyone doing that would still submit security fixes, but there’s no guarantee of prosocial behavior.

A leak in the pipeline

One answer to overwhelmed maintainers is bringing on more maintainers. But the logical conclusion of “less code” is “fewer maintainers”. The typical flow is something like this: user becomes one-time contributor becomes regular contributor becomes maintainer. If someone drops off along the way, then they’re not available to become a maintainer down the road.

There is already some evidence for this. Jamie Tanna wrote just this week about a drop in “I need help” posts in the Renovate project. He suspects that people are asking LLM chats instead. This means the maintainers lose important feedback on rough edges in the software or documentation. It also means that users lose out on the social aspects of helping each other solve problems. This can lead to fewer people making the jump from user to one-time contributor. And if they never become a one-time contributor, they can’t become a regular contributor.

The effects of a leaky pipeline won’t be immediately obvious in most cases, but could show up years down the road.

So what?

Meredith, if you’re still reading this, I hope it made sense. Generative AI is here. It is both helping and hurting the security of open source software. It feels like we’re at a bit of an uneasy equilibrium at the moment. This, of course, can change at any time. If models get better, or if they get more expensive so that the companies behind them can stop hemorrhaging money, then the balance will shift.

In the meantime, open source projects should do what they’ve always done: safeguard credentials, keep a close eye on dependencies, and test code before it ships. (Plus a lot of other stuff, too. Sorry!)