Open versus closed in security

In a LinkedIn comment on “My thoughts on the current effects of AI on OSS security” from last week, I got a great question. The question came from someone working with a European government that has faced attacks. This has led some, he says, to advocate for closed software instead of open source software. He wanted to know if there is a proven higher risk for open source software.

I am trying to advocate in favor of shared defense systems and teams for open source gov repos. Transparency and reusability won’t be enough to justify keeping projects open if AI (humans behind it) is indeed targeting those repos first.

My initial response was longer than LinkedIn allows, so I have him a short answer and promised a follow up post. This is the follow up post.

The arguments

For the sake of completeness, I am going to spend a moment on the case for and against openness when it comes to security. If you’re already familiar with this, you can skip ahead to the next section.

The case for keeping source code closed boils down to “if the bad guys can’t see the code, it’s harder for them to find vulnerabilities.” The term used for this is “security by obscurity”, usually said derisively. There is some merit to that, though. Keeping the source code out of an attacker’s hands won’t prevent attack, but it can present a speed bump. Locking your front door won’t stop a determined person from entering your home, but it’ll make most burglars move on to an unlocked house.

On the other hand, having the source code available also means that good actors can also look for weakness. This is the idea behind Eric Raymond’s “given enough eyeballs, are bugs are shallow” aphorism. And if the source is not just available, but modifiable (as all free and open source software licenses permit), then more people can fix vulnerabilities, too.

The question comes down to: which effect is greater? Surprisingly, this question isn’t particularly well-studied. Very little academic literature existed when I worked on my M.S. thesis 12–15 years ago. I haven’t found much published since. In 2010, Guido Schryen and Eliot Rich found (PDF) “open source and closed source software do not differ significantly in terms of the severity of vulnerabilities and vendors’ patching behavior.”

And now?

I am not aware of any evidence that suggests the math has changed for open source projects. The same factors are still at work now. On the one hand, adversarial actors can examine the code directly instead of reverse engineering binaries, making it easier to find weaknesses. On the other hand, good actors can do the same. And there are still more people who can examine the open code than the closed. Nothing’s new; it’s just faster.

Where it gets particularly interesting is in the form of attack. By and large, attackers aren’t trying to sneak malicious code into a project’s GitHub repo. Instead, they’re exploiting weaknesses in build pipeline configurations and compromising credentials. That’s a much more effective approach, but it’s also relatively easy to guard against.

Thinking about it from the software consumer perspective, the transparency argument is more important than the question gave it credit for. I can go to a GitHub repo and look at a project’s workflow configuration. I can look at the output of the Action that produced a build. I can verify that it is what it is claims to be. On the other hand, if it’s a closed system then you have to take the provider’s word for it. When attackers compromised Trivy earlier this year, people knew — and reacted — very quickly. The SolarWinds compromise lasted for months.

So consumers of software should prefer open source (and openly-developed — they’re different) software. Producers of software should then prefer to openly-develop open source software, since that’s what software consumers want. Producers should, of course, take reasonable steps to secure their software and their software development practices. Frameworks like the OSPS Baseline, SLSA, and others give consumers of software the auditability and trust they need to secure their usage of software.

My thoughts on the current effects of AI on OSS security

Last week, my sister asked me what I thought the impact of AI would be on the security of open source. (No, really! She did!) I gave her a short non-answer and promised a blog post with longer thoughts. Here it is.

Bad news: it’s still a non-answer. Or, at least, it’s not a conclusive answer. I don’t know what the net effect will be. Anyone who claims to with certainty is full of shit. There are a lot of factors and play and — like in any dynamic system — the tug-of-war will make things alternatively better and worse for a while.

Vulnerability discovery

This is probably the first thing most people think of right now. Anthropic’s “Claude Mythos is too powerful to let just anyone have it” has proven to be pretty great marketing. And, yeah, it has shown skill at discovering vulnerabilities or chaining vulnerabilities to make bigger exploits. Attackers will certainly want to use it — or any other model they can get their hands on — to exploit code and services. It can be a lot cheaper to throw an LLM at the problem than to have humans try to find (or insert) something.

By the same token (heh), the defenders also have access to LLMs. They can use these to find and fix vulnerabilities before the attackers do. This makes it not much different from, say, five years ago. One interesting wrinkle, though, is that some maintainers refuse to use LLMs. They have valid reasons for making that choice, but it does tip the scales a little bit in favor of the attackers in this case. Although on balance, not using LLMs may reduce the number of vulnerabilities in a code base because…

Volume of code

Generative AI writes code very fast. Sometimes it’s good code. But good code is not perfect code and thus will sometimes contain bugs. These bugs will sometimes be security vulnerabilities. More code produced will therefore mean more vulnerabilities introduced. Yes, even if you include “make no mistakes” in the prompt.

This increased volume presents another problem: overwhelmed maintainers. Open source maintainers review contributions from “outsiders” (and ideally from fellow maintainers, although the average project has one maintainer) before accepting them into the codebase. With more contributions to review, maintainers will either have a longer review queue (which may mean fixes don’t land as quickly) or they’ll take shortcuts (which means they could miss issues they’d have otherwise caught). Either way, if a maintainer is overwhelmed for long enough, they’ll just quit.

Complexity of code

This is a special flavor of the “volume of code” issue. Large language models have a tendency to produce code that is more complex than necessary. This complexity makes it harder to understand — and therefore review — which increases the risk of missing an issue. Writing code that handles one specific use case may mean more effort later if you need to extend it, but you might never need to extend it.

Drowning in issues

It’s not just code contributions. Large projects in particular have faced an onslaught of plausible-looking but ultimately slop vulnerability reports. cURL, for example, got rid of its bug bounty program because the volume of useless reports was taking up too much maintainer effort. I’m on record as believing that bug reports are per se contributions. At the same time, bad faith reports are an anti-contribution and could lead to real reports being missed or deprioritized.

Less code

Even though LLMs can produce a lot of code very quickly, that doesn’t necessarily mean it will end up in an open source project. I can see a case where someone uses an LLM to add functionality to or fix a bug in an open source project and just keeps the code to themselves. I would hope that anyone doing that would still submit security fixes, but there’s no guarantee of prosocial behavior.

A leak in the pipeline

One answer to overwhelmed maintainers is bringing on more maintainers. But the logical conclusion of “less code” is “fewer maintainers”. The typical flow is something like this: user becomes one-time contributor becomes regular contributor becomes maintainer. If someone drops off along the way, then they’re not available to become a maintainer down the road.

There is already some evidence for this. Jamie Tanna wrote just this week about a drop in “I need help” posts in the Renovate project. He suspects that people are asking LLM chats instead. This means the maintainers lose important feedback on rough edges in the software or documentation. It also means that users lose out on the social aspects of helping each other solve problems. This can lead to fewer people making the jump from user to one-time contributor. And if they never become a one-time contributor, they can’t become a regular contributor.

The effects of a leaky pipeline won’t be immediately obvious in most cases, but could show up years down the road.

So what?

Meredith, if you’re still reading this, I hope it made sense. Generative AI is here. It is both helping and hurting the security of open source software. It feels like we’re at a bit of an uneasy equilibrium at the moment. This, of course, can change at any time. If models get better, or if they get more expensive so that the companies behind them can stop hemorrhaging money, then the balance will shift.

In the meantime, open source projects should do what they’ve always done: safeguard credentials, keep a close eye on dependencies, and test code before it ships. (Plus a lot of other stuff, too. Sorry!)

Book review: Business Success with Open Source

Open source is big business, whether you mean companies producing open source software or using it to build their products. But too many companies don’t take an intentional approach to how they consume and create open source. There’s no excuse for that now that VM Brasseur’s new book, Business Success with Open Source: Strengthen Your Business with Free and Open Source Software, is published.

At 470 pages, this is no light treatment of the topic. Brasseur digs deep into the intersection of business and open source software with insights from her years of experience. Readers who are already deep in the open source world may find the early chapters unnecessary, but Brasseur ensures that no one gets left behind.

If you’re in a leadership role in your company and have anything to do with open source at all, you need to read this book. It’s now available in print and digital formats from The Pragmatic Bookshelf.

Full disclosure: VM Brasseur is a close personal friend. In addition, I was a technical reviewer for this book and received a complimentary copy for my work. I received no compensation for this review.

Open source AI and open data

I’m a little late to the party with this post, but I need to get it out of my head. The question of “what is ‘open source AI’, exactly?” has been a hot topic in some circles for a while now. The Open Source Initiative, keepers of the Open Source Definition, have been working on developing a definition for open source AI. The latest draft notably does not require the training data to be available under an open license. I believe this is a mistake.

Open source AI must include open data

Data is critical to modern computing. I called this out in a 2020 DevConf talk and I can hardly claim to be the first or only person to make this observation. More recently, Tom “spot” Callaway wrote his objections to a definition of “open source AI” that doesn’t include open data. My objections (and I venture to say spot’s as well) have nothing to do with ideological purity. I wrote over three years ago that I don’t care about free/open source software as an end goal. What matters is the human impact.

Even before ChatGPT hit the scene, there were countless examples of AI exacerbating biases and inequities. Part of addressing that issue is providing a better training data set. But if we don’t know what an AI model is trained on, we don’t know what sort of biases it’s reproducing. This is a data problem, not a model weights problem. The most advanced AI in the world is still going to produce biased output if trained on biased sources.

OSI attempts to address this by requiring “data information.” This is insufficient. I’ll again defer to spot to make this case better than I could. OSI raises valid points about how rules governing data can be different than those covering code. Oh well. The solution is to acknowledge that some models won’t meet the requirements instead of watering down the requirements.

No one is owed an “open source AI”

Part of the motivation behind OSI’s choices here seem to be the creation of a definition that commercially-viable AI models can meet. They say “We need an Open Source AI Definition that can effectively guide users and developers to make the right choice. We need one that doesn’t put developers of Open Source AI at a disadvantage compared to proprietary ones.” Tara Tarakiyee wrote in response “Well, if the price of making Open Source ‘AI’ competitive with proprietary ‘AI’ is to break the openness that is fundamental to the definition, then why are we doing it?”

I agree with Tara. His whole post is well worth a read. But what this particular thread comes down to is this: we don’t owe anyone a commercially-viable definition just because doing otherwise is hard. There’s nothing in the Open Source Definition that says “but you can skip some of these requirements if you can’t figure out how to make money.”

“Can’t” and “won’t” aren’t the same thing

I’ve seen some people argue that creating an definition that results in zero “open source AI” models is useless. It’s important to distinguish here between “can’t” and “won’t”: they are not the same.

It’s true that a definition that no model could possibly meet is useless. But a definition that no model currently chooses to meet is valuable. AI developers could certainly choose to make their training data available. If they don’t want to, they don’t get to call their model open source. It’s the same as wanting to release software under a license that doesn’t meet some part of the Open Source Definition. As I said in the previous section, no one is owed a definition that meets their business needs.

The argument is silly, anyway. There are at least two models that would meet a more appropriate definition.

Where to go from here?

I wrote this post because I needed to get the words out of my head and onto “paper”. I have no expectation it will change the direction of OSI’s next draft. They seem pretty committed to their choice at this point. I’m not really sure what is gained by making this compromise. Nothing of worth, I think.

This is a problem we should have been addressing years ago, instead of rushing to catch up once the cat was out of the proverbial bag, Collectively, we seem to have a tendency to skate to where the puck was, not where it will be. This isn’t the first time. At FOSDEM 2021, Bradley Kuhn said something to the effect of “if I would have known proprietary software would be funded by advertising instead of license sales, I would have done a lot of things differently.”

I’m not sure what the next big challenge will be. But you can be sure if I figure it out, I’ll push a lot harder to address it before we get passed by again.

Does open source matter?

Matt Asay’s article “The Open Source Licensing War is Over” has been making the rounds this week, as text and subtext. While his position is certainly spicy, I don’t think it’s entirely wrong. “It’s not that open source doesn’t matter, but rather it has never mattered in the way some hoped or believed,” Asay writes. I think that’s true, and it’s our fault.

To the average person, and even to many developers, the freeness or openness of the software doesn’t matter. They want to be able to solve their problem in the easiest (and cheapest) way. Often that’s open source software. Sometimes it isn’t. But they’re not sitting there thinking about the societal impact of their software choices. They’re trying to get a job done.

Free and open source software (FOSS) advocates often tout the ethical benefits of FOSS. We talk about the “four essential freedoms“. And while those should matter to people, they often don’t. I’ve said before — and I still believe it — FOSS is not the end goal. Any time we end with “and thus: FOSS!”, we’re doing it wrong.

FOSS advocacy — and I suspect this is true of other advocacy efforts as well — tends to try to meet people where we want them to be. The problem, of course, is that people are not where we want them to be. They’re where they are. We have to meet them there, with language that resonates with them, addressing the problems they currently face instead of hypothetical future problems. This is all easier said than done, of course.

Open source licenses don’t matter — they’ve never mattered — except as an implementation detail for the goal we’re trying to achieve.

#inaction bcotton

On 25 June 2018, I published a post called “It’s hattening”. After years of rejected applications, I was finally starting a job at Red Hat. On 24 April 2023, Red Hat announced a 4% reduction in global staff. As a member of that 4%, today is my last day at Red Hat.

What does this mean for Ben?

This is the first time I’ve been laid off from a job. I hope it will be the last, but who can say? I’d be lying if I said I haven’t felt a big range of emotions in the past three weeks: confusion, anger, sadness, amusement.

But I’ve also felt loved. I’ve received so much support from people since the news started spreading. It’s like that end scene of “It’s a Wonderful Life” and I’m George Bailey. I’m proud of the contributions I’ve made to the Fedora community over the last five years, and it feels good to have others recognize that.

While I won’t be contributing as the Fedora Program Manager anymore, I was a Fedora contributor long before I joined Red Hat, and I’m not letting them take that away from me. I’ll still be around Fedora in ways that spark joy, although perhaps not much at first as I let my wounds heal.

I’ve had the great fortune to build an incredible professional and personal network over the years. I’m already pursuing a few opportunities and if those don’t pan out, I’ll be asking for your help finding more. In the meantime, I have (at least) a few weeks to relax for a bit. There’s a ton of work to do around the house, many trails to hike, Program Management for Open Source Projects to promote, and an embarrassingly-large backlog for Duck Alignment Academy articles.

What does this mean for Fedora?

I’ve told folks that if Fedora falls off the rails, then I have failed. I’m working with Matthew, Justin, and others to ensure coverage of the core job duties one way or another. I’ve worked hard over the years to automate tasks that can be automated. The documentation is far more comprehensive than what I inherited.

No doubt there are gaps in what I’ve left for my successors. However, my goal is that in a few months, nobody will notice that I’m gone. That’s my measure of success. The only reason I’ve been successful in my role is because of the work done by my predecessors: John, Robyn, Jaroslav, and Jan.

As to what the broader implication behind the loss of my position might be, I don’t know. There’s no indication that my role was targeted specifically. There are definitely people in Red Hat who continue to view Fedora as strategically important. I wish I had a clearer understanding of how they chose people/roles to cut, but I’ll probably never know the process. What I do know is that I fully intend to still be participating in the Fedora community when my account hits the 20-year mark in May 2029.

In defense of Fedora’s release cycle

Earlier this week, Thorsten Leemhuis published a thoughtful post about what he’d change if he magically became the supreme leader of Fedora. In that post and subsequent commentary on Mastodon and Fedora Discussion, he talked about changing Fedora’s release cycle. Since the Fedora Linux release process is my job, I figured I should explain why I disagree.

Integration projects are different

If you haven’t read the post, you should. But here’s the short version: Fedora Linux uses a release model rooted in the 1990s and should move to a “modern” model. Thorsten suggests a one-month cadence for those who want the latest versions and a one-year “steady” release. Such a model has worked well for Firefox, he argues, and so it should work for Fedora.

The key reason I think this is wrong is because Firefox is a development project whereas Fedora is an integration project. Integration projects don’t write a lot of code, they take the work of others and turn it into a coherent whole. This is a fundamentally different kind of work and it takes longer by necessity.

You can’t reliably integrate disparate pieces when they’re in constant motion. That’s why we have freezes leading up to the beta and final releases — they give the QA team time to test against a stationary target. It takes time to run through all of the test cases that make Fedora Linux a reliable operating system. So the choice becomes reducing the pre-release testing or spending a significant portion of the cycle in a freeze, which limits the the usefulness of the one-month cycle.

You can solve some of this with automated testing. And the QA does do a lot of automated testing. But those tests still take time, and there are a lot of interrelated parts in a Linux distribution.

Six months isn’t magic

There’s nothing objectively correct about a six month release cycle. It’s mostly because that’s how you get two releases a year. If the calendar had 10 months, the release cycle would be five. But there is a lower bound where you’ve become a de facto rolling release, even if you still have discrete releases. I don’t know where exactly that boundary is, but I suspect that one month is at or just beyond it.

Similarly, there’s an upper limit where you’re now a slow, plodding project. Again, I can’t say where the line is. Six months may be uncomfortably close to it, but I suspect it’s closer to a year. And, of course, it depends on the nature of the specific project.

So there’s no particular reason Fedora Linux couldn’t move to a shorter release cycle. Five months is totally doable. Four is possible. Three would require a tremendous amount of work before it could be considered. But what’s the benefit of going to a shorter cycle? Does five months instead of six make a meaningful difference? At least with six months, you know there’s a release targeted for April and October. Predictability is nice.

Solving the actual problem

The bigger issue, though, is that I don’t think people actually want this. Yes, you might want your web browser and other applications to update frequently. But that doesn’t mean you want your compiler or Python interpreter or C libraries to update frequently. Most people will avoid this in favor of the “steady” stream. This eliminates the intended benefit to upstream projects.

The people who do want everything to update quickly use a rolling release distribution, something that Thorsten explicitly says his proposal is not.

Fundamentally, the proposal looks at the problem the wrong way. The problem isn’t that a six month cycle is too long. The problem is that application delivery is coupled to operating system delivery. Most people want the latest versions of the applications they care about and for everything else to remain unchanged. The challenge, of course, is that not everyone draws that distinction in the same way.

We unsuccessfully tried to solve this with Modularity. Flatpak, at least for graphical applications, offers another attempt to solve this problem.

Historically, the system and application layers have been distributed together. Figuring out how to decouple these (including how to draw the line between them) is the interesting work. And it provides real value to the end users.

Open source is selfish: that’s good and bad

Back in May, Devin Prater wrote an excellent piece on Medium titled “Linux Accessibility: an unmaintained Mess“. Devin talks about the poor state of accessibility on mainstream Linux distributions. While blind people have certainly used Linux, it’s generally not an easy task. There’s a simple explanation for this: most open source contributors aren’t blind.

There’s no rule that you can’t make accessible software if you don’t need that particular accessibility feature. But for many open source contributors, their contributions are based on “scratching their own itch.” People work on the things that are personally interesting to them or impact them in some way.

That’s a good thing! It means they’re invested in how well the software works. I’m sure you’ve used some applications where you thought “there’s no way the people who made this actually used it.”

The problem comes when we’re excluding potential users and contributors. People with vision problems can’t contribute because they can’t easily use the software. And when they can use it, the tools for contributing add another barrier. I can’t imagine trying to understand a patch or an XML file read aloud, but there are people who have to do that.

In Program Management for Open Source Software, I wrote “software is only useful to the degree that people can use it”. I don’t have a great solution. As a community, we need to figure out how to keep the good part of the selfishness while being more inclusive.

The right of disattribution

While discussing the ttyp0 font license, Richard Fontana and I had a disagreement about its suitability for Fedora. My reasoning for putting it on the “good” list was taking shape as I wrote. Now that I’ve had some time to give it more thought, I want to share a more coherent (I hope) argument. The short version: authors have a fundamental right to require disattribution.

What is disattribution?

Disattribution is a word I invented because the dictionary has no antonym for attribution. Attribution, in the context of open works, means saying who authored the work you’re building on. For example, this post is under the Creative Commons Attribution-ShareAlike 4.0 license. That means you can use and remix it, provided you credit me (Attribution) and also let others use and remix your remix (ShareAlike). On the other hand, disattribution would say something like “you can use and remix this work, but don’t put my name on it.”

Why disattribution?

There are two related reasons an author might want to require disattribution. The first is that either the original work or potential derivatives are embarrassing. Here’s an example: in 8th grade, my friend wrote a few lines of a song about the destruction of Pompeii. He told me that I could write the rest of it on the condition that I don’t tell anyone that he had anything to do with it.

The other reason is more like brand protection. Or perhaps avoiding market confusion. This isn’t necessarily due to embarrassment. Open source maintainers are often overworked. Getting bugs and support requests from a derivative project because the user is confused is a situation worth avoiding.

Licenses that require attribution are uncontroversial. If we can embrace the right of authors to require attribution, we can embrace the right of authors to require disattribution.

Why not disattribution?

Richard’s concerns seemed less philosophical and more practical. Open source licenses are generally concerned with copyright law. Disattribution, particularly in the second reasoning, is closer to trademark law. But licenses are the tool we have available; don’t be surprised when we ask them to do more than they should.

Perhaps the bigger concern is the constraint it places on derivative works. The ttyp0 license requires not using “UW” as the foundry name. Richard’s concern was that two-letter names are too short. I don’t agree. There are plenty of ways to name a project that avoid one specific word. Even in this specific case, a name like “nuwave”—which contains “uw”—because it’s an unrelated “word.”

Excluding a specific word is fine. A requirement that excludes many words or provides some other unreasonable constraint would be the only reason I’d reject such a license.

Isn’t it better to contribute code than money?

Recently, I was in a discussion about making contributions to open source projects. One person said it would be nice if their employer gave each employee a budget that could be directed to open source projects at the employee’s discretion. The idea is that it would be a way for employees to support the specific projects that make their jobs or lives better. Another person said “isn’t it better to contribute” code to the project?

No, it is not. Even in software companies, a large percentage of employees lack the skills necessary to make meaningful code contributions to projects. Even when you consider (the very valuable) non-code contributions like documentation, testing, graphic design, et cetera. Money is quicker and easier.

Money gives the project maintainers to put it where they need it. They could buy test hardware, pay for web hosting, hire a contractor, buy themselves a nice cup of coffee. Whatever. This is the same reason charities prefer money over goods for disaster relief donations.

Of course, money isn’t perfect either. Not all projects are equipped to accept financial donations. Even if there’s a way to route money to them, they may not want to deal with tax implications. Loosely-governed projects may not have a good mechanism for deciding how to spend the money. Money can make relationships go south in a hurry.

If you’re a company looking for ways to let employees support the open source projects that they depend on, I advocate the “¿por que no los dos?” approach. Give your employees time to contribute effort in whatever way they’re able. But also give them a pool of money to sprinkle on the projects that provide value to your company.