LISA wants you: submit your proposal today

I have the great honor of being on the organizing committee for the LISA conference this year. If you’ve followed me for a while, you know how much I enjoy LISA. It’s a great conference for anyone with a professional interest in sysadmin/DevOps/SRE. This year’s LISA is being held in Nashville, Tennessee, and the committee wants your submission.

As in years past, LISA content is focused on three tracks: architecture, culture, and engineering. There’s great technical content (one year I learned about Linux filesystem tuning from the guy who maintains the ext filesystems), but there’s also great non-technical content. The latter is a feature more conferences need to adopt.

I’d love to see you submit a talk or tutorial about how you solve the everyday (and not-so-everyday) problems in your job. Do you use containers? Databases? Microservices? Cloud? Whatever you do, there’s a space for your proposal.

Submit your talk to https://www.usenix.org/conference/lisa18/call-for-participation by 11:59 PM Pacific on Thursday, May 24. Or talk one of your coworkers into it. Better yet, do both! LISA can only remain a great conference with your participation.

sudo is not as bad as Linux Journal would have you believe

Fear, uncertainty, and doubt (FUD) is often used to undercut the use of open source solutions, particularly in enterprise settings. And the arguments are sometimes valid, but that’s not a requirement. So long as you make open source seem risky, it’s easier to push your solution.

I was really disappointed to see Linux Journal run a FUD article as sponsored content recently. I don’t begrudge them for running sponsored content generally. They clearly label it and it takes money to run a website. Linux Journal pays writers and that money has to come from somewhere. But this particular article was tragic.

Chad Erbe uses “Four Hidden Costs and Risks of Sudo Can Lead to Cybersecurity Risks and Compliance Problems on Unix and Linux Servers” to sow FUD far and wide. sudo, if you’re not familiar with it, is a Unix command that allows authorized users to run authorized commands with elevated privileges. The most common use case is to allow administrators to run commands as the root user, but it can also be used to give, for example, webmasters the ability to restart the web server without giving them full access.

So what’s wrong with this article?

Administrative costs

Erbe argues that using sudo adds administrative overhead because you have to maintain the configuration file. It’s 2017: if you’re not using configuration management already then you’re probably a lost cause. You’re not adding a whole new layer, you’re adding one more file to the dozens (or more) you’re coordinating across your environment.

Erbe sets up a “most complicated setup” strawman and knocks it down by saying commercial solutions could help. He doesn’t say how, though, and there’s a reason for that: the concerns he raises apply to any technology that provides the solution. I have seen sites that use commercial solutions to replace sudo, and they still have to configure which users are authorized to use which commands on which servers.

Forensics and audit risks

sudo doesn’t have a key logger or log chain of custody. That’s true, but that doesn’t mean it’s the wild west. Erbe says configuration management systems can repair modified configuration files, but with a delay. That’s true, but tools like Tripwire are designed to catch these very cases. And authentication/authorization logs can be forwarded to a centralized log server. That’s probably something sysadmins should have set up already.

sudo provides a better level of audit logging compared to switching to the root account. It logs every command run and who runs it. Putting a key logger in it would provide no additional benefit. The applications launched with sudo (or the operating system itself) would need it.

Business continuity risks

You can’t rollback sudo and you can’t get support. Except that you can, in fact, downgrade the sudo version if it contains a critical bug. And you can get commercial support. Not for sudo specifically, but for your Linux installs generally.

Lack of enterprise support

This sems like a repeat of the last point, with a different focus. There’s no SLA for fixing bugs in sudo, but that doesn’t mean it’s inherently less secure. How many products developed by large commercial vendors have security issues discovered years later? A given package being open source does not imply that it is more or less secure than a proprietary counterpart, only that its source code is available.

A better title for this artice

Erbe raises some good points, but loses them in the FUD. This article would be much better titled “why authorization management is hard”. That approach, followed by “and here’s how my proprietary solution addresses those difficulties” would be a very interesting article. Instead, all we get is someone paying to knock down some poorly-constructed strawmen. The fact that it appears in Linux Journal gives it a false sense of credibility and that’s what makes it dangerous.

DevOps is dead!

“$thing is dead!” is one of the more annoying memes in the world of technology. A Tech Crunch article back in April claimed that managed services (of cloud infrastructure) is the death knell of DevOps. I dislike the article for a variety of reasons, but I want to focus on why the core argument is bunk. Simply put: “the cloud” is not synonymous with “magical pixie dust.” Real hardware and software still exist in order to run these services.

Amazon Web Services (AWS) is the current undisputed leader in the infrastructure-as-a-service (IaaS) space. AWS probably underlies many of the services you use on a daily basis: Slack and Netflix are two prime examples. AWS offers dozens of services for computation, storage, and networking that roll out updates to datacenters across the globe many times a day. DevOps practices are what make that possible.

Oh, but the cloud means you don’t need your internal DevOps team! No. Shut up. “Why not simply teach all developers how to utilize the infrastructure tools in the cloud?” Because being able to spin up servers and being able to effectively manage and scale them are two entirely different concepts. It is true that cloud services can (not “must”!) take the “Ops” out of “DevOps” for development environments. But just as having access to WebMD doesn’t mean I’m going to perform my own surgery, being able to spin up resources doesn’t obviate the need for experienced management.

The author spoke of “managed services provider” as an apparent synonym for “IaaS provider”. He ignored what I think of as “managed services” which is a contracted team to manage a service for you. That’s what I believe to be the more realistic threat to internal DevOps teams. But it’s no different than any other outsourcing effort, and outsourcing is hardly a new concept.

At the end of the article, the author finally gets around to admitting that DevOps is a cultural paradigm, not a position or a particular brand of fairy dust. Cloud services don’t threaten DevOps, they make it easier than ever to practice. Anyone trying to convince you that DevOps is dead is just trying to get you to read their crappy article (yes, I have a well-tuned sense of irony, why do you ask?).

December Opensource.com articles

Here are the articles I wrote for Opensource.com in December:

SysAdvent 2015

I contributed to the SysAdvent blog this year, again as an editor. I had the privilege of working with three great authors on outstanding posts:

Once again, the content overall is great. I liked the mix of technical and non-technical content. In the eight years of SysAvent, many wonderful articles have been written, but the best article may be this year’s Fear and Loathing in Systems Administration by H. “Waldo” Grunenwald. It should be required reading for every sysadmin.

Book review: AWS System Administration

In his forthcoming book, Mike Ryan aims to introduce Amazon Web Services (AWS) to developers and systems administrators. Correctly creating and managing an AWS environment is a cross between development and administration, so anyone coming from a straight admin or dev background would probably miss key components.

Unfortunately, in aiming for two audiences, he produces a book that doesn’t seem to quite satisfy either. The book goes into a lot of unnecessary detail, for example a lot of Postgresql detail in the backup chapter, and a lot of Puppet specifics scattered throughout.

My biggest complaint is the way the book is organized. Basic AWS concepts like regions aren’t introduced in the beginning. Several concepts appear in passing before they are explained. EC2 security groups are lumped into the chapter at IAM roles, but it makes more sense to separate those.

Much of the book focuses on a single example, without a lot of discussion of other use cases. However, the use of auto scaling and Elastic Load Balancers in various cases is very well explained. The use and limitations of IAM roles is excellent as well.

This book could benefit from some reorganization and a more focused audience. With more information about AWS and less on implementation details for specific environments, the second edition could be a valuable resource.

AWS System Administration is scheduled to be released on July 25. It is published by O’Reilly Media.

Unlimited vacation policies, burnout, etc.

Recently, my company switched from a traditional vacation model to a minimum vacation model. If you’re unfamiliar with the term, it’s essentially the unlimited vacation model practiced by Netflix and others, with the additional requirement of taking a defined minimum of time off each year. It’s been a bit of an adjustment for me, since I’m used to the traditional model. Vacation was something to be carefully rationed (although at my previous employer, I tended to over-ration). Now it’s simply a matter of making sure my work is getting done and that there’s someone to cover for me when I’m out.

I’m writing this at 41,000 feet on my way to present at a conference [ed note: it is being published the day after it was written]. I’m secretly glad that the WiFi apparently does not work over the open ocean (I presume due to political/regulatory reasons). Now, don’t get me wrong, one of my favorite things to do when I fly is to watch myself on FlightAware, but in this case it’s a blessing to be disconnected. If a WiFi connection were available, it would be much harder to avoid checking my work email.

It took me a year and a half at my job before I convinced myself to turn off email sync after hours. Even though I rarely worked on emails that came in after hours, I felt like it was important that I know what was going on. After several weekends of work due to various projects, I’d had enough. The mental strain became too much. At first, I’d still manually check my mail a time of two, but now I don’t even do that much.

This is due in part to the fact that the main project that was keeping me busy has had most of the kinks worked out and is working pretty well. It also helps that there’s another vendor managing the operations, so I only get brought in when there’s an issue with software we support. Still, there are several customers where I’m the main point of contact, and the idea of being away for a week fills me with a sense of “oh god, what will I come back to on Monday?”

i’ve written before about burnout, but I thought it might be time to revisit the topic. When I wrote previously, I was outgrowing my first professional role. In the years since, burnout has taken a new form for me. Since I wrote the last post, two kids have come into my life. In addition, I’ve gone from a slow-paced academic environment to a small private sector company which claims several Fortune 100 companies as clients. Life is different now, and my perception of burnout has changed.

I don’t necessarily mind working long hours on interesting problems. There are still days when it’s hard to put my pencil down and go home (metaphorically, since I work from a spare bedroom in our house). But now that I have have kids, I’ve come to realize that when I used to feel burnt out, I was really feeling bored. Burnout is more represented by the impact on my family life.

I know I need to take time off, even if it’s just to sit around the house with my family. It’s just hard to do knowing that I’m the first — and sometimes last — line of support. But I’m adjusting (slowly), and I’m part of a great team, so that helps. Maybe one of these days, I’ll be able to check my email at the beginning of the work day without bracing myself.

Being a good troubleshooter

Not too long ago, my friend Andy said “I think that’s how i convince so many people I’m decent at being a sysadmin, I just look at logs.” It was a statement that really struck a chord with me. On the one hand, I feel like I’ve had a pretty successful career so far, and if I haven’t been excellent, I’ve at least been sufficiently competent. On the other hand, I don’t feel like I have a lot of the experience that “everyone” has. I’ve never run email servers, I’ve only done trivial DNS and DHCP setups, and so on.

In my current job, I work with some really smart people, few of whom have much if any sysadmin experience. Andy and I, because of our sysadmin background, have become the go-to guys for sysadmin questions. It’s a role I enjoy, particularly when I’m able to solve a problem. Sometimes it’s because I have direct experience with the issue, but more often than not, it’s because I know how to poke around until I find the answer.

There are many skills required for successful systems administration. Troubleshooting is high on the list. The ability to troubleshoot new and unusual problems is particularly helpful. I’ve found my own troubleshooting abilities to depend on the ability to build off of previous (if unrelated) experience and being able to piece together disjointed clues. It’s sort of like being a detective and also a human “big data” system.

Fishing for clues in log files is great, too, if you can find the needle in the haystack. But what seems to separate the good troubleshooters from the bad that I’ve known is intellectual curiosity. Not just asking what happened, but finding out why. Being willing to ask questions and learn about unfamiliar areas instead of immediately deferring to those more knowledgeable builds a strong skill base.

Of course, without Google, we’d all be out of luck.

Another great SysAdvent

Once again, a group of volunteer writers and editors came together to put together 25 posts related to systems administration for the SysAdvent blog. Although I have contributed several articles over the years, I much prefer editing. All of this year’s posts are great, but I’m very proud of the posts that I had a hand in editing. As usual, the writers did most of the work, my suggestions were always minor.