Apache recently held a week-end “infrathon” to sweep the dust out of the corners, squash a few old bugs, drink a wee bit of ale (and maybe a wee bit more than a wee bit) and get their hands dirty with the Zen of maintaining their infrastructure. MadPenguin.org crashed the party in search of the secrets of getting into the “zone” while peeking into the grittiest of the nitty gritty of one of the darling projects of open source, Apache.
There's a lot of talk about “community” in the open source world, and learned papers by the hundreds are chasing the mystery of why highly talented hackers give away code for free. But for those of use who don't hack code, maybe we just have to try to sit and stare at a blank wall for an hour or so to experience the “Zen” of something as grinding and nitty gritty as infrastructure maintenance.
Maybe it's sort of like the tedious poses that yoga practitioners do; or the mind-numbing, repetitive training that star athletes engage in; or the hours of relentless practice of top violinists slashing away at their violins. Maybe true beauty can't be bought by the billions of dollars stashed away in Microsoft's banks. Maybe the only way to “get” the Zen of open source community is to hunker down and grind out deadly dull infrastructure maintenance work. Maybe you actually gotta “feel the burn” to get the high.
There were no corporate press releases for this event. No splashy media ads or glossy print magazine ads. The event did not take place in a big, loud conference center such as San Francisco's spacious Moscone Center. There was no hoard of gawkers hoping to catch sight of someone famous. There were no booth babes or flashy booth displays or big deal keynote speakers giving presos in front of massive, wall-sized video flat screen panels;and no banners or security guards or fancy speaker passes. It took place at Brian Behlendorf's house, which is not even in a gated community. We drove right past his house the first time while looking for his address.
The guys that I interviewed were among some of the brightest minds in open source; Brian Behlendorf; Upayavira; Greg Stein; and Roy Fielding, all of whom are well known and regarded (or deserve to be). These guys have the skills to be Microsoft millionaires, but instead but instead flew thousands of miles to sit slouching on couches and squatting on cushions hacking infrastructure maintenance for free, primarily just to hang out with each other, even though they could have done the same thing on line.
In public statements and in its 2004/9/1 SEC 10-k mandatory legal filing, Microsoft calls open source projects like Apache the second greatest profitability concern behind a weak global economy. Yet Microsoft doesn't “get” why their profitability is imperiled by a movement that their Chairman called a group of “communists.” Maybe Chairman Bill would "get" Open Ssource better if he hunkered down at some infrathons himself.
When I asked these guys what they liked about the infrathon, some of them did talk about seeing the physical manifestations of the stuff that they were “seeing” in code form: they liked to visit the co-location facilities, the cages, and the actual boxes that are the work horses of the minor miracle that is Apache. They talked about the massive numbers of committers they support, the huge volumes of server hits and email messages, and the security measures protecting all of that massive activity, and they liked the fact that their work here makes all of that stuff go. But mostly, they just liked to hang out with each other.
I'm sure that each of them, upon reading this article, will say that I have made a big deal out of nothing. Each of them was busily volunteering each other's availability when I told them that I was going to interview them for this article.
I spoke with Brian Behlendorf, Upayavira, Greg Stein, and Roy Fielding, in that order, and their interviews have been transcribed, snipped, and presented in that order below, for no particular reason.
Mad Penguin: Brian, what are you guys doing here this weekend for Apache?
Brian Behlendorf: Well, this is what we're calling an Infrathon. This is the infrastructure team for the Apache organization, all volunteers, all people who are programmers on different projects, who also play a volunteer role in helping keep up the infrastructure. By “infrastructure,” I mean the web servers, the CVS and Subversion trees, the bugzilla installation and other bug databases, and make sure that it kinda runs well and smoothly.
It actually is a lot of work. There are over one thousand contributors to Apache and probably over five million hits per day, and probably close to a million email deliveries a day. Maintaining that infrastructure is something that companies probably would pay a lot of money for, but we just kinda volunteer to maintain it, in the same way that we volunteer to write code.
We do this ourselves because we like getting our hands dirty with running these kinds of systems. It gives you some real world practical grounding in how these things work. People here are very hands on and like solving problems. It can be frustrating at times and fun at times, but it was worth it.
Yesterday, we had to move the machines from one co-lo facility to another because the old facility that we were in was closing down. So a group of about twelve of us went down, unracked everything, went to the new colo, put it in place, and for most people, I think that it was the first time, or one of the first times, that they had ever been to a server co-location facility. So there was a lot to learn from doing so.
MP: What kind of machines do you use?
BB: The main web server and CVS trees live on a machine called Minotaur, like the Greek mythical figure. Minotaur is a dual Pentium 2.4 gigahertz dual Xeon with 10 disks in a RAID array that provide about 425 GB of disk space, and we use about 80% of that disk space. That's kinda the main beast. It's a box that we bought from a FreeBSD vendor locally here in the Bay Area called NetExpress. We spent a lot of time configuring it. It runs FreeBSD, and it's been a real work horse for us.
We also have four machines that were contributed by IBM, and we use those to do builds. One is our mail server, that we run various different auxiliary programs on. We also just got a box from Sun, it's a V40z which we will be using for running interesting tests, and we will probably also be running Solaris 10 on it.
MP: What's this infrathon all about, is it long over due stuff, or routine maintenance, or is to build out new features?
BB: Really, the move from one co-lo to the next needed to be done, because the old co-lo was being closed, and so it's a lot easier to do when there's five or six of you than just one or two. There are a few people who live locally here in San Francisco, myself included, who have keys to the cages and things like that, but none of us wanted to be the only one or two to move this stuff, right? So that was one benefit.
There primary difference with the new co-lo is that there will be better power and new facilities, so it'll be a lot better than the old co-lo was. But there's other things that we're working on here, too. There's migrations from old bug database systems to new ones; there's a migration of the old wiki that we have; and basically, there's just this big long queue of things to work on that the volunteers never really got the time and energy to focus on.
Now that we're here in person, we can have these face to face conversations that make it easier to do these kinds of things. So even though most open source work takes place over email and electronically, being able to get together physically in the same place and joke, and go out and get a beer, or have this kind of a shared experience and to see each other face to face, I think is very important for open source communities to do from time to time. That's why people go to conferences and things like that.
MP: What's the most important thing about this event?
BB: I'm not sure this is an event worthy of Slashdot [laughing]. All of the work that we do is done on a public mailing list called “infrastructure”, and a lot of the work that we do is on an IRC channel that's public as well. So even though most of us are here in person, we're still using the tools that allow others to participate as well, so that they can follow it and give helpful commentary and advice as well.
So yeah, it is a nitty gritty kind of thing where someone just has to dive in and immerse themselves in a problem before they can come out with a solution. That's hard when you day-to-day job isn't doing these kinds of things. One thing I should mention is that running these kinds of infrastructure things is something that CollabNet does professionally for other companies. And it costs a lot of time, money and effort to do it right. So I think it's become clear to everyone who it's not just throwing a random consumer PC into a rack and running Apache and walking away. It's really hard work to do.
There's just lots and lots of small little details to deal with. Lots of desires for bug fixes, or slight changes in behaviors, I can actually email you the to do list so that you can crawl over it if you interested in that. [Note to readers; if you're interested in this level of nitty gritty, let MP know, and we can get that info for you.]
MP: When did this event start, and how long is it going to last?
BB: A few people arrived Thursday, to stay here, at my house, and a few more dribbled in on Friday, you know, about four people stayed at my house. And we'll keep hanging out and working on items through Monday night. It's very informal. Most people have to fly back or drive back to somewhere on Monday night or Tuesday morning. I really don't want to build it up to be too much.
MP: So who are some of the people who are here and where did they come from?
BB: There's two from the Netherlands; there's one from Germany; there's one from the UK, his name is Upayavira , he's a Buddhist, and he has some interesting thoughts as to how open source is like Zen Buddhism, actually. A couple people drove up from Southern California, and there are maybe three of four others from other parts of the Bay Area.
So Brian then introduced me to Upayavira, who had this to say about his experiences at the Apache infrathon...
