Massachusetts might have been the FOSS shot heard 'round the world, but California may be quietly building pressure for an open source earthquake of its own. On the face of it, the California Air Resources Board (ARB) is not setting the world on fire with its well-documented adoption of free open source software. It is using FOSS primarily in the back office, just like so many other governmental agencies and businesses. But if you dig just a little deeper, as shown in this Mad Penguin™ interview of the ARB staffers responsible for moving ARB toward a more FOSSy future, you can see that the seeds of more profound change gradually developing.
This Mad Penguin™ interview brings together the CIO of ARB, Bill Welty (BW); Narcisco Gonzalez (NG), one of the Staff Programmer Analysts, and Harry Ng (HN), the Applications Development Manager. The interview panel also included Bill Fell (BF), the ARB Webmaster who teamed with Harry to blend the use of FOSS code with ARB's scientific and engineering culture of collaboration, peer review and information-sharing. The result has been the implementation of an Internet-based community of regulators, the industry members they regulate, and clean air advocates, worldwide.
This Mad Penguin™ ARB interview is interesting for several reasons. First, the State of California encouraged its departments to take a look at open source via the Governor's 2004 California Performance Review. Second, the culture of collaboration at ARB is being energized by the confluence of the agency's decidedly scientific bent, the use of open source, and involving the regulated communities in the development of regulations. Third, as the air quality manager for the most populous state in the US, ARB is recognized globally as a trend-setter, and its methodology stands to be copied world-wide, including its emphasis on sharing information, building community, and deploying FOSS.
Like the San Francisco fog, open source at ARB has crept in on little cat feet. The IT staff at ARB weren't out to change the world. They were initially interested in coping with limited budgets. These folks are passionate about fighting air pollution, and so they have been highly motivated to stretch their dollars rather than cut back on their delivery of services, despite years of lean budgets since the dot-bomb leveled California tax revenues. Open source fit right into that picture of coping with lean budgets, according to this interview. Equally important, their data is their passion, and so they were not keen on having to essentially repurchase their data by renewing hardware or software they didn't need, simply because a vendor said it was time for new stuff.
The California Performance Review 2004 helps set the background for these reviews, because it cites ARB's work as an example of a FOSS test case that is working:
Summary: Each year the State of California spends millions of dollars on software, software maintenance and renewal. Many private and governmental organizations are turning to open source software as a cost-effective alternative to closed source software. The state should more extensively consider use of open source software, which can in many cases provide the same functionality as closed source software at a much lower total cost of ownership....Some state agencies have adopted open source solutions as well. Examples are provided below.
Air Resources Board (ARB) makes use of open source in their web environment. For example, ARB runs the Linux operating system with the Apache web server, MySQL database, Perl and PHP coding and a Swish-E search engine. ARB has used open source from the inception of their web install. [5] http://cpr.ca.gov/report/cprrpt/issrec/stops/it/so10.htm – b
So while it is great news that the governor of Massachusetts has announced the continuation of the OpenDocument policy preference in the wake of Peter Quinn's widely publicized retirement from Massachusetts' IT agency, keep your eye on California for some big future developments. California is beginning to understand that it is OK to try open source software.
Mad Penguin: Why is ARB using open source software?
Bill Welty: If you are looking for a word or a phrase that is a backdrop for ARB's success with open source software, it would have to be that ARB runs on collaboration as a philosophy of governance. From the Chairman's office all the way to the support staff, it's a very flat organizational structure. The rule- making process, as well as the IT program, is all based on collaboration. In light of the collaborative nature of our organization, it was a very natural thing to go to the Internet for software and to put our faith in those open source products we adopted. It did not strike us that this was a wild leap, because collaboration is part of our culture.
Narcisco Gonzalez: My biggest fear is that you get locked in with the proprietary solutions. We do a lot of research here. We collect, compile and process a lot of data, and we report it. That's really what we do. That's our business. We generate regulations to control air pollution, and the whole process of developing regulations is open. We need to keep that information available to the public we regulate. We don’t want to require a group we regulate to have to buy proprietary software to read how we developed the regulations.
MP: So, Bill Welty, when you first found out that your guys were using open source software, did you have any concern about stability, accountability, or reliability of open source software?
BW: No, because collaboration brings with it, and is driven by, having a considerable amount of confidence in the people that you work with. It's like surgeons around a table. They're all in there to do a job, and they all have a lot of confidence in what they are doing. Harry is the equivalent of a skilled surgeon. When he advised that open source and Linux were working for him (CentOS or Red Hat), I had every confidence that it would work. He's a true IT professional. And he delivered. If there was a case where he was not delivering, then we would have different thoughts about it.
At the 2005 National Association of State Chief Information Officers (NASCIO) meetings, one of the questions that I was asked by Peter Quinn was, “Have you ever had a significant or catastrophic failure,” and I had to pause and think about it. In the eleven years we've been doing open source, we have not, in my view, had one failure in applying open source solutions. That includes Linux, Apache, MySQL, PHP, or any of the rest of them. We have learned that so long as you follow standard systems development life cycle steps, like requirements analysis, prototyping, testing, and so on, then why wouldn’t open source work?
MP: Let's talk about the role of open source software and tight governmental budgets. Does open source help you accomplish your mission within your budget?
BW: Obviously, if you can reduce the cost of licensing software, then the overhead of running your shop is going to be less. And because the software is stable, powerful, quite competitive with anything else on the market, when there are budget crises none of our projects are slowed or defunded. We just kept pushing forward.
MP: Has the discussion in Massachusetts caught your ear as CIO of ARB?
BW: I think that it is a huge story. It's just one example of the change in the climate, and the change in the culture throughout government and business in the US and throughout the world. Software is becoming a commodity. People are being educated in computer skills from a very early age, and so software is no long a novelty. People are wondering why they need to spend so much money for things that may or may not provide them with the results that they're looking for.
You are seeing a gradual sea change. It's not radical. But the Massachusetts thing is one more example of how governments are reacting to the high cost of software when lower cost open source alternatives may exist. There are alternatives. Governments don't want to be locked out of their data. They want some assurance that they will be able to read and own their data into the future. This is the point that Narci made earlier - sovereignty and perpetual access to content. It's a very good point.
You read statistics today about the market share that Microsoft has, and you read statistics about how open source performs versus how Microsoft performs on older machines. I discount a lot of current day statistics because it's always a snapshot, a point in time. Open source is basically going to dominate over the next ten to fifteen years. The businesses that are currently in the software business, like Microsoft, are going to have to change. People can't afford the current business model. If Microsoft decides to adopt an open source business model, it might be a different look entirely. But I think that the Massachusetts thing is merely an indication that there is something very real occurring in the IT business, and people need to pay attention to it.
MP: Can you give some examples of that? What have you heard?
BW: The State CIO has set up a working group to be comprised of nine to twelve individuals, CIOs primarily, and other participants, to assess open source solutions and advise on how open source may be used within State government. The California Performance Review (CPR) had a section called “State Operation Number 10” that basically said state departments should look to open source, and to try open source where feasible, to look for opportunities to test it as an alternative to the standard procurement methods. The CPR didn't give a preference for open source or proprietary solutions. The goal was simply to get “best value” for the State. The working group is going to identify cases and opportunities in which departments can share ideas and collaborate, like we do here at the ARB.
MP: Speaking of collaboration, you were speaking earlier of a case in which ARB was able to adopt a rule change that asked a great deal of one of the industries affected by your rule making, but the industry moved along with the decision because the rule was adopted in a more collaborative method. Can you tell how that happened?
BW: Right. ARB has an on-line newsletter called “Air Waves.” Any ARB employee can write an article for this newsletter. It's one way for staff to share their experiences. It's part of the collaboration thing. When they understand how collaboration can work for them, ARB becomes more efficient. One of the articles we wrote was an interview with Mike Waugh regarding the public benefits of ARB's web services. In this case, ARB was in the process of developing the portable diesel engine air toxic control measure (ATCM). We were trying to reduce the toxic emissions coming from portable diesel engines. The ACTM was going to cost business about $500 million over the next 16 years. That's a lot of money for the manufacturers of portable diesel engines. But because of the collaborative process, from the notices going out via list serves, to having full disclosure of all the regulatory activities, including all of the activities on the web, everything was in sight. When it came time to approve, amend or disapprove the proposed regulation at the Board hearing, nobody objected to the ARB staff recommendation. The proposal had been fully vetted over a period of 12 to 18 months through all of these collaborative efforts. The stakeholders were involved from day one which is a good thing if you are a regulatory agency.
MP: Let's talk about preservation of data. You have to look down the road over several business cycles. How long is your time horizon, and how do open source and open standards address those needs?
BW: Virtually, every official document that ARB produces ends up on the web. If anyone wants to know anything about any meeting, any hearing, anything that was public, it's available on the web, and it goes into an archive. We save it forever. Because it's HTML, or it's a PDF file format, it's readable by any browser. There are no real issues about proprietary file formats. To the extent that those two protocols remain, then I don't think that we'll have a problem in perpetuity. The attractive thing about the Massachusetts effort is that maybe open data standards can be instituted within an organization even as diverse as the State of California.
MP: Why do you need to hold onto those documents for so long?
BW: People want to know how certain regulations were made. What was the history; what was the thinking; who was in the work group; what was the philosophy, what were all the things that went into any particular regulation? So that information is very useful to have over time. There is also the historical nature of these documents. ARB has the reputation of being a world leader in the air quality management arena. People look to ARB for leadership in this area. By having all of these documents available on our site, our achievements can be shared worldwide. In fact, we spend a lot of time with a lot of foreign countries sharing with them how to improve their air quality. It's important for future generations that we preserve this information.
MP: You mentioned earlier that you think that open source and collaboration has positively affected morale at ARB. Could you elaborate on that?
BW: In my experience as the CIO of ARB, these things have invigorated the morale of staff throughout our organization. And it's not just because open source is new. It's because it works, it's dynamic, it's part of the Internet community. The Internet community is where technology is. By being part of open source, our collaborative efforts go farther. It's just one more piece of this collaborative thing that we are doing. The synergy is incredible.
We recently put a job notice out for a senior level position, and one of the things that we added to the notice was that we are an open source shop. I got a number of applications in which people said that they were willing to take a pay cut if they could come to us, because of our open source culture. So it helps us attract quality people. There are a lot of people who feel “trapped” in their organizations, and they can't really experiment and be part of these new changes that are sweeping through the Internet. ARB can give them that opportunity.
MP: Narci, you mentioned earlier that you had been using Microsoft and Oracle tools, but that open source has increasingly become part of your life here at ARB. How did that happen?
NG: I started many years ago with the bulletin board system, on a DOS-based computer. It was back in the early nineties. We were running a database of district rules on an old bulletin board system (BBS), so that California’s 35 local air districts could update their rules into a centralized database, so that everyone would be aware of what rules exist throughout the State. Right about that time, the web was coming along. I got a web server going running Windows NT 3.0 or 3.5 and Website Pro as the web server. We developed some applications with a framework that was specific to Website Pro in Visual Basic. Meanwhile Harry was experimenting with Linux and CERN server. Soon, Harry came along and said, “How about Apache?” The available literature was telling us that Apache was building serious momentum. Well, Apache early on wasn't Windows-based, it was all Linux-based. Needless to say, we slowly started migrating off of Windows onto Linux, and now it's mostly running on Apache.
Harry Ng: When I first started 25 years ago, the data center was where we would build databases, things like RAMIS databases, hierarchical databases, and so forth. Later on, as we were tasked to build a new toxics emissions database, we were one of the first to work along with the data center to say, “hey, let's use something other than IBM's MVS system and those standard hierarchical databases.” We worked with them on a pilot to test out Oracle. We decided at that point, well, Oracle runs not only on big IBM mainframes, but also on smaller systems like Unix computer systems. We did benchmarks with them using Unix and Oracle on a DEC Ultra system, as opposed to the MVS system. We found out the DEC systems gives you good performance at a much lower cost than using IBM mainframes. That's where we started picking up expertise in the use of Unix. Because of the fact that we knew Unix, it was not very hard for us to pick up on the use of Linux.
One of my programmers one day bought a book on Red Hat Linux and lo and behold in the back cover, there is a disk that happens to have Linux version 4, I think it was. At that point, we installed it. He heard about a great server called CERN, that is for showing web pages, and a browser called Mosaic. We got that installed, and got it working under Linux, and we saw that there was real potential in this technology. You can see the graphics, you can see text, we can actually put it out there. At that time, we also started up a proxy service using Apache, and set up a firewall and our Domain Name Servers. Initially we used Oracle database and the Oracle web server software to do our database applications across the web. When we heard about PHP, we gave that a try. PHP was a much better language for writing applications for the web. It gave us a lot better functionality. It was being developed at a much faster rate that the Oracle server was being built. The main thing we liked about it was that it didn't tie us to a particular database. So PHP became the one application language that we could use to cut across multiple databases and applications anywhere within ARB.
NG: It's been difficult to move off of Windows and Oracle and move onto Linux, but most applications have been migrated over to Linux-Apache-MySQL and PHP (LAMP). Hopefully, within the next couple of years we'll have taken everything off of any Windows-based product and have switched everything to the LAMP stack. But there is no huge hurry sometimes when things are running fine. The timing is sometimes dictated by when a system requires a re-write.
MP: What other experiences did you have with proprietary systems? Was the support working, was it not working, what happened.
NG: The hardware vendor's going to say it's the fault of the software vendor, and the software vendor's going to say it's a hardware problem. At that point, I just said no. I've gotta take more control. I felt like I was totally out of control. I couldn't control the hardware or the software. At least with Linux, you can control the software side.
MP: What other advantages do you find in using open source?
NG: Being able to go out and do some research on the Internet, go into some forums, like Source Forge, look at the bug track reports, it really helps you decide what you are going to choose. For example, right now we are trying to decide whether to go with a wiki or a content management system for a small web site that we’ve been asked to develop. Harry decided that he's going to look at the wikis, and I'm going to look at CMS tools. I'll install it, test, and say, oh, this was a horrible install, or I don't like this interface, or I don't like this level of granularity and control. Then I can just uninstall it and go out and get something else. To do that with proprietary software is a little bit more difficult. You have to say, “I'd like to buy XYZ software.” You could get a trial version, but you may still have to go through that procurement process, and that takes you three months! So using open source software really speeds things up. You can get prototypes up and running pretty quickly. Then, once you get a prototype up, you ask yourself if you are going to need something more robust. And you can do it very cheaply.
