How failed: the technical aspects

(Also read parts 1, 2 and 3.)

A lot of how works is opaque. This makes it hard to say authoritatively where all the problems lie and even harder to say how they can be solved. Clearly my knowledge is imperfect and thus my critiques are not perfect either. I am left to critique what the press has reported and what has come out in public statements and hearings. I can make reasonable inferences but my judgment will be somewhat off the mark because of the project’s opacity.

It didn’t have to be this way. The site was constructed with the typical approach used in Washington, which is to contract out the mess to a bunch of highly paid beltway bandit firms. Give them lots of money and hope with their impressive credentials that something will emerge that is usable. It was done this way because that’s how it’s always done. Although lots of major software projects follow an open source approach, this thinking hasn’t permeated the government yet, at least not much of it. Open source projects mean the software (code) is available to anyone to read, critique and suggest improvements. It’s posted on the web. It can be downloaded, compiled and installed if someone has the right tools and equipment.

It’s not a given that open sourcing this project was the right way to go. Open source projects work best for projects that are generic and used broadly. For every successful open source project like the Apache Web Server there are many more abandoned open source projects that are proposed but attract little attention. Sites like are full of these.

In the case of, an open source approach likely would have worked, and resulted in a system that would have cost magnitudes less and would have been much more usable. It would still have needed an architectural committee and some governance structure and programmers as well, principally to write a first draft of the code. Given its visibility and importance to the nation it would have naturally attracted many of our most talented programmers and software engineers, almost all of who would have donated their time. Contractors would have still been needed, but many of them would have been engaged in selecting and integrating suggested code changes submitted by the public.

If this model had been used, there probably would have been a code repository on Programmers would have submitted changes through its version control system. In general, the open source model works because the more eyes that can critique and suggest changes to code, the better the end result is likely to be. It would have given a sense of national ownership to the project. Programmers like to brag about their genius, and some of our best and brightest doubtless would have tooted their own horns at their contributions to the code.

It has been suggested that it is not too late to open source the project. I signed a petition on asking the president to do just this. Unfortunately, the petition process takes time. Assuming it gets enough signers to get a response from the White House, it is likely to be moot by the time it is actively taken up.

I would have like to have seen the system’s architecture put out for public comment as well. As I noted in my last post, the architecture is the scaffolding on which the drywall (code) is hung. It too was largely opaque. We were asked to trust that overpaid beltway bandits had chosen the right solutions. It is possible that had these documents been posted early in the process then professionals like me would have added public comments, and maybe a better architecture would have resulted. It was constructed instead inside the comfort of a black box known as a contract.

After its deployment, we can look at what was rolled out and critique it. What programmers can critique is principally its user interface because we can inspect it in detail. The user interface is important, but its mistakes are also relatively easy to fix and have been fixed to some extent. For example, the user interface now allows you to browse insurance plans without first establishing an account. This is one of these mistakes that are obvious to most people. You don’t need to create an account on to shop for a book. This was a poorly informed political decision instead. Ideally someone with user interface credentials would have pushed back on this decision.

What we saw with the system’s rollout was a nice looking screen that was full of many things that had to be fetched through separate calls back to the web server. For example, every image on a screen has to be fetched as a separate request. Each Javascript library and cascading style sheet (CSS) file also has to be fetched separately. In general, all of these have to complete before the page can be usable. So to speed up the page load time the idea is to minimize how many fetches are needed, and to fetch only what is needed and nothing else. Each image does not have to be fetched separately. Rather a composite image can be sent as one file and through the magic of CSS each image can be a snippet of a larger image and yet appear as a separate image. Javascript libraries can be collapsed into one file, and compressed using a process called minifying.

When the rollout first happened there was a lot of uproarious laughter from programmers because of all the obvious mistakes. For example, images can be sent with information that tells the browser “you can keep this in your local storage instead of fetching it every time the user reloads the page”. If you don’t expect the contents of a file to change, don’t keep sending it over and over again! There are all sorts of ways to speed up the presentation of a web page. Google has figured out some tricks, for example, and has even published a PageSpeed module for the Apache web server. Considering these pages will be seen by tens or hundreds of millions of Americans, you would expect a contractor would have thought through these things, but they either didn’t or didn’t have the time to complete them. (These techniques are not difficult, so that should not be an excuse.) It suggests that at least for the user interface portion of the project, a bunch of junior programmers were used. Tsk tsk.

Until the code for is published, it’s really hard to assess the technical mistakes of the project, but clearly there were many. What professionals like myself can see though is pretty alarming, which may explain why the code has not been released. It probably will in time as all government software is in theory in the public domain. Most likely when all the sloppy code behind the system is revealed at last, programmers will be amazed that the system worked at all. So consider this post preliminary. Many of’s dirty secrets are still to be revealed.

Pocket Money, Part Two

This is a follow up to my entry Pocket Money published on April 28th. In that entry I said I was thinking of selling my web services for money, since I was getting regular interest.

Over the last month, I have earned $119 from customers on the Internet. In my real job, it would not even pay for three hours of my time. Nevertheless, this is real money. It is not funny money that some alleged “barrister” in Sierra Leone wants to send me (if I can just let them use one of my accounts). As we speak, it is sitting in my checking account. It is urging me to spend it on something frivolous. I have already succumbed. I spent part of that money buying Seasons 2 and 3 of The West Wing. The rest will be blown on other frivolities, perhaps when we visit Paris next month. Yes, there is money to be made online. It can be done from the comfort of your home computer. You can do it in your boxer shorts while eating Cheetos if you want. It may not be a boatload of money, but it is money nonetheless.

Since you rarely meet your customer face to face over the Internet, you generally want a financial broker to transfer money. My broker is the only one I know about: PayPal. They charge a small percent of any money that I receive.

For that money, I have installed the phpBB Digest modification that I wrote two years ago on three different web sites. The Digest modification sends an email digest to phpBB forum users on either a daily or a weekly basis. In addition to my Digest modification, I have installed the phpBB Attachment modification twice. This modification is popular because it allows users to upload photos, documents and the like to the forum to share with others. Like my Digest modification, it is a pain to install, but a truly neat feature once it is installed.

It is a good thing that I learned early in my career (when I sold lawn and garden equipment for Montgomery Ward) that the customer is always right. Okay, they are not always right. Sometimes they are dead wrong. Still, if you want repeat customers and something like a positive reputation, it helps to go that extra mile. It also helps not to be squeamish. One of my customers runs a gay oriented site. Apparently, there are some gay men that are into dirt, urine and body odor and he is one of them. He asked somewhat tentatively whether I would do the work because of the mature theme of his site. Of course, I told him. How he and his friends get their jollies may not my cup of tea, but it puts money into my bank account. If it is not blatantly illegal, why allow my squeamishness over a particular sexual fetish get in the way of my financial enrichment? Business is business after all, and these sorts of moral dilemmas never kept a Republican from getting rich. Ask Pat Robertson about his Nigerian investments.

My first customer is already a repeat customer. After installing the Digest modification, he saw my phpBB site and liked the way I added a column to the right hand side on the main index. It shows post per day, recent posts, recent visitors and integrated RSS news feeds. Could I do the same thing for his site? Not a problem: $65 please and I included 30 days of support in case something needed tweaking.

One thing I am learning from this kind of Internet commerce is that no software is truly free. The “free” open source software model (which is how phpBB is licensed) provides the illusion of free software. Yet money there is still money to be made from free software. MySQL, the open source database vendor, figured it out: give out the database for free (at least for non-commercial use), but offer paid support regardless. Only if you do not value your time does open source software really become free. MySQL is quite profitable, and it is making bigger and proprietary vendors like Oracle sweat bullets. Some, like Computer Associates, have followed the MySQL model. The Ingres database, owned by CA, is now also open source, and has been renamed OpenIngres.

Another thing I am learning is that this kind of work on the Internet requires risk. The risk is not on my end, since I require cash up front. Once the money is in my PayPal account, it might as well be in the bank. However, it is risky to my customer. Can they trust me? For to do the work I need administrator access to their web servers. Will they give me, a total stranger, sensitive information like this? Apparently, yes. Fortunately, I have no malicious intent. However, if I wanted to I could install viruses, or even erase their whole site. By building up a referenced clientele, I hope to assuage such concerns.

I am still not sure how long that I will stay in this business. It may be that I will lose interest, or customers will dry up. If so, I do not need the income. Perhaps that is the best part of this work: I can take it or leave it. However, at least in my case, it amounts to more than pocket money.

Pocket Money

It used to be that adjunct teaching was my source of pocket money. Unfortunately, I have not had the opportunity to teach for about a year. I travel on business a fair amount. I do not feel that it was fair to my students to commit to teaching a weeknight course if I were likely to miss a class or two during a semester. Therefore, I have been looking for Saturday courses to teach instead. However, usually there are not enough students to fill a Saturday class, so the class gets canceled.

Fortunately, I do not need the money. Still there was something nice about the extra money coming in. While I certainly spend money on items for my own amusement, it seems like there are more important things I should do with my money. Or perhaps I should I say our money. I am a married man, after all, the primary breadwinner and there are many bills to pay.

With my adjunct teaching money (which as any adjunct teacher knows does not amount to very much) I felt like this extra money was all mine. I could spend it on whatever whim I wanted. Most of the time I would use the monies to buy expensive trifles for myself I was otherwise too cheap to purchase. My digital camera was one of these trifles.

Sometimes genuine financial opportunities knock on your door. You just have to be mindful of them. I get a couple of unsolicited requests per month from distant strangers on the internet who want me to do computer work for them. Unlike my wife, who can take apart, rebuild and build computers for people, they do not want me for my hardware expertise. This is good because I have none. I am a software guy. I do however have modest skills doing things on web servers. This is good because I manage a large, complex, dynamic web site for a living. It pays the bills. Since these requests keep coming in, I am thinking in lieu of teaching that maybe this could be a new source of pocket money.

phpBB is forum software that is widely used on the web. Although you may not know the name, you see it everywhere. You see it because it is open source (i.e. free to own and modify) and of very high quality. I started running a phpBB forum of my own in 2002. It was very useful out of the box. However, since I can program, I could not leave well enough alone. I began to tinker with it. Actually, I have done a lot of tinkering. Moreover, I have written some significant modifications to phpBB. Yes, of course there is a whole community of phpBB developers out there. They extend phpBB to meet uses never envisioned by its creators. Of course, this is also true of thousands of other open source software projects as well.

My innovation back in 2003 was to create the first usable digest software for phpBB. Most phpBB modifications are pretty small and straightforward. The digest modification though was a big modification. While there are certainly bigger modifications out there, (the Attachment mod comes immediately to mind) and even some wholesale rewrites, the Digest mod filled an important niche. It allowed people to get emails containing posts in the forums they wanted. To make it work I had to learn a lot about the way phpBB is written.

I have also made my own custom modifications for my forum. For example, I have integrated news from other sources into my forum. I have done other neat things like putting up real time graphics showing the total posts per day. More recently, I completed a modification I call Smartfeed (not to be confused with this Smartfeed effort). It allows phpBB forums to be accessed using RSS or Atom newsfeed protocols.

Still, demand for enhancements to phpBB keep coming in. phpBB may be free, but it is like buying a stripped down car. Those with the requisite skills add their favorite modifications from the phpBB modifications database, or the phpBB Hacks site. Many others though simply do not have the skills, or do not want to bother. In addition, they are having a hard time finding someone to install these modifications and maintain the site for them. Moreover, some of them are willing to part with money to keep their “free” board patched and feature rich.

Just last night I got such a request. A forum for PhDs wanted me to install my digest modification. So far, I had said, “no way”. I do not run a charity ward. Then I started thinking about it. Maybe there was a way to make some money from these requests. I said I would install it for free, but warned them that if they wanted more modifications or needed maintenance applied, it was going to cost money. I warned them there was a long-term cost to having a forum. Still, it is often irresistible to get something for free up front (my labor) and still discount the long-term costs.

This work is not too hard for someone with my skills. Actually, it is somewhat boring and tedious. Nevertheless, someone has to do it. In addition, there seems to be a persistent demand out there for phpBB modifications. As long as I do a bit of it on the side, and do not turn it into a second career it may be a great way to pick up some pocket money.

And truth be told, since a friend showed me her video iPod, I have been sort of wanting one of my own. Perhaps I have found a guilt free way to buy one.

The coming pervasive open source software world

I think I’ve become an open source zealot. Having recently returned from Orlando and a four-day MySQL User Conference, I think I have the religion.

Open source is software that tends to be free and whose source code (before compilation) can be freely downloaded, inspected and compiled if necessary. It’s not that I don’t think commercial software has its place anymore. But when looking for information technology solutions I just increasingly question why any organization or company would not first look at the open source software market.

In my agency, the U.S. Geological Survey, we have a big enterprise license to use Oracle software. We are being gently pushed and nudged into looking for ways that we can utilize Oracle software in our information technology (IT) projects. In my part of the universe, the water resources area, we are Oracle free. Some visionary developers who arrived years before I did realized that the USGS could serve its water data to the public reliably and cheaply using an “open source” database product called MySQL. This was 1999. No, it didn’t have all the bells and whistles of an Oracle or a SQL Server database. But it was 90% of the way there and was good enough. More importantly it was free to use. In a cash-strapped agency we didn’t have hundreds of thousands of dollars to purchase Oracle licenses. Moreover we sure didn’t want the expenses to have to train a staff in using Oracle technologies. MySQL is straightforward to use, simple to set up, serves data with blinding speed and is rock solid reliable. We didn’t feel vulnerable picking it as our solution. But just incase the developers, MySQL AB went out of business we still had the source code and could improve or modify it as necessary.

In 1999 this was a very innovative act for a government agency. Use of open source software back then was limited largely to the Apache web server and the Linux operating system. More typical was the agency I was working for at the time. It was a pure Oracle database shop. Now unquestionably Oracle runs very fast. It has all the latest gee whiz sorts of features including XML support. But we weren’t using any of them. MySQL would have been fine for many of our needs, had our agency bothered to look at the product. And we would have saved hundreds of thousands or millions of dollars.

But open source is not just about saving money. It’s about allowing software to become platforms that can be extended in ways that suits a customer. By buying into Oracle we found it was difficult not to lock into Oracle. I found in my last job that when I compared Oracle Reports to Crystal Reports, Oracle Reports won because we could get it cheaper through our enterprise license agreement. It was more complex to run and set up than Crystal Reports, but that didn’t matter. “We’re an Oracle shop” was the message. Oracle was preferred and the default solution. But its products are black boxes that rarely can be installed and configured to run efficiently by themselves. If you want those sorts of extras you had best hire Oracle Consulting to come in. For $2500 a day they’ll be glad to tune these applications to make them work reliably in your environment.

MySQL, on the other hand, is just a database. It doesn’t have pretensions to be anything better. It is optimized to be a fast, open standard, and rock solid reliable database. Yes, it is missing some features that I have taken for granted in commercial products but it will get there in time. For now it is more than good enough. It is a bargain.

It turns out that MySQL is most beneficial as part of a toolset that can be used to plug into different products. It is part of an unofficial open source uber-suite called LAMP: Linux, Apache, MySQL, Perl/PHP/Python. These products are likely the highest quality open source tools out there. Linux provides a rock solid operating system. Apache provides the most feature rich and reliable web server available. MySQL provides reliable and fast persistent storage. And the Perl/PHP/Python scripting languages provide an easy way for fairly junior programmers to serve and collect real time data over the web.

On top of this rock solid and best of class software developers are scrambling to put together new products, some open source, some not. Within the open source community the numbers of projects that utilize these technologies are breathtaking. Look at the Apache site alone to see what clever collaborative communities are doing.

How do companies make money when all this software is free? Some do it by allowing open source software to act as a common denominator platform that they build on top of. Others, like MySQL AB have a dual license: one for non-commercial use which is free and one for commercial use that is not. But even the commercial licenses are very affordable and offer huge cost savings over closed source software. Other companies make money through support agreements. Lots of businesses don’t want to have to track down and fix bugs in software and will naturally want companies to do it for them. This is what we are doing at the USGS. Our use of MySQL is noncommercial, since we are a government agency. But we spent six thousand dollars a year for a support agreement with MySQL AB anyhow incase we need technical support.

I should point out that most open source software is entirely free for everyone. Apache and Linux, for example, are free. Most open source software is protected by a GPL license. It requires that anyone who improves the software make it available free of charge. There are other public licenses that can be used with open source software that aren’t quite as open. But there is usually a model that will work for any company interested in providing an open source product.

Internally we have systems using OpenIngres as our database solution. We are already feeling the pressure to move away from OpenIngres (which costs us tens of thousands of dollars a year in licensing and support costs) to Oracle. But it’s clear to me that by the time MySQL makes it to version 4.1 that it, rather than Oracle, would be a much more logical database to use internally.

If I were Oracle CEO Larry Ellison I’d be cashing in my stock options. I see the open source wave as something that will likely increase until it becomes a tidal wave. Eventually it will probably sink Oracle, or render it into a company that will have to compete on drastically reduced margins. Even Microsoft is not immune. With the success of open source Linux desktop packages and office suites like OpenOffice, Microsoft will too eventually feel the heat. If Microsoft were more agile, it would be beginning its transition to open source now. Its failure to do so might eventually prove to be its undoing.

Businesses that write closed source software will increasingly have to market their software in unique niches unlikely to be targeted by the open source communities. Open source solutions are likely to become more pervasive because the cost of entry is so small. No one just dips his or her toes into Oracle. Your checkbook better have at least six figures on it.

MySQL and Oracle might not appear to have much in common. But today they have deployed roughly the same number of licenses. By next year MySQL is likely to be the number one deployed database in the world. Oracle is a large corporation with lots of big buildings, guys in suits and Lear jets. MySQL AB is about 130 people, many of them living in Sweden, who dress in T-shirts and who are very approachable. I had a nice long discussion at my conference with Monty Widenius, cocreator of MySQL. He is still very much a full time software developer. You can bet that if I went to an Oracle conference I couldn’t get within a hundred feel of Larry Ellison.

It’s time for businesses and government to look at open source solutions first. We should buy commercial software only when there is a compelling reason to do so. For most IT needs these days, these compelling needs don’t exist.