The interview

The Thinker by Rodin

This blog is principally a written work, but today you can hear my voice if you want. Simply click on the audio player and listen to the MP3 recording. The interview is 24:09 and is 34.8MB in size. (Download)

I was interviewed by NetRoots Radio as a voice in the federal shutdown. The interview was with “Fripp”, a co-host of the Netroots Radio Tuesday show “Kicking Ass”. The interview is not actually part of the show, but is available as a podcast. I simply give my two cents about what it is like to be furloughed and offered a few political opinions. If you read this blog regularly, you won’t learn much new except you get to hear what I sound like.

“Fripp” is actually a man named Tom who is my age and living near Portland, Oregon. Tom and I have been friends since 4th grade so he is unquestionably my oldest friend, thus I was a natural person for him to interview. The interview was done via Skype on Monday afternoon. The natural pauses in sentences have been removed by software. I don’t normally speak so succinctly.

You can learn more about Tom here.

Healthcare.gov and the problems with interactive federal websites

The Thinker by Rodin

Today’s Washington Post highlights problems with the new healthcare.gov site, the website used by citizens to get insurance under the Affordable Care Act. The article also talks about the problems the federal government is having in general managing information technology (IT). As someone who just happens to manage such a site for the government, I figure I have something unique to contribute to this discussion.

Some of the problems the health care site are experiencing were predictable, but some were embarrassingly unnecessary. Off the top of my head I can see two clear problems: splitting the work between multiple contractors and the hard deadline of bringing the website up on October 1, 2013, no matter what.

It’s unclear why HHS chose to have the work done by two contractors. The presentation (web-side) was done by one contractor and the back end (server-side) was done by another. This likely had something to do with federal contracting regulations. It perhaps was seen as a risk mitigation strategy at the contracting level, or a way to keep the overall cost low. It’s never a great idea for two contractors to do their work mostly mindless of the other’s work. Each was doing subsystem development, and as subsystems it’s possible that each worked optimally. But from the public’s perspective it is just one system. What clearly got skipped was serious system testing. System testing is designed to test how the system behaves from a user’s perspective. A subset of system testing is load testing. Load testing sees how the system reacts when it is under a lot of stress. Clearly some of the requirements for initial use of the system wildly underestimated the traffic the site actually experienced. But it also looks like in an effort to meet an arbitrary deadline, load testing and correcting the problems from it could not happen in time.

It also looks like the use cases, i.e. user interaction stories that describe how the system would be used, were bad. It turned out that most initial users were just shopping around and trying to find basic information. It resulted in a lot of browsing but little in the way of actual buying. Most consumers, particularly when choosing something as complex as health insurance, will want to have some idea of the actual costs before they sign up. The cost of health care is obviously a lot more than just the cost of premiums. Copays can add thousands of dollars a year to the actual cost of insurance. This requires reading, study, asking questions of actual human beings in many cases, and then making an informed decision. It will take days or weeks for the typical consumer to figure out which policy will work best for them, which means a lot of traffic to the web site, even when it is working optimally.

The Post article also mentions something I noticed more in my last job than in my current one: that federal employees who manage web sites really don’t understand what they are managing. This is because most agencies don’t believe federal employees actually need experience developing and maintaining web sites. Instead, this work is seen as something that should be contracted out. I was fortunate enough to bring hands on skills to my last job, and it was one of the reasons I was hired. In general, the government sees the role of a federal employee to “manage” the system and for contractors to “develop and maintain” the system. This typically leads to the federal employee being deficient in the technical skills needed and thus he or she can easily make poor decisions. Since my last employer just happened to be HHS, I can state this is how they do things. Thus, it’s not surprising the site is experiencing issues.

Even if you do have a federal staff developing and maintaining the site, as I happen to have in my current job, it’s no guarantee that they will all have all the needed skills as well. Acquiring and maintaining those skills requires an investment in time and training, and adequate training money is frequently in short supply. Moreover, the technology changes incredibly quickly, leading to mistakes. These bit me from time to time.

We recently extended our site to add controls that give the user more powerful ways to view data. One of these is a jQuery table sorter library. It allows long displays of data in tables to be filtered and sorted without going back to the server to refresh the data. It’s a neat feature but it did not come free. The software was free but it added marginally to the time it took the page to fully load. It also takes time to put the data into structures where this functionality can work. The component gets slow with large tables or multiple tables on the same page. Ideally we would have tested this prior to deployment, but we didn’t. It did not occur to me, to my embarrassment. I like to think that I usually catch stuff like this. This is not a fatal problem in our case, but it is a little embarrassing, but only to the tune of a second or two extra for certain web pages to load. Still, those who have tried it love the feature. We’re going to go back and reengineer this work so that we only use it with appropriately sized tables. Still, the marginal extra page load time may be so annoying for some that they choose to leave the site.

Our site like healthcare.gov is also highly trafficked. I expect that healthcare.gov will get more traffic than our site, which is thirty to 40 million successful page requests per month. Still, scaling web sites is not easy. The latest theory is to put redundant servers “in the cloud” (commercial hosting sites) to use as needed on demand. Unfortunately, “the cloud” itself is an emerging technology. Its premier provider, Amazon Web Services, regularly has embarrassing issues managing its cloud. Using the cloud should be simple but it is not. There is a substantial learning curve and it all must work automatically and seamlessly. The federal government is pushing use of the cloud for obvious benefits including cost savings, but it is really not ready for prime time, mission-critical use. Despite the hassles, if high availability is an absolute requirement, it’s better to host the servers yourself.

The key nugget from the Post’s article is that the people managing these systems in many cases don’t have the technical expertise to do so. It’s sort of like expecting a guy in the front office of a dealership to disassemble and reassemble a car on the lot. The salesman doesn’t need this knowledge but to manage a large federal website you really need this experience to competently manage your websites. You need to come up from the technical trenches and then add managerial skills to your talents. In general, I think it’s a mistake for federal agencies to outsource web site development. Many of these problems were preventable, although not all of them were. Successful deployment of these kinds of sites depends to a large extent on having a federal staff knowing the right questions to ask. And to really keep up to date on a technology that changes so quickly, it’s better to have federal employees develop these sites for themselves. Contractors might still be needed, but more for advice and coaching.

Each interactive federal web site is its own unique system, as healthcare.gov certainly is. The site shows the perils of placing too much trust in contractors and in having a federal managerial staff with insufficient technical skills. Will we ever learn? Probably not. Given shrinking budgets and the mantra that contracting out is always good, it seems we are doomed to repeat to these mistakes in the future.

Don’t say I didn’t warn you.

(Update: 10/28/13. This initial post spawned a series of posts on this topic where I looked at this in more depth. You may want to read them, parts 1, 2, 3 and 4.)

America begins its death throes

The Thinker by Rodin

My federal salary has been frozen for two years by law because previous administrations and congresses failed in their duty as national stewards. I am not happy about it, but a frozen salary beats unemployment. Republicans are anxious to do a whole lot more than just freeze federal salaries. They want to cut government, not intelligently like a surgeon with a scalpel, but like a medieval soldier with a battle-axe. Apparently, just freezing salaries is for weenies. Cutting federal salaries shows manhood. Republicans have all sorts of ways they want to show their manhood. Most of them actually suggest insanity. Many are actually hoping to put the federal government into default. They will do this by not extending the federal debt ceiling in the spring. Why? Because they say they were elected not to increase the deficit, but mostly because they can and because inflicting pain on people is fun.

Tea Party Republicans want to cut $100 billion from non-defense discretionary this year, for a fiscal year that ends September 30. Apparently they think a trail of unprocessed social security checks and furloughed air traffic controllers are going to improve us. In fact, a Republican plan calls for a fifteen percent reduction in the civil service and a five-year freeze on federal salaries. One Republican congressman wants to furlough federal employees for two weeks a year. Naturally there is nothing like a plan for doing these things intelligently. They goal is to maim with the hope of killing altogether. It’s like setting fire to your crazy neighbor’s house. It’s quite a belly laugh and it feels so good. Only defense spending is sacrosanct.

Governments everywhere are having problems matching revenues with needs and obligations. The State of Illinois is raising taxes dramatically to do minor things like pave its roads and house its prisoners, after papering over its deficits with accounting tricks for many years. Virtually every state with the possible exception of North and South Dakota are not just tightening belts, but also often dramatically cutting services. States are thinking of doing things that were previously unthinkable, such as releasing minor offenders from state prisons to save money. Some Oregon school districts can’t even afford put their children in school five days a week. The children have to hope that four days of education will allow them to adequately compete in adult life. More likely, they will be competing for positions of stock clerks and cashiers at Wal-Mart. Cities are not exempt either. Harrisburg, Pennsylvania, which hosts the state’s capital, is bankrupt and the financially stressed state government will not bail them out. Meanwhile, to make the new health care reform law work, the federal government is requiring states to add more poor people to its Medicaid roles, which requires more revenue. The states are resisting.

States cannot go bankrupt because they have the power to tax. Those state employee pension plans may be expensive, but to make them solvent states could simply raise taxes. This doesn’t sit well with taxpayers, of course, which is why state bankruptcy holds appeal for some. Everyone sort of assumed that the economy would expand forever, with a few hiccups here and there. When the money got tighter, legislatures did what they often did: they rolled the die and hoped for the best.

Where some see crisis, others see opportunity. “Others” would include former disgraced House speaker Newt Gingrich, who wants the federal government to allow states to declare bankruptcy. He figures it will work the same magic on those evil state and local employee unions that it worked on the auto unions. By freeing states from their pension obligations, states can rapidly get back into solvency. Of course there is the minor matter of state employees, who joined the public sector for inadequate wages on the promise that compensation would come in the form of a decent pension. If Gingrich has his way, they will be eating dog food in their retirement, assuming they can even afford that.

Federal employees like me are not necessarily immune either. A law can make anything retroactive. If a law can freeze my pay for two years, it can reduce my pay by ten percent or find other ways to cut my retirement income. I am eligible to retire in May 2012. Looking at Greece as a worst-case model, I figure a twenty five percent cut in my pension in the name of fiscal solvency is possible, if not probable. It might be more. Some future Congress, citing a grave financial crisis that previous congresses inflicted, may just defund our pensions altogether. (Somehow, I’m betting tax cuts for millionaires would keep going.)

Changing this depends on whether we can keep growing as a nation. If we can then tax revenues eventually start gushing in. Curiously, despite all the rhetoric coming out of Washington, politicians don’t seem to care that much about creating growth, at least the kind that puts money into the pockets of ordinary working people. The House of Representatives, sent to create jobs, is focused on the impossible task of repealing “Obamacare” instead. They are also giving priority to tightening abortion laws, by trying to outlaw abortion coverage in private sector health insurance plans. (So much for getting the government out of our business.) Former Pennsylvania senator Rick Santorum, rumored to be angling to run for president in 2012 has prioritized protecting blastocysts in a petri dish above the life of an actual two-year-old girl.

Collectively, our nation is dealing with the repercussions of thirty years of prioritizing ideology and short-term thinking over the nation’s long-term needs. Our new Congress could, for example, be working on ways to make Medicare solvent. That might indicate that they are actually earning their salaries. Instead, at least in the House of Representatives, they are more concerned about investigating all aspects of the “corruption” in the Obama Administration. These actions point to putting axes to grind ahead of the needs of the nation. No end is in sight.

What we need is not a Congress of short-term ideologues, but a Congress willing to address the long term needs of the nation. You know, people charged with being stewards of our nation, looking beyond reelection in two years to making sure the nation is secure, solvent and prosperous twenty and fifty years from now. In short, we need politicians who care as much about our nation fifty years from now when they are dead than two years from now.

I can’t see how to possibly sell this to a political set of ideologues and voters only concerned about the short-term. Other nations, like China, do not have these problems. They instead have long-term strategies and systematically execute them. That we can no longer do this points to the real reason for our national decline. Unless we can collectively envision and work toward the same common future, our national decline is guaranteed. American exceptionalism? Hardly. Instead, America has evolved into a nation that promotes systemic national dysfunction.

And unless you care a whole awful lot about our future and demand these kinds of politicians, your future, your children’s future and our nation’s future will only get bleaker. As for my pension: I had best not count on it, or Medicare, and maybe not even Social Security.  The sad truth appears to be that our nation’s glory days are over and we are in for a long and painful decline. In fact, it is already well underway.

USGS: a great place to work

The Thinker by Rodin

For about two years now, I have been working for the U.S. Geological Survey. I work at their headquarters building in Reston, Virginia. I am a civil servant with twenty-three years of federal service. The USGS is actually the fourth federal agency where I have hung my hat. For me there is absolutely no question about it: working for the USGS is a wonderful and stimulating experience. For twenty years, I worked at agencies full of mediocrity. Sometime they bordered on being dysfunctional. Consequently, sometimes my hard work was not appreciated. Now, I look forward to coming to work. There is no reason for me to look anywhere else in Club Fed. USGS is where I will hang out until I retire. The only thing that upsets me is I had to spend twenty years wandering the federal wilderness before I found a home at USGS.

If you take the time to visit the USGS jobs site, you can read exactly why it is a great place to work. For the most part the information on the page would apply to any federal agency. Arguably, these days any one of these standard federal benefits would qualify it as a great place to work. Try getting a defined pension benefit plan as a new employee even at IBM these days.

One of the reasons I like working at USGS is that, of the four agencies for which I have worked, it feels the least like a bureaucracy. It is more than the casual dress. USGS is part of the Department of Interior, and our unofficial department motto seems to be, “We don’t need no stinking suits and ties!” Of course, since we manage federal lands many of us spend our days outdoors getting very personal with nature. However, many of us are still tied to our desks. Except for some in the Department of Interior headquarters in D.C. and various senior executives scattered across the country, few of us do the suit and tie thing. Even my Associate Director usually arrives in slacks and a button down shirt (no tie). He keeps a sports coat and some emergency ties discreetly in his office should the situation warrant. Casual Friday? I am trying to imagine how that would be different. Every day is casual day where I work. I wear jeans to work every day. I generally avoid wearing T-shirts, although many employees wear them routinely. I could wear sneakers too but I prefer wearing modest hiking shoes instead. The only time I have to play the dress up game is when I am going to an important meeting offsite. For example in December, I had to attend a meeting at the National Science Foundation. I still skipped the suit, but I felt compelled to do the dress pants, shiny shoes, long sleeve shirt and tie thing.

Of course dressing casual is more the business norm these days than dressing up. However, those of us who live and work around Washington, DC usually have to play the dress up game. The degree of dressiness is directly proportional to your distance from the White House. Particularly if you reach a certain federal grade level (generally GS-13 or above) the peer pressure to dress up can get quite strong. For more than twenty years, I did the dress up lite routine, which meant everything but the suit. In later years as I advanced to the upper grades I learned to keep a sport coat in my office for those occasions when I had to interact with people more than a grade above me. Needless to say it didn’t fit me. I always felt I was projecting the wrong image of myself when I dressed up for work. I am more of a jeans and polo shirt kind of guy.

So perhaps the casual dress culture is not that much of an asset. For me the most amazing thing about the USGS is that employees are fully empowered. There is of course a top down hierarchy; it is just that most of the time it does not matter. My associate director, for example, is a man named Bob. He expects a relative peon like me to also call him Bob. Everyone I meet feels fully vested in the agency and knows that their work matters. It matters because their work really does matter. USGS is, after all, an institution chock full of scientists. Scientists as a rule are far more concerned about science than they are about politics or hierarchies. Nothing is more precious to us than our reputation for accurate science.

In other federal agencies where I worked, many employees were clock-watchers. It’s not that they hated their jobs, it’s just that their evenings were far more enjoyable than their working hours. At USGS, most of us do not watch the clock. We are too busy happily engaged in our jobs. I trust that all of my employees will accurately account for their time and I am sure they do. Some I know will routinely work many more hours than they can charge for without authorization. They do it because they are involved with their work. They know that their contributions make a tangible difference to the quality of our science and the products that we put out. Consequently, their job becomes fun instead of a chore.

USGS is a very spread out agency. It has to be that way since ours is a big country. We need to be close to where the science is happening. Each state generally has a central office, and most have branch offices. To collaborate you have to work across geographical boundaries. Of course, this means a lot of conference calls and online Webex sessions. It also means a fair amount of travel. I am sure we have employees who never travel anywhere, but I think they are the exception. It is an unusual employee who does not have to travel somewhere on business at least once a year. Last year I was on an airplane five times for my job. I could have likely been on an airplane many more times had I elected it. Mostly I go to Denver, but last year I also visited Helena, Atlanta, San Francisco and Portland, Oregon. It is good to get out of the office during the year for a change of perspective and scenery. My job has just the right amount of business travel. Often I have the opportunity to see some unique aspects of the areas that I visit. In my other jobs, I could go years between business trips, if I went on any at all.

No matter where I go though, the people who work for USGS are uniformly friendly, professional and interesting. Despite seeing many of them only once or twice a year, it is as if they are just down the hall from me. The many conference calls between business trips fill in the gaps. We are truly one big team. The only challenging part is dealing with the time zone problems. Invariably for those of us on the East Coasts this means our many conference calls are packed into the afternoons.

As far as I can tell, the only downside to working at USGS is we cannot own certain kinds of energy stocks. Since geology is the best-known part of our business (we also do water resources, biology and cartography), those engaged in geology may have insight into areas that are profitable for oil and natural gas exploration.

I suspect there are other federal agencies that are similar to USGS, but not many. I would bet NOAA and the National Science Foundation share many of our values too. I do know that I feel very valued and engaged at USGS. I appreciate the non-hierarchical culture; it is a perfect fit for me. If you want to impress people at USGS, do better science. For the most part though we are too engaged in our science to care too much about whether our own egos are puffed up or not. We are professionals in the best sense of the word.

The True Size of Government

The Thinker by Rodin

My agency is going through another one of its periodic, politically required, staff reductions. My agency is hardly unique. Cutting the number of federal employees has become the key metric for demonstrating that government itself is smaller. In 1996 when President Clinton declared the era of big government was over there were approximately 1.9 million of us on the payroll. This follows a downward trend that President Bush is only accelerating.

If government is getting smaller why does it still feel so big? Our office space has not shrunk, and I don’t pass rows of empty cubicles every day. The answer becomes a lot clearer when I look at who is now occupying cubicles of the departed. Their badges are not white. Their badges are pink. In my agency this means they are a contractor. It should not be news that the federal government has relied more on contractors and less on civil servants to get its work done. Indeed in my agency the political imperative to contract out is written into the performance plan of every manager. These contractors are doing work that previously was done by federal employees. Those in favor of a leaner and meaner federal government should applaud. But is the government really any leaner?

Clearly the cost of a federal employee is not a trivial expense. We come with generous leave allowances and decent health care plans. Those of us who remain often have other benefits, such as flexible work schedules and, increasingly, flexible work locations too. And no civil servant I’ve run into can candidly admit they haven’t seen fellow employees abuse the civil service rules and get away with it. Clearly some reforms are needed. I can report after nearly 20 years in the federal trenches that the stereotype of the lazy government bureaucrat is a rare exception to the rule.

One motivation for hiring a contractor instead of a federal employee is that they are expendable. Or are they? I found a curious thing when I arrived at my agency in 1998. Many of the contractors had been working for and supporting the agency longer than many of the federal employees. On those infrequent occasions when contract companies changed, the new contractor invariably picked up these old time contractors. Even more alarming from my perspective is that they often had sole subject matter expertise. If all our contractors were to leave tomorrow it is not even clear that my agency could even function in any meaningful sense. In the information technology shop where I work, many of us “feds” would be hard pressed to modify a line of code, and would be harder pressed to find it. If a contractor looks like a fed, talks like a fed, and squawks like a fed, isn’t it a fed? Clearly the Bush Administration doesn’t think so, and Congress shares this opinion. To admit otherwise would be to admit that government is not leaner than it was.

A leaner government should be able to squeeze more value for the taxpayer. As a taxpayer I certainly hope this is the case, but I am skeptical. Most of the contractors I encounter work on services contracts. While there are exceptions most of them work in-house. My agency provides them virtually all the standard services it would provide a federal employee. Their cubicle may be a bit smaller, but they use the same phones and copiers. When they travel, they use government travel services and get the same discount airfares. But there are a few things that are different. Some of them have to take leave on federal holidays. And work cannot be directly delegated. It must go through contracting supervisors, which can create lag times. In addition the contract can be performance based.

I do know that of those contracts I have seen that the billing rates have raised my eyebrows. I know there are indirect costs (such as the cost of the infrastructure) that must be added to my direct costs that make my official salary nowhere near my true cost to the government. But these are mostly services we provide to our in-house contractors. So it is tempting, though perhaps not completely accurate, to compare direct federal costs vs. contract billing rates.

If our contractors were federal employees I’d guess their average grade would be a GS-13 making perhaps $65,000 a year. Let’s add a generous 70% for other direct employment costs such as employer contributions to social security and amortizing costs for retirement then if they were federal employees they would cost the taxpayer about $110,000 a year in direct costs. This amounts to about $53 per hour.

How much is the government being billed by the contractor for these services? If you were to add 50 to 100 percent you would be in the ballpark. Ah, but contractors are disposable! Congress could come by tomorrow and wipe out the program they support and off they would go. But of course Congress hardly ever wipes out programs. So contractors stay. And their meters keep running.

The true size of government is hard to calculate. Statistics are hard to come by because it appears that agencies don’t want to collect this information. The Brookings Institution published a persuasive book called “The True Size of Government” in 1999 that argue as of 1996 there were in excess of 12 million fulltime federal employee equivalents. Even if the true number is half that amount, the true size of the federal work force is growing.

Federal employees keep retiring at a brisk pace, often spurred on by early retirement options provided by agencies desperate to make the latest politically motivated head count. Those who remain grow grayer. It is increasingly difficult for agencies to bring in new employees to replace them. It is a safe bet that domain knowledge is being transferred to contracting staff. This assumption means that large numbers of government contractors are in effect federal employees performing inherently governmental functions. And contracting agencies are likely making very nice profits.

As a federal employee I am concerned about this trend. Congress needs to examine the true size of government and think about what it means if inherently governmental functions are being done by those who are not federal employees. New and meaningful metrics on the true size of government are needed. As a taxpayer you should consider that increasing the number of federal employees might well be in your interest, provided they are coupled with meaningful reforms in the civil service system. Be suspicious of numbers you are hearing about how the size of government has shrunk. Most likely you are being sold snake oil.