The glitch that stole Christmas

ICANN's failure to deal with a flaw in its computer software speaks to a bigger problem with the organization itself

It was supposed to be ICANN's swansong. A program more successful than anyone had dared to expect. An expansion of the Internet that would put the organization at the heart of a revolution; where anyone could apply for any Internet extension they wished.

Even considering its size and scope, the new gTLD project had not been an easy ride. Delays measured in years rather than months. Heated policy debates. High-level politics. And then, just weeks before it was due to go live, a Washington broadside that saw no less than three national newspapers, two Congressional hearings and one highly critical FTC report, all say the same thing: hold off, you're not ready. Despite the pressure, and even admitting that the program was unfinished, ICANN threw itself into the hands of fate and launched on 12 January.

That is why the discovery of a "glitch" in the custom software ICANN used for storing the applications is all the more exacerbating. On the very day that the application window was due to close, ICANN announced it was suspending its TLD Application System (TAS). The suspension then moved from a day to a few days, then to a week, then two weeks.

On the day that all the extensions were originally due to be announced - 31 April - ICANN's CEO finally appeared to answer questions. He succeeded only in confusing the situation further. Everything was fixed and working, Rod Beckstrom assured us, and all those affected will be informed by the end of the week. But then, in the same breath, he warned that it may still take until the end of June - nearly eight weeks away - to finish up the process. Why? Performance upgrades on the system, is the official line. Belief isn't high.

Here's what we do know from the little information ICANN has released: in a flurry of last-minute online activity, details of some people's applications were revealed to others. It was a problem that was noticed and reported more than a month earlier, but ICANN says it was not able to reproduce the problem and so it kept the system up and running. By the last day however, the problem had grown so large that it had no choice but to shut it down.

It is with some irony that it now appears the organization had a success on its hands: 1,268 people registered before it was pulled offline. Since each is allowed to apply for multiple extensions, the actual number of extensions applied for could be as high as 3,500 - not bad when you consider that the entire program was built around an expectation of 500 and that currently there are less than 300 Internet extensions in the world.

That success story has now been lost forever though thanks to ICANN's weak handling of the situation. The organization has also achieved the unenviable goal of undermining confidence not only in its ability to handle a crisis but in the organization as a whole. It is a self-inflicted injury that has been a long time coming.

Lack of accountability

As with many organizations that make important decisions in the public eye, ICANN has never been particularly forthcoming about how it makes decisions, preferring to simply state what decision it has reached.

However at the heart of the organization is a dichotomy: it is philosophically bound to the idea of being as open and transparent as possible, while at the same time facing complex policy issues that are much more easily handled in private.

ICANN is not a government body or international organization and so it is not required to work to regulations or to follow accepted norms and standards. Instead, for the past decade the organization has developed its own rules and systems to keep itself in check.

Unsurprisingly, those systems have consistently fallen short and resulted in a steady stream of scandals. Despite three attempts in five years to fix the organization's accountability and transparency failings, the problems continue and this time have been magnified by the sheer scale of the public failure of its application software.

The organization may not yet realize it, but time has been called on its habit of lurching from crisis to crisis, relying on its wits to survive. Through the gTLD process, the organization will now have to face up to thousands of companies that have never interacted with ICANN before but who have handed the company $185,000 and simply will not accept that frequent delays with no information are "just the way it is" in the Internet registry world.

It seems unlikely either that those companies would have agreed to the $60,000 of each application fee that is earmarked for ICANN's legal department to cover "risk costs". They weren't there when the policy was drawn up. But now they will likely get to see their money - which could reach $100 million - being spent on protecting the organization from a mistake of its own devising.

Conservatism

The roots of the problem lie in the organization's history. For more than a decade, its peculiar brand of Internet policy-making on the fly worked largely because the organization adopted a highly conservative attitude. As chair for seven years, Vint Cerf imbued an engineer's careful approach into the fiber of the organization. He was able to do so because as "Father of the Internet", Cerf possesses a near legendary status in the Internet community.

At the same time as a dominant chair kept things careful at the Board level, the organization was built and dominated by two lawyers, working hand-in-hand as general counsel and outside counsel.

Anti-trust expert Joe Sims devised a structure of advisory committees and supporting organizations around a core, all-powerful Board whose members were chosen by those same groups. Combined with the fact that ICANN is based in California and is a not-for-profit, the structure - criticized at the time as "byzantine" - effectively shields ICANN from lawsuits. It hasn't failed yet.

While Sims developed a protective shell, Louis Touton, a patent expert, ensured ICANN was not pulled in the wrong direction over trademarks and navigated a course that enabled the organization to sail on, albeit with a lot of unhappy travelers. Put together, this intelligent conservative approach was very effective, if not very popular.

This era also saw two expansions of the Internet's address space: in 2000, seven from a group of 44 were chosen by ICANN's Board and approved; and in 2004, just 10 groups applied under stricter rules and only two were not approved.

Going off the rails

The pressure on ICANN to live up to its mandate to create competition could not be delayed indefinitely and the "new gTLD" process that would liberalize the entire top-level of the Internet slowly developed. As it did so, key figures in the organization changed too.

Cerf's consensus-based conservatism ended and was replaced with an aggressive, liberal approach from new chair Peter Dengate Thrush. He oversaw approval of the controversial dot-xxx domain, despite strong government disapproval, and forced through a Board vote on the new gTLD process with just minutes left in his chairmanship.

Meanwhile, ICANN's general counsel John Jeffrey attempted to maintain his predecessors' locked-down approach against all the odds and despite the increasingly negative consequences elsewhere in the organization (the FTC's critical report for example focused in on ICANN's failure to maintain compliance with its contracts, which was itself the direct result of a decision to place the compliance division under the risk-adverse general counsel's office).

What started out as strong self-protection mechanisms have long since devolved into a lack of accountability. Mistakes are inevitable in such a fast-moving world as Internet policy, but as the level of accountability over those that make the decisions has fallen, so the number of bad decisions has increased.

For example, when ICANN does finally emerge from the TAS fiasco, it will have to rely on a system it created called "digital archery" that will split up applicants into batches of 500. Each batch will be dealt with in turn, with each turn expected to take six months. The digital archery solution relies on the monitoring of Internet-speed responses from applicants all over the world as they "click" as close as possible to a pre-set date.

Prior to the failure of its online application system, everyone thought this was just a bad system devised by lawyers' fearful they would be sued for running an illegal lottery; now everyone wonders whether ICANN is even capable of running it.

Similar mistakes have been made at virtually every level of the organization. Some have been noticeable: such as the $2 million spent on an awareness program for new gTLDs that comprised little more than sending its CEO on a world tour where he spoke in total to a few hundred people; or ICANN's failure to secure the crucial IANA contract from the US government, being told embarrassingly that it had "not met the criteria" despite having run it for more than a decade.

Others manifestations are internal but no less damaging: the death of a hundred good ideas for improvement; a culture of paranoia and mistrust; decisions made by the most powerful rather than the best informed; short term fixes to long-term problems.

And perhaps the most dangerous of all: a belief that it is always in the right, that admitting error is unnecessary and that criticism can be explained away.

Fresh blood

Ultimately, the failure of its TAS system will claim a scalp - that of the CEO, Rod Beckstrom. But then Beckstrom was already leaving on 31 June after the Board refused to extend his contract. A new CEO has already been decided but any announcement is on hold until this latest crisis is over.

On his (or hers) first day in office, the new CEO will face an impossible task dealing with a number of pressing and complicated matters, not least of which will be shepherding a now damaged gTLD process through to completion.

But the biggest task lies in fixing the culture at ICANN. It will be no easy feat. Executive management is dominated by two types: veterans entrenched in the organization that will resist any efforts at change; and loyalists brought in under the outgoing CEO who will likely jump ship at the first sign that they are not in favor. Everyone else fled or was kicked out under the current CEO. It leaves COO Akram Atallah, CFO Xavier Calvez and CSO Jeff Moss - none of whom have been at the organization very long - as a core around which a new team would need to be built.

Within ICANN, there exists no culture of being held to account. Neither does the organization revisit its decisions, good or bad, so it never learns. Adding to the problems, the Board, which ultimately makes all the decisions, has grown increasingly comfortable with holding meetings in secret, even to the extent of refusing to admit when it has held one.

This degree of secretiveness, where legal fears take precedence over good governance, and where the organization feels little or no obligation to explain itself is unsustainable.

We may never know who or what really caused ICANN to suspend its flagship program, or the depth of the problem. What we can be fairly sure of is that the extensive delay to reopening the system stems from legal fears of being left open to liability. The moment someone fails to get the Internet extension they had spent years working toward, ICANN can expect to see a lawsuit. As a result, until the lawyers are happy, nothing moves, including finding possible alternatives to gathering applicant information (there is nothing wrong with good old-fashioned paper, you know).

"The quality of this program is more important to everyone involved than the specific date and time," Beckstrom said this week in response to a question about when ICANN would finally get back on track. "We’re all focused on quality here and not just doing things in a hurry."

Of course he could have said: "I don't know. Sorry." But then that's not ICANN's style.