Interview with Jeff Moss over ICANN software glitch

The following video was released by ICANN on Thursday 19 March, one week after it has taken its application system for new Internet extensions offline. A full transcript appears below the video.

Brad White (BW), ICANN Director of Media Affairs: Jeff, you're ICANN's Chief Security Officer. You were brought in to look at this glitch in the new gTLD application system, to see if it was in anyway connected with hacking or a cyber attack. What did you determine?

Jeff Moss (JM), ICANN Chief Security Officer: We didn't find anything. So we analyzed all the logs and looked at any other indicators that might suggest an intrusion, unusual activity, network activity. We didn't find anything.

BW: So at this point, no indication of that?

JM: Correct.

BW: Why did you take the system offline?

JM: Well, it was safest thing to do. So generally without knowing if it's a security incident, without knowing if there's a... Other persistent and may be a data corruption problem, the safest thing to do is to take it offline. The problems that, had we kept the system running, only to find out it's a bigger problem down the road would have been kind of catastrophic for us. So it's the safest, most prudent thing to do.

BW: What exactly is that problem?

JM: Well, under certain circumstances that were hard to replicate, users that had previously deleted a file could end up seeing the file name of another user who had uploaded the file. It's a little complicated, but what it really means is certain data was being revealed to users, who are not seeking the data that would just show up on their screen.

BW: So if I understand what you're saying correctly, some applicants could see the user names and file names of other applicants.

JM: Correct. But they wouldn't be able to see the contents of the file because they didn't create the file, the file commissions would prevent them from looking inside the file. But they could see the name.

BW: How many applicants had the ability to view the user names or file names of others?

JM: So it's definitely a minority of incidences. And we're in the process of winnowing that down even more, to be able to tell you with a much more specificity, exactly how many people.

BW So you're trying to determine exactly how many people could see the file names and user names. And let me ask you the opposite question. How many users had these file names and user names potentially exposed to other applicants? Do you know that figure yet?

JM: Yes, we know that figure. I'm not going to tell you that figure because we are verifying it.

BW: So you are still trying to confirm that.

JM: Right. So we're manually verifying it through two independent processes, and when the two independent processes match and get us the same result, we have high enough confidence that we can reveal that information.

BW: You've explained what happened. Do you know what caused the issue?

JM: So we're very confident that we understand what caused the issue, and we've corrected the issue. So what we're doing now is we're going back through all of our logs and analyzing exactly who was affected, and when. So with confidence we'll be able to notify everybody. Were they at risk ever, of having someone see their file name, or weren't they?

BW: So you will be able to determine, it is your hope, not merely the numeric values, how many file names and user names were visible, but who might have had the ability to actually see those file names and user names?

JM: Correct. We're putting everybody on notice that we know what file names and user names were displayed to what people were logged in, and when. And we want to do this very publicly because we want to prevent any monkey business. We are able to reconstruct which file names and user names were displayed.

BW: So it is your hope to actually be able to determine, with a fair degree of accuracy, how much of a real problem this is.

JM: Correct. We'll know very specifically.

BW: Once you have the information, if I'm an applicant, and there's even a chance that my user name or file name might have been seen, will you notify me?

JM: We're going to notify everyone. Whether yours was potentially seen or not, we'll notify everyone and tell them the status.

BW: Was any applicant information ever compromised?

JM: We have nothing to point to a compromise. We have no indicators of compromise. Everything points to a problem in the way that interrupted file deletions occurred.

BW: ICANN has publicly stated that the first report of this problem was March 19, almost a month before this system was taken offline. Why the delay?

JM: Well, in hindsight now, we've realized that the 19th was the first expression of this problem. But at that point in time, we had no reason to believe that there was a problem. The information that was displayed didn't make any sense to the applicant. Those appeared to be random numbers. And we never heard back anymore further information. We couldn't reproduce this problem. So only later on, when we started to see the glitch express itself did we realize that, "Aha, something happened on the 19th that was related." But at that point there were no dots to connect.

BW: Is this the sort of problem we could have looked for in pre-launch testing?

JM: We do look for everything in pre-launch testing, and this problem did not express itself. So we're going back and analyzing exactly what our test suites were and what our test cases were to find out why wasn't this caught in testing. But we ran a pretty comprehensive test suite. And you noticed the problem only emerged at the very end of the... Toward the end of the window. Why didn't it express itself sooner? We're trying to understand that.

BW: What has to happen before you reopen the application window?

JM: We have to make sure that the patch we've developed is 100% accurate, and it addresses the underlying problem. Because remember it expressed itself in several different ways, and we want to make sure we don't get bitten by this bug again.

BW: Why is this testing that you keep talking about such a prolonged endeavor?

JM: Well, the stakes are very high, and we want to be absolutely certain when we make statements, that we understand and don't have to retract what we are saying. So we don't want to give any misinformation inadvertently to the public, and we ourselves want to understand precisely what's occurring. As we found, the problem has several ways it can express itself. And if you look at the history of this bug, this glitch. We would solve it one way, and it would appear another way. We would solve it another way, and it would appear a third way. And at some point, we were just uncomfortable feeling that we understood the core issue. And that's when we took the system offline.

BW: You seem to be telling me that if ICANN is not spewing forth a whole lot of information, it's because you want to be sure you're right before you go public.

JM: Right, we don't want to withhold any information. We want to be transparent. But we also don't want to speak just for the nature of speaking. We want to make sure that what we say is factual and accurate because people are making big business decisions based on this information. And we don't want to cause any more concern than what's already happening.

BW: How high of a priority is the resolution of this problem within ICANN?

JM: This is the number one priority at ICANN. For anybody involved in the new gTLD program, they're focused on this problem 24/7. What we're trying to do is to be as accurate as possible. We don't want to have another problem reoccur. We want to be able to speak with authority when we tell all the applicants who has been affected and who hasn't. This is a time consuming problem, and what we're trying to do is do this right instead of trying to do it fast, and I hope everyone understands that.