Friday, July 21, 2017

Something is very wrong with the Belfer bug rediscovery paper

This is what the paper says about its Chromium data:
Chrome: The Chrome dataset is scraped from bugs collected for Chromium, an open source software project whose code constitutes most the Chrome browser. 20 On top of Chromium, Google adds a few additional features, such as a PDF viewer, but there is substantial overlap, so we treat this as essentially identical to Chrome.21 Chrome presented a similar problem to Firefox, so to record only vulnerabilities with a reasonable likelihood of public discovery, we limited our collection to bugs labeled as high or critical severity from the Chromium bug tracker. This portion of the dataset comprises 3,397 vulnerability records of which there are 468 records with duplicates. For Chrome, we coded a vulnerability record as a duplicate if it had been merged with another, where merges were noted in the comments associated with each vulnerability record, or marked as a Duplicate in the Status field.

The problem with this methodology is simply that "merges" do not indicate rediscovery in the database. The vast majority of the findings relied upon for the paper are false positives.

To look at this, I went through their spreadsheet and collected out those mentioned 468 records. Then I examined them on the Chromium bug tracking system. The vast majority of them were "self-duplicates" from the automated fuzzing and crash detection systems.
I'm a Unix hacker so I converted it to CSV and wrote a Python script to look at the data. Happy to share scripts/data.

Looking at just the ones that have a CVE or got a reward makes more sense. There's probably only 45 true positives in this data set (i.e. the ones with CVE numbers). That's 1.3% which agrees with the numbers from the much cleaner OpenSSL Bug Database (2.4%) from this paper.

Example false positives in their data set:

  • This one has a CVE, but doesn't appear to be a true positive other than people noticed things crashed in many different ways from one root cause.
  • Here someone at Google manually found something that clusterfuzz also found.
  • Here is another clear false positive. Here's another. Literally I just take any one of them, and then look at it.
  • Interesting one from Skylined, but also a false positive I think.

Thursday, July 20, 2017

Decoding Kasperksy

Although various internet blowhards are hard at work asking for "More information to be released" regarding why the US is throwing Kaspersky under the bus, that's never going to happen. It's honestly easiest to get in the press by pretending to be in disbelief as to what the United States is doing in situations like this.

I say pretend because, it's really pretty clear what the US is saying. They are saying, through leaks and not-so-subtle hints, that Kasperksy was involved in Russian operations. It's not about "being close to the Kremlin" or historical ties between Eugene Kaspersky and the FSB or some kind of DDoS prevention software. Those are not actionable in the way this has been messaged at the highest levels. It's not some sort of nebulous "Russian Software" risk. It's about a line being crossed operationally.

The only question is whether you believe Eugene Kaspersky, who denies anything untoward, or the US Intelligence Community, which has used its strongest language and spokespeople as part of this effort and has no plans to release evidence.

And, in this particular case, the UK intel team (which has no doubt seen the evidence) is backing the US up, which is worth noting, and they are doing it in their customary subtle but unmistakable way, by saying at no point was Kaspersky software ever certified by their NCSC.

The question for security consultants, such as Immunity, is how do we advise our US-based clients - and looking at the evidence, you would have to advise them to stop using Kaspersky software. Perhaps your clients are better off with VenusTech?

I'm pretty sure this AV company is deceased!

Bugmusment 2017

The Paper itself:

Note that the paper in the selected area would be TS/SCI for both us and China. :)

(To be honest, I don't think it even does this)

Cannot be true.

Ok, so I can see how it went. After the Rand paper on 0day collisions came out, existing paper writers in the process of trying to point out how evil it was the Government knew about 0day were a bit up a creek without a paddle or even a boat of any kind.

Because here's the thing: The Rand paper's data agrees with every vulnerability researcher's "gut feeling" on 0day collision. You won't take a 5% over a year number to a penetration testing company and have them say, "NO WAY THAT IS MUCH TOO LOW!"

But if you were to take a 20% number to them, they would probably think something was wrong with your data. Which is exactly what I thought.

So I went to the data! Because UNLIKE the Rand paper, you can check out their GitHub, which is how all science should work. The only problem is, when you dig into the data, it does not say what the paper says it does!

Here is the data!

From what I can tell, the Chromium data is from fuzzers, which naturally collide a lot. Especially when in most cases I can click on the rediscovery is from the exact same fuzzer, just hitting the same bug over and over in slightly different ways. The Android data I examined manually had almost all collisions from various libstagefright media parsing bugs, which are from fuzzers. A few seemed to be errors. In some cases, a CVE covers more than one bug, which makes it LOOK like they are collisions when they are not. This is a CVE issue more than anything else, but it skews the results significantly.

Ok, so to sum up:
The data I've looked at manually does not look like it supports the paper. This kind of research is hard specifically because manual analysis of this level of data is time consuming and requires subject matter experts.

It would be worth going in depth into the leaked exploits from ShadowBrokers etc. to see if they support any of the figures used in any of the papers on these subjects. I mean, it's hard not to note that Bruce Scheier has access to the Snowden files. Maybe there are some statistics about exploits in there that the rest of us haven't seen and he's trying to hint at?

This was the paragraph in the paper that worried me the most. There is NO ABILITY TO SCIENTIFICALLY HAVE ANY LEVEL OF PRECISION AS CLAIMED HERE.

Wednesday, July 19, 2017

An important note about 0days

For some reason the idea that patches == exploits and therefor any VEP-like program that releases patches is basically also trickling out exploits is hard to understand if you haven't done it.

Also, here's a very useful quick note from the head of Project Zero:

Issues with "Indiscriminate Attacks" in the Cyber Domain


The fundamental nature of targeting in the cyber domain is very different from conventional military standards. In particular, with enough recon, you can say to a high degree "Even though I released a worm that will destroy every computer it touches, I don't think it will kill anyone or cause permanent loss of function for vital infrastructure."

For example, if I have SIGINT captures that say that the major hospitals have decent backup and recovery plans, and the country itself has put their power companies on notice to be able to handle computer failures, I may have an understanding of my worm's projected effects that nobody else does or can.

Clearly another historical exception is if my destructive payload is only applicable to certain very specific SCADA configurations. Yes, there are going to be some companies that interact poorly with my exploits and rootkit, and will have some temporary damage. But we've all decided that even a worm that wipes every computer is not "destroying vital infrastructure" unless it is targeted specifically at vital infrastructure and in a way that causes permanent damage. Sony Pictures and Saudi Aramco do still exist, after all, and they are not "hardened targets".

The main issue is this: You cannot know, from the worm or public information, what my targeting information has told me and you cannot even begin to ask until you understand the code. Analyzing Stuxnet took MONTHS OF HARD WORK. And almost certainly,  this analysis was only successful because of leprechaun-like luck, and there are still many parts of it which are not well understood.

So combine both an inability to determine after-the-fact if a worm or other tool was released with a minimal chance for death or injury because you don't know my targeting parameters with the technical difficulty of examining my code itself for "intent" to put International Law frameworks on a Tokyo-level shaky foundation. Of course, the added complication is that all of cyber goes over civilian infrastructure - which moots that angle as a differentiating legal analysis.

Many of the big governmental processes try to find a way to attach "intent" to code, and fall on their face. The Wassenaar Arrangement's cyber regs is one of them. In general, this is a problem International Law and Policy students will say is in every domain, but in Cyber, it's a dominant disruptive force. 

In other words, we cannot say that NotPetya was an "indiscriminate weapon". 

Monday, July 17, 2017

The Multi-Stakeholder Approach

I was struck listening to this policy panel as it weaved and dodged to avoid really confronting the hard questions it raised.

Asking for lexicons is weak sauce.
True enough.
He wants to have separate conversations on everything because that seems simpler. But it's a mirage - everything is connected. This is the kind of policy plan that works fine until any rubber has to hit any road whatsoever.
Jane Holl Lute never has anything interesting to say on Cyber. Here's a review of her BlackHat Keynote: click. It's pretty telling about the policy community that she still gets on these panels.
Ok, here are some of the hard questions that got left on the cutting room floor:

What can we do about having the information security community not trust the USG when it comes to what we say? 

What are the values of the information security community, and what would it mean to respect them in a multi-stakeholder environment on the Internet? 

What are we going to do if we can't detangle all these issues?

Wednesday, July 12, 2017

What Kaspersky Means for Cyber Policy


Kaspersky has officially and unofficially denied any wrongdoing of any kind. But on the other hand, the recent actions by the US Government have not been subtle. The question is whether you believe McCain and Rubio and the IC over Eugene Kaspersky. It is clear from public reports that there is damning, but classified evidence which the US has no intention of releasing.

And there will be impact from the ban: While it's true that government agencies are "free" to still buy Kaspersky products, its unlikely any agency will do so, other than as a migration plan onto a GSA approved product.

If you've been to any US conference recently you've seen the sad sad Huawei booth, run by a "reseller" who would just as soon have the Huawei name removed from his equipment lines and unread brochures. This is what awaits Kaspersky in the US market, and there does not seem to be a way to fight it.

While this action only directly affects US Agencies (further bans may follow in legislation), it would be difficult to be a US Bank (aka, Critical Infrastructure) and continue using their software, and this could have widespread repercussions (as almost all banks are tightly connected and that is a huge market to lose for Kaspersky). Likewise, cyber insurance plans may require migrating off Kaspersky as a "known risk".

Examining what Kaspersky could have done to generate this reaction, you also have to note there are no mitigating factors available for recourse. The offer of looking at the source code means nothing since Kaspersky's AV is by definition a self-updating rootkit. So let's go over the kinds of things it  could have been:

  • Hack Back assistance (aka, "Active Countermeasures", as hinted at in the Bloomberg Report)
  • HUMINT cooperation (i.e. especially at their yearly Security Analyst Conference)
  • Influence operations (aka, ThreatPost, which is an interesting side venture for an AV)

The USG has not said what Kaspersky did that was so bad. What we've said is one clear thing: There is a line. Don't cross it.

As most of my friends say: It's about time.