What kind of military cyber team can't tell a fake ICS plant from a real one?

Evidently, the PLA is either the most incompetent Army in the world or is tasked with exploiting anything and everything that they can, including obvious honey pots. A paper and BlackHat talk by Kyle Wilhoit of Trend Micro got a lot of press including this article at MIT Technology Review "Chinese Hacking Team Caught Taking Over Decoy Water Plant".

My first reaction when I saw this headline was why would anyone bother? Every ICS expert that I know discounts the potential harm that a hacker might be able to do against a water system. My second reaction was - How the f__k would a hacker who knows SCADA systems not know that he was attacking a fake water plant?

I asked my friend Dale Peterson, a world-renowned authority in this area, the same question and he was as perplexed as me. A friend of his who attended BlackHat agreed. "Have you ever seen a plant with one pump?", he asked?

So what does this mean? In my opinion, it raises questions about who Comment Crew aka APT1 aka PLA Unit 61398 really is because they clearly don't know shit about Industrial Control Systems.

Add to Cart View detail

APT1, Shanghai Jiao Tong university, and Xenophobia

A few things have caught my attention recently which I'd like to share with you all in a somewhat abbreviated manner (meaning I'm swamped but this is important):

A Security Engineer's Forensic Review of Mandiant's APT1 report

Please read this security engineer's forensic review of the evidence contained in Mandiant's Appendix. He's discovered a lot more evidence which casts doubt on Mandiant's conclusions.

Shanghai Jiao Tong University's Collaboration with U.S. InfoSec Companies

Shanghai Jiao Tong University School of Information Security Engineering is just that - one of many Chinese universities that teaches information security. It is not a PLA school nor does it engage in hacking attacks. If it did, then I doubt that BreakingPoint Systems, a company that conducts "cyber warrior training" and does "cyber range deployments" for the U.S. government would have signed a "strategic cooperation agreement" with them.

Mandiant CSO Richard Bejtlich's view on Hiring Foreign Nationals

While I've disagreed often with Mandiant and Richard Bejtlich's views on China, I never heard him say anything remotely as awful as this quote from the Washington Examiner. I hope he was misquoted:

Bejtlich said he opposed placement of any foreign citizen of a suspect country like China in any sensitive government position.
"If you're considering them for a job at a national lab or a government agency, I think we're at the point now where it's recognized that's probably not a good idea," he said.

If that's an accurate quote, I can only hope that U.S. companies will ignore that incredibly poor advice. I think that most intelligent people in today's globalized economy have experienced working side by side with honest, talented, and skillful "foreigners" in many high technology settings including national labs and other environments. In fact, the U.S. would be hard-pressed to continue to innovate without them. The above quote is an example of xenophobia that's not far removed from McCarthyism and other witch-hunts and it has no place in the U.S. in 2013.

Add to Cart View detail

Mandiant's APT1 "Mission" problem

Mandiant's APT1 report's table of proof listed six categories that Mandiant deduced tied APT1 to PLA Unit 61398. The first, which Mandiant called the Mission area, made the claim that PLA Unit 61398 "targets strategic emerging industries in China's 12th Five year Plan" (see table 12 on p.59). Earlier in the report the authors claimed that "APT1 has targeted at least four of the seven strategic emerging industries that China identified in its 12th Five Year Plan" (p.24).

The Mission evidence is particularly of interest to me because I've been mining adversary state R&D since 2009 and while knowing what a potential adversary state is after is important, it cannot be done at the 50,000 foot view which is what China's Five Year Plans do. Taia Global published a white paper almost a year ago (a copy of which was requested by one of Mandiant's executives) which provided a similar high level look at 13 nation state R&D priorities and it too was not sufficiently granular to be of much use in an attribution effort however it does make clear that certain technologies are of value to at least a half dozen threat actors (see below). And frankly, this is a very valid approach, if done properly, to help a company understand which files may be at risk. In fact, that's precisely what Taia Global's new product Chimera is being developed to do. However, it's not enough to just say that because "energy" is part of China's FYP, then it must be China whenever an energy company is attacked. France, Germany, and Russia are also spending money on Energy related research and all three of those states have engaged in industrial espionage. But even that's not sufficient evidence to blame a state actor. What's more likely in my opinion is that a professional hacker group is making money by stealing valuable IP and selling it to competitors, state-run companies, and/or the states themselves.

Here are the seven new strategic industries identified in China's 12th FYP. The report didn't disclose which 4 of 7 were targeted:

• Energy conservation and environmental protection industries
• New-generation IT industry
• Biological industry
• High-end equipment manufacturing industry
• New energy industry
• New material industry
• New-energy automobile industry

Below are some of the R&D priorities for six other nation states who have engaged in industrial and cyber espionage. It's not exhaustive but it illustrates how little deviation there is at the broadest level of international R&D. We can safely say that companies in these industry segments are being targeted for their IP. We can't say that only China is doing the targeting.

France:

• Energy
• Biotechnology
• IT (Information Technology)
• Space
• Transportation

Germany:

• Energy
• IT and Telecommunications
• Manufacturing
• Biotechnology
• Medicine
• Climate research

Israel:

• Telecommunications
• Medicine
• Chemistry
• Information Technology
• Biotechnology
• Nanotechnology

Pakistan:

• Telecommunications
• Agriculture
• Medicine
• Education

Russia:

• Energy
• Robotics
• Information and Telecommunications
• Nanotechnology
• Life sciences
• Environment

South Korea

• Manufacturing
• Nanotechnology
• Semiconductors
• Transportation
• Chemicals

Add to Cart View detail

More on Mandiant's APT1 Report: Guilt by Proximity and Wright Patterson AFB

The blog post that I wrote earlier in the week "Mandiant Report APT1 Has Some Critical Analytic Flaws" was based upon my history of interacting with some Mandiant folks online and in person as well as my interpretation of the facts as they were presented in the report. Thanks to some feedback that I received from readers as well as a teleconference that I had with three Mandiant executives yesterday, I've learned some new things that color my earlier article.

1. Mandiant has expanded their original definition of APT

Yesterday, I spoke with three Mandiant executives and learned that their meaning of the term has evolved with the times and it no longer represents a Who, but a What; or more precisely, a well-documented multi-staged process that attackers from multiple nation states have adopted. Mandiant has not formally announced this change (although they probably will later this year) so when I wrote my article on their APT1 report, I was referencing their former definition which I know now is no longer in use. While Mandiant often sees Chinese hackers at work stealing trade secrets and intellectual property, they also acknowledge that other countries may be doing the same thing. I'm happy to report this change because it's been a point of contention between myself and some folks at Mandiant ever since 2010. I'm glad that we're closer to being on the same page.

2. Mandiant did some negative analysis before publishing their report

Another thing I learned from that phone meeting was that there was an effort made to look at alternative  scenarios that might explain the facts that Mandiant had before them. Mandiant isn't a part of the Intelligence Community (even though they have some ex-IC folks working there) and they don't have the time, resources, or manpower to do the same type of analysis that is performed at Langley. It's also not their mission to do nation state attribution so I want to give them at least some credit for the counter-analysis that they did do, even though the significance of their conclusion demanded a more rigorous methodology in my opinion.

Thanks to input from my readers, I've also learned some additional negatives about the report.

1. Mandiant's reliance on proximity to prove its claim that PLA Unit 61398 is Comment Crew aka APT1 is harmed by simple geographical mistakes such as:

• p.10 of Mandiant's report refers to Hebei as a borough in Shanghai. Hebei is actually a province about 600 miles and 3 provinces away from Shanghai.
• NEC and Intel along with many other high tech companies operate less than 8 miles from PLA Unit 61398 and all would be served by the same fiber optics cable provided by China Unicom.
• There are more free proxy servers in China than anywhere else in the world and some of those proxy servers overlap with the IP blocks identified in the Mandiant report. 
• An IP registration for UglyGorilla was described by Mandiant as being "across the river" from Unit 61398. In fact, it was 33 kilometers away.

2. Speaking of guilt by proximity, one of the "obviously false" IP address registrations according to Mandiant was for an address in Yellow Spring, Ohio. It should have been spelled "Yellow Springs". However, a cursory check shows that the address is real except for that one missing "s". Even more interesting is that it is located 13 miles from Wright-Patterson Air Force Base which is the Air Force's "boot camp for cyber warriors".

Directions via Google Maps

Either this is a bizarre coincidence or someone on the Comment Crew has a wicked sense of humor. As it turns out, Michael Murphy is a real person who lives in Yellow Springs, Ohio and who used to be the director of admissions at Antioch College whose office is located at 795 Livermore St., Yellow Springs, OH - the address that Mandiant assumed was fake.

3. (UPDATED 23 FEB 13)  On page 11 of the report, under "Size and Location of Unit 61398's Personnel and Facilities", Mandiant wrote "public sources confirm that in early 2007, Jiangsu Longhai Construction Engineering Group completed work on a new building for Unit 61398 located at Datong Road 208 within the Pudong New Area of Shanghai. At 12 stories in height and offering 130,663 square feet of space, we estimate that this building houses offices for approximately 2,000 people." In reality, it's the Unit's pre-school:

English translation via Google Translate

And this isn't all of the errors. It's just a fraction. While each may seem minor, collectively they call into question Mandiant's final conclusion and, at the very least, should serve as a lesson to policy makers not to rush to judgment on matters of attribution. There's plenty of evidence that China engages in cyber espionage without upping the ante by trying to claim the Peoples Liberation Army is involved. 

At the end of the day it's important to remember that Mandiant isn't a U.S. government agency nor are they trained to do intelligence collection and analysis at the same level that it's done at Langley. They're a group of highly skilled professionals who serve their customers as incident responders and have a well-deserved reputation for excellence. 

Add to Cart View detail

Mandiant APT1 Report Has Critical Analytic Flaws

Mandiant's APT1 report is the latest infosec company document to accuse the Chinese government of running cyber espionage operations. In fact, according to Mandiant, if a company experiences an APT attack, then it is a victim of the Chinese government because in Mandiant-speak, APT equals China.

"We tend to perceive what we expect to perceive" 

- Richard J. Heuer, "The Psychology of Intelligence Analysis

The fact that Mandiant refuses to acknowledge that other nation states engage in cyber espionage when the facts show otherwise demonstrates what Heuer calls an "expectation bias", but it's much worse than that.

Mandiant's alleged proof is summarized in Table 12 (pp. 59-60): "Matching characteristics between APT1 and Unit 61398". Mandiant's entire premise that APT1 is PLA Unit 61398 rests on the connections made in that table and that no other conclusion is possible:

"Combining our direct observations with carefully researched and correlated findings; we believe the facts dictate only two possibilities: Either a secret, resourced organization full of mainland Chinese speakers with direct access to Shanghai-based telecommunications infrastructure is engaged in a multi-year, enterprise scale computer espionage campaign right outside of Unit 61398’s gates, performing tasks similar to Unit 61398’s known mission or APT1 is Unit 61398." (APT1, p. 60)

If this report were written by a professional intelligence analyst at CIA, it would most likely undergo a vetting process known as ACH (Analysis of Competing Hypotheses):

"Analysis of competing hypotheses, sometimes abbreviated ACH, is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve."

In other words, ACH forces the intelligence analyst to look for all alternative hypotheses and assess them one at a time to see which best fits the data collected. This is rarely if ever done by information security companies, and it's the single biggest objection that I have when it comes to individuals making claims of attribution to nation states. Heuer's iconic "Psychology of Intelligence Analysis" explains why ACH is so important:

"The way most analysts go about their business is to pick out what they suspect intuitively is the most likely answer, then look at the available information from the point of view of whether or not it supports this answer. If the evidence seems to support the favorite hypothesis, analysts pat themselves on the back ("See, I knew it all along!") and look no further. If it does not, they either reject the evidence as misleading or develop another hypothesis and go through the same procedure again. Decision analysts call this a satisficing strategy. (See Chapter 4, Strategies for Analytical Judgment.) Satisficing means picking the first solution that seems satisfactory, rather than going through all the possibilities to identify the very best solution. There may be several seemingly satisfactory solutions, but there is only one best solution." 

"Chapter 4 discussed the weaknesses in this approach. The principal concern is that if analysts focus mainly on trying to confirm one hypothesis they think is probably true, they can easily be led astray by the fact that there is so much evidence to support their point of view. They fail to recognize that most of this evidence is also consistent with other explanations or conclusions, and that these other alternatives have not been refuted."

If Mandiant or another organization were to use ACH on this evidence, here's how Heuer recommends it be done. It's an 8-step process:

1. Identify the possible hypotheses to be considered. Use a group of analysts with different perspectives to brainstorm the possibilities.
2. Make a list of significant evidence and arguments for and against each hypothesis.
3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyze the "diagnosticity" of the evidence and arguments--that is, identify which items are most helpful in judging the relative likelihood of the hypotheses.
4. Refine the matrix. Reconsider the hypotheses and delete evidence and arguments that have no diagnostic value.
5. Draw tentative conclusions about the relative likelihood of each hypothesis. Proceed by trying to disprove the hypotheses rather than prove them.
6. Analyze how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation.
7. Report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.
8. Identify milestones for future observation that may indicate events are taking a different course than expected.

I don't have the time to run Mandiant's evidence through an ACH process but I'd like to propose that a volunteer group of intelligence students at Mercyhurst Institute of Intelligence Studies do that very thing. My friend Professor Kris Wheaton who teaches there and writes the outstanding Sources and Methods blog is an expert in this area and I'm hopeful that he'll pick up the challenge.

In the meantime, the following table has four columns. The first three are from Mandiant's table 12. The "Other" column contains a partial group of alternatives that I've provided for each of Mandiant's "characteristics". These alternatives need to be analyzed and ruled out using a rigorous analytic process like ACH before Mandiant or anyone else can claim that APT1 is a part of China's Peoples Liberation Army.

In summary, my problem with this report is not that I don't believe that China engages in massive amounts of cyber espionage. I know that they do - especially when an executive that we worked with traveled to Beijing to meet with government officials with a clean laptop and came back with one that had been breached while he was asleep in his hotel room.

My problem is that Mandiant refuses to consider what everyone that I know in the Intelligence Community acknowledges - that there are multiple states engaging in this activity; not just China. And that if you're going to make a claim for attribution, then you must be both fair and thorough in your analysis and, through the application of a scientific method like ACH, rule out competing hypotheses and then use estimative language in your finding. Mandiant simply did not succeed in proving that Unit 61398 is their designated APT1 aka Comment Crew.

UPDATE (22 FEB 2013): I've published a follow up to this article: "More on Mandiant's APT1 Report: Guilt by Proximity and Wright-Patterson AFB"

Add to Cart View detail