In OSINT, All Sources Aren't Created Equal

"In evaluating open-source documents, collectors and analysts must be careful to determine the origin of the document and the possibilities of inherent biases contained within the document."
- FM2-22.3: Human Intelligence Collector Operations, p. I-10

"Source and information evaluation is identified as being a critical element of the analytical process and production of intelligence products. However there is concern that in reality evaluation is being carried out in a cursory fashion involving limited intellectual rigour. Poor evaluation is also thought to be a causal factor in the failure of intelligence."
- John Joseph and Jeff Corkill "Information Evaluation: How one group of Intelligence Analysts go about the Task"

These two quotes illustrate the long-running problem that has plagued commercial cyber security reporting for many years. There are very few unclassified OSINT standards of source evaluation and even less for cyber threat intelligence; at least that I could find while doing research for this article. 

The field of cyber intelligence is fairly new and fortunately, thanks to the Software Engineering Institute at Carnegie Mellon and the work of Jay McAllister and Troy Townsend, we can take a credible look at the state of the practice of this field:

"Overall, the key findings indicate that organizations use a diverse array of approaches to perform cyber intelligence. They do not adhere to any universal standard for establishing and running a cyber intelligence program, gathering data, or training analysts to interpret the data and communicate findings and performance measures to leadership."
- McAllister and Townsend, The Cyber Intelligence Tradecraft Project

The one thing that isn't covered in their report is the issue of source validation and how that contributes to the validity or value of the intelligence data received. However they did write a follow-up white paper with Troy Mattern entitled "Implementation Framework - Collection Management (.pdf)" 

Please take some time to study the framework and read the white paper. It's an ambitious and very thorough approach to helping companies understand how to get the most value from their cyber intelligence products. Unfortunately, while it specifies data evaluation and source validation, it doesn't provide any specific guidelines on how to implement those two processes.

Fortunately, there has been some great work done on source analysis for Human Intelligence (HUMINT) that I believe can be applied to Cyber intelligence and OSINT in general. It's a paper written by Pat Noble, an FBI intel analyst who did his Masters work at Mercyhurst University's Institute for Intelligence Studies: "Diagnosing Distortion In Source Reporting: Lessons For HUMINT Reliability From Other Fields"

A PowerPoint version of Noble's paper is also available. Here are a few of the slides from that presentation:

We recognize these failings when it comes to human intelligence collection but for some reason we don't recognize them or watch for them when it comes to OSINT. The crossover application seems obvious to me and could probably be easily implemented. 

I started this article with a quote from the Army Field Manual FM2-22.3: Human Intelligence Collector Operations (.pdf). Appendix B in that manual contains a Source and Information Reliability Matrix which I think is also applicable to Cyber intelligence or any analytic work that relies upon open sources.

I think a graph like this could be applied with very little customization to sources referenced in cyber intelligence reports or security assessments produced by cyber security companies. 

The West Australian Police Force study by John Joseph and Jeff Corkill "Information Evaluation: How one group of Intelligence Analysts go about the Task" recommended the use of the Admiralty Scale which is identical to the Army's matrix shown above:

Again, these scales were developed to evaluate human sources, not published content, but they certainly seem applicable with some minor tweaking. 

It's important to note that only part of the problem lies in the lack of source evaluation methods. Another very large contributing problem is the lack of standardized cyber intelligence tradecraft pointed out by McAllister and Townsend in their Cyber Intelligence Tradecraft paper:

"Tradecraft: Many government organizations have adopted the intelligence community standard of consistently caveating threat analysis with estimative language and source validation based on the quality of the sources, reporting history, and independent verification of corroborating sources. Numerous individuals with varying levels of this skillset have transitioned to cyber intelligence roles in industry and academia, but the practice of assessing credibility remains largely absent. The numerous analytical products reviewed for the CITP either did not contain estimative or source validation language, or relied on the third-party intelligence service providing the information to do the necessary credibility assessment." (p.11)

And of course due to the newness of the field there's no standard yet for Cyber Intelligence training (McAllister and Townsend, p. 13). 

IN SUMMARY

There are numerous examples of cyber security reports produced by commercial and government agencies where conclusions were drawn based upon less than hard data, including ones that I or my company wrote. Unless you're working in a scientific laboratory, source material related to cyber threats is rarely 100% reliable. Since no one is above criticism when it comes to this problem, it won't be hard for you to find a report to critique. In fact, it seems like a different information security company is issuing a new report at least once a month if not once a week so feel free to pick one at random and validate the sources using any of the resources that I compiled for this article. 

If you know of other source evaluation resources, please reference them in the comments section. 

If you're a consumer of cyber intelligence reports or threat intelligence feeds, please ask your vendor how his company validates the data that he's selling you, and then run it through your own validation process using one of the tools provided above. 

I'd love to hear from any readers who implement these suggestions and have experiences to share, either in confidence via email or in the comments section below.

UPDATE (11/24/13): A reader just recommended another excellent resource: Army Techniques Publication 2.22-9 "Open Source Intelligence". It discusses deception bias and content credibility, both of which must be accounted for in source validation.

Add to Cart View detail

Where's the "Strike" in CrowdStrike?

I've had mixed feelings about CrowdStrike from the moment that it launched in stealth status last February. On the one hand, I'm a big fan of how Shawn Henry (President of CrowdStrike Services) helped move the FBI from a terribly incompetent position vis a vis cyber investigations (circa 2005-06) to one of the world's premiere cyber investigative bodies in just a few short years. On the other hand, I detest McAfee and I've openly ridiculed their so-called "reports" on more than one occasion. As an Israeli friend of mine put it, Anti-Virus companies aren't security companies. And I might add, they aren't intelligence organizations either. The one thing that McAfee does have are rich executives, including CrowdStrike co-founders Gregg Marston, Dmitri Alperovich, and George Kurtz who arranged CrowdStrike's $26 million Series A funding from Warburg Pincus where Kurtz was an Executive-in-Residence after McAfee was acquired by Intel for $7.86 Billion in cash.

A LinkedIn search shows that the company has been attracting/recruiting lots of talent but so far they haven't announced much in the way of a product line. They did launch an open source reverse-engineering portal called CrowdRE which lets anyone play with a highly regarded Disassembler called IDAPro in a cloud-based server. The benefit to CrowdStrike is that in exchange for providing the portal, it can quickly grow a database of reverse-engineered malware that it can utilize on behalf of its paying customers.

The question that I and others have been asking since last February's launch has to do with the "offensive" hook that CrowdStrike advertises via its tag line "You don't have a malware problem. You have an adversary problem"(tm). The company website claims to offer "Enterprise Adversary Assessment" where "we identify the adversary and find out what they're after." And how do they do that? Back to the website: "Through hunting operations, including host-based detection, threat-specific network analysis, and victim threat profiling".

In case you have any doubt as to who the adversary is, their cool t-shirt makes it pretty clear:

Gee, what a surprise. CrowdStrike has determined that the adversary is China. And that's a continuation of the piss-poor intelligence that Dmitri Alperovich published while at McAfee: Operation Shady Rat (China), Operation Aurora (China) and Operation Night Dragon (China). There's over 30 nation states developing computer network attack, defense, and exploitation capabilities and at least a dozen that are highly proficient and actively conducting cyber espionage yet somehow McAfee's "intelligence analysts" only see China. Not Israel, Russia, Taiwan, France, Germany, or South Korea - just the PRC. In a video interview, CrowdStrike's Director of Intelligence Adam Meyers talks about identifying adversaries via toolmarks and the usual TTPs that every so-called cyber intelligence firm narrowly focuses their attention on but that's not analysis (See Michael Tanji's recent article on the subject "Malware Analysis: The Danger of Connecting The Dots"). In the intelligence community, that's a cognitive trap known as target fixation. If after looking at all of the technical parameters, the only nation state that you see is China, you need to find another job because you suck as an intelligence analyst.

Getting back to CrowdStrike's "offensive" marketing theme, in Shawn Henry's keynote at BlackHat last summer, he made it clear that CrowdStrike wasn't advocating hacking back; that such activities were still illegal. CrowdStrike's latest high profile FBI hire Steven Chabinsky has also made it clear that the laws currently don't support even something as mild as a company encrypting its own data found on a foreign server. So what's the point in promoting a "take the fight to the adversary" approach when it's impossible to do in the current legal climate?

The bottom line is that, in my opinion, CrowdStrike cannot currently deliver anything unique in the infosec space that Mandiant and other companies aren't already doing unless it significantly improves its sources and methods regarding identifying adversary state and non-state actors and pushes the envelope on active defense. It's not enough to have a cool t-shirt that says "Change the Game". They literally have to do it.

Add to Cart View detail

The Poor State of Cyber Intelligence

I recently had the privilege of speaking at a government cyber conference which was sponsored by one of the three-letter agencies and which included analysts from all 16 agencies that comprise the U.S. Intelligence Community (IC). Besides myself there were a number of other well-known and well-respected speakers. My session focused on Russia and their technology priorities, but the first question that the moderator asked me had to do with the the fact that I was apparently wrong regarding who created Stuxnet. His point in raising that issue was not to embarrass or shame me but to have me talk about how intelligence analysts must not be afraid to be wrong; about how important the role of negative analysis is along with the dangers associated with mirror imaging (i.e., a cognitive trap in which an intelligence analyst imagines that the target thinks like he does). Another cognitive trap is target fixation, where an analyst becomes fixated on one hypothesis and only sees the evidence that supports it. I see "cyber intelligence" analysts falling into that trap almost all the time.

Regardless of the problems faced by trained analysts in the IC, the state of cyber intelligence as its practiced by information security practitioners and others who are not trained in the science of rigorous analysis is often exponentially worse. The word "intelligence" is used to describe everything from a clipping service to threat data. The only thing worse are the marketing pitches promoting what their so-called "cyber intelligence" product will do for the customer - which is everything short of bringing him to orgasm. Don't call the result of your work analysis if you haven't performed any negative analysis to test your hypothesis. Call it conjecture, or opinion, because that's what it is.

I'm writing a chapter on this topic for my next book "Assumption of Breach" and my paper on the same subject will soon be published by the U.S. Air Force so I'm not going to go into further detail here except to say that if cyber intelligence analysts want to do justice to their craft, I encourage them to read Dick Heuer's "Psychology of Intelligence Analysis" (.pdf) and find ways to apply it to their work in the cyber field. Another excellent resource is "Understanding Rigor in Information Analysis". Right now, between mirror-imaging and target fixation, many cyber intelligence analysts are missing huge gaps in the threat landscape and are doing a great disservice to both their customers and their craft.

Add to Cart View detail

Intelligence on Russian Information Warfare Activities

Threat Intelligence and Cyber Intelligence are phrases that are tossed around both frequently and casually these days. Threat intelligence as it's used by the information security community has to do with malware and malicious IPs. Cyber intelligence is used even more loosely and may cover everything from Threat Intelligence to discovering who the members of Anonymous are. My company Taia Global Inc. has been providing highly targeted open source intelligence reports on foreign corporations' government connections as well as the information warfare activities of individual nation states since 2009. Since most of our foreign government clients are interested in the IW activities of the Russian Federation, we focus a lot of attention there. Here is what we've produced in the last few months alone:

• Center for Computer Emergency Response of the Russian Federation (RU-CERT)
• Roskomnadzor and the Cyber Vigilantes
• Russian Federal Security Service Center for Electronic Surveillance of Communications - Military Unit (Vch) 71330
• Russian Federation Security Council and the Evolution of Russia’s Information Security Doctrine
• Federal State Unitary Enterprise Scientific Research Institute Kvant (Federal Security Service)
• Federal Security Service (FSB) Internet Monitoring Vendors
• Federal Security Service (FSB) Administrative Centers for Information Security

Apart from these specialized reports, we also produced the 2011 Russian Federation Information Security Reference.

If Russia is an important piece of your organization's business or security plans and you'd like more information about our intelligence services for the Russian Federation or other countries in Asia, the EU or elsewhere, you can contact us via the Taia Global website.

Add to Cart View detail