Coordinated Disclosure is Complicated

The problem of when to publicly disclose discovered vulnerabilities in control systems is a much debated topic. Theoretically, I think that everyone in the control system vender and owner world would prefer to see researchers coordinate their disclosures so that the vendor has a chance to correct the vulnerability and the owner/operators have a chance to fix their systems before the vulnerability becomes public knowledge. But even when a researcher is committed to coordinated disclosure things can get complicated.

Coordinated Disclosure or Not

Yesterday Adam Crain, a well-known and well documented researcher, posted an interesting blog entry on his web site. In it he discusses a method of dealing with recalcitrant vendors that apparently make no effort to correct a vulnerability in their product that has been identified by an independent researchers and coordinated with both the vendor and ICS-CERT.

Historically (an odd term to use is in this young field) one of the reasons given by many grey hat researchers for not coordinating disclosures is that vendors have ignored them or failed to take action to correct identified vulnerabilities. In general the control system industry has gotten much better at responding to coordinated disclosures. This has been a major contributor to the decline in the number of Alerts that ICS-CERT has had to publish recently.

Unresponsive Vendors

But for every trend, there is an outlier. In this case it appears that Adam has identified one such outlier. Now Adam and his partner Chris Sistrunk have been the poster children for the coordinated disclosure process with the series of DNP3 vulnerabilities in 28 products that they have reported over the last year or so. They have been active proponents of fuzz testing of control system components, but they have been scrupulous in coordinating the disclosure of their serious vulnerability discoveries with vendors and ICS-CERT. But, even Adam and Chris have their breaking point.

Hidden Components

What is particularly disconcerting in this case is that this is not a vulnerable device that can be exposed to the community and provide owner/operators with the choice of how to deal with their vulnerable systems. In this case the identified vulnerability is in software library that may have been used by any number of vendors in developing the software and firmware for a wide range of control system products.

And the system owner/operators have no way knowing if that library has been used in any of their devices, so there is no way for them to protect their systems or even know if their systems need protecting. Okay, I suppose there is a way; Adam has made his fuzzing tool available free of charge and an energetic and system savvy owner could probably use the tool to find the hidden vulnerabilities. I’ll have to ask Adam about how likely is that his fuzzer would find these particular vulnerabilities in an off-line control system (NOTE: Never fuzz test a live control system).

This is an ongoing problem in the control system community (okay in the entire Cybersecurity community) where few organizations have the necessary talent or time to produce every line of code in-house that goes into their control system components. When a control system device (or software) vender buys (or uses open source) software they also get any vulnerabilities in that software. Identifying those underlying vulnerabilities and tying them to all other uses of that code is complicated at best.

We, the ICS security community, need to develop a methodology for identifying and tracking down all of the vulnerable iterations of code that are identified in one place as being vulnerable and used in other systems and applications. The ICS ISAC SARA program is designed to address part of this problem, but I am not sure that it will be capable of going deep enough into the architecture of control systems to really help alleviate this problem.

Continuing the Fight

Adam, of course, is not going to rely solely on outsiders to get this problem fixed. While he has still not publicly identified the particular vulnerability he is doing more than just writing this blog post of his about this specific vender. He is also taking the information to the ICS community; outing the vendor as one of the uncooperative ones. Not only is he attempting to use community coercion to encourage cooperative compliance, but he is in effect warning other vendors that there may be a bug in their systems that needs to be addressed.

I wish him the best of luck, but I don’t expect to see much action, at least publicly. Very few vendors have gotten to the point that they self-identify vulnerabilities in their systems (Siemens is a major exception) even if the vulnerability comes in with outside code.

UPDATED 4-17-14 7:30 CDT - Adam's blog post got an interesting response  (see first comment) from the company involved. They did not like him outing them. Too bad. If they had just talked with ICS-CERT and Adam this whole thing would never have blown-up like this.