From the beginning

The story of Ripple20 begins September 2019, when we decided to begin some part-time research into the Treck TCP/IP library. Within a few months, we realized there was some troubling data on potential vulnerabilities and contacted Treck to share our information as well as kick off a coordinated vulnerability disclosure (CVD) process. We understood from the beginning that Treck worked with many different vendors, with the potential to impact a large number of users. However, we simply had no idea of the scale and sheer magnitude of the situation, nor how complex the supply chain had become. 

Our outreach to Treck was initially challenging. Treck is a small company serving a niche clientele, and not generally available for individual consumer approach. They also never appear to have been the target of independent security research. Eventually, we managed to track down an email contact at Treck and received an introduction. We notified them that we found a critical security vulnerability in their TCP/IP stack library, and it was our preference to work directly with them. Initially, we were faced with minimal communication. We later learned that Treck was assessing the information internally and with their legal advisors. (We also noticed litigation lawyers checking us out on LinkedIn and elsewhere).

Subsequently, we started to contact a few select vendors (such as Digi, HP, Intel, and Quadros) that we knew used Treck products and its vulnerable TCP/IP stack library, and asked them to help us make the appropriate connection with Treck. After this vendor outreach, the ball started rolling with Treck.

Working with Treck

Once Treck understood the gravity of the vulnerabilities, we worked cooperatively together, and JSOF provided them with descriptions of the vulnerabilities and some other information.

It was important to us that Treck would make sure their clients were notified of the vulnerabilities at hand. Due to NDAs and other complexities, Treck was not able to provide us with a list of its clients and users of the code library. This was our first brush with some of the supply chain challenges. As a result, Treck ended up taking the lead on the disclosure process, as they were best suited to notify their clients and provide the appropriate patches.

Interestingly, the process of remediating the vulnerability led some of Treck’s clients to renew support contracts, thus Treck seems to have ended up profiting from the situation. When these companies were notified about Ripple20 and well understood the potential risks, they soon realized the necessity of ongoing maintenance and importance of access to patches. In the end, many of the companies worked with Treck to either renew their contracts or make other arrangements. This is a thought-provoking lesson for both small and large vendors wary of facing security issues. A proactive security response approach means clients have a reason to pay for maintenance and support in order to keep the software up to date.

Standard industry practice is not to publicize a vulnerability until there is a patch available to fix it. We agreed on a standard 90-day period within which Treck would fix the vulnerabilities and notify their clients of the patch. Treck had the patch available around the end of March – 45 days before the 90-day deadline, and informed us that they would be reaching out to all affected clients to inform them of these vulnerabilities. 

The disclosure was postponed twice after requests for more time came from some of the participating vendors, with some of the vendors voicing Covid-19-related delays. Out of consideration for these companies, the time period was extended from 90 days to over 120 days. Even so, some of the participating companies became difficult to deal with, as they made extra demands and some, from our perspective, seemed much more concerned with their brand’s image than with patching the vulnerabilities.

After some deliberation, June 16 was the date chosen for publicizing Ripple20. In the interim, once we understood the extent of the vulnerabilities and the sheer numbers involved, we focused on how to best use our time (120 days of patience) to identify and help to address all of the parties that could be at risk.

An international effort

Since Treck was unable to supply us with a comprehensive list of their clientele, we decided to create our own, in order to find out who was affected and ensure that everyone was notified and would have the vulnerability patched. We used a creative supply chain tracking approach, in an attempt to understand the scale of distribution and the extent of the problem. We soon began to realize and appreciate just how complex the supply chain picture really was.

It became clear to us that tracking the extent of Treck library distribution was too large for just one small team. We could track the supply chain trails, but we needed to work with international organizations to extend our reach within organizations and domains for which we had no access.

This is why the Ripple20 disclosure process is being coordinated and overseen by multiple national computer emergency response team (CERT) organizations and regulators. All are collaborating in order to reach as many affected vendors as possible before the vulnerabilities became public. Our collaborators initially included:

• The CERT Coordination Center (CERT/CC), the worldwide center for coordinating information about Internet security at Carnegie Mellon University. This is the first (and most well-known) CERT.
• The Cybersecurity and Infrastructure Security Agency (CISA), part of the Department of Homeland Security (DHS)

CERT groups focus on ways to identify and mitigate security risks. For example, they can reach a much larger target group of potential users with blast announcements, “mass-mailings” that they broadcast to a long list of participating companies to notify them of the potential vulnerability. Once users are identified, mitigation comes into play. While the best response might be to install the original Treck patch, there are many situations in which installing the original patch is not possible. CERTs work to develop alternative approaches that can be used to minimize or effectively eliminate the risk, even if patching is not an option.

JSOF assisted with coordinating the disclosure process, including providing proof of concept scripts for some of the vulnerabilities, suggesting mitigations, and providing additional lists of users of the Treck library. We tracked these users down in a variety of creative ways, including working with partners as well as some interesting open-source intelligence collections .

Working with international groups was essential in this case for another reason. Back in the 1990s, Treck collaborated with a Japanese company named Elmic Systems. Published information suggests the two companies co-developed the TCP/IP stack, though we are not sure exactly how this collaboration evolved over the years. Treck and Elmic later split apart and went their separate ways. Treck now markets this stack as Treck TCP/IP in the U.S., while Elmic Systems, now called Zuken Elmic, markets it as Kasago TCP/IP in Asia. This resulted in two separate branches of the TCP/IP stack devices – one managed by Treck and one managed by Elmic Systems – marketed in totally separate geographic areas, with no cooperation between them to the best of our knowledge, until we reported the vulnerabilities. We had tried to contact Zuken Elmic early on, but were not successful; they stopped replying after a short email correspondence. (We even tried to send some emails in Japanese!) Later on, CERT/CC was able to coordinate with JPCERT/CC, a Japanese national CERT organization, who is handling the follow-up with Elmic Systems and other affected companies in the Japanese/Asian supply chain. We have confirmed with Zuken Elmic that their TCP/IP stack is affected by some of the Treck vulnerabilities. Initial research shows Kasago to be in widespread use, providing the beginning of a completely different supply chain.

Teamwork: in everyone’s best interest

We have been collaborating with security vendors CyberMDX (medical device cybersecurity solution) and Forescout (device visibility and control). In our lab, we worked with the Treck TCP/IP library to identify the relevant network signatures. We then shared the network signatures with CyberMDX and Forescout. Each checked their own extensive client networks for these signatures, to identify more affected devices and components.

Note that this type of collaboration in the cybersecurity community benefits all participants, and is the first of its kind to our knowledge. We expanded the range of companies and devices that can be identified, while CyberMDX and Forescout are able to already provide their clients with mitigations and visibility, even before the issue is publicized. Additional companies were involved that have asked not to be named. These collaborations proved effective even though they were begun quite late in the disclosure process. We have no doubt that future collaborations of this type can be started earlier and provide immense value for all parties.

We also developed a script that companies can run themselves, to identify Treck products in their own networks. This will not be 100% effective at this stage, but can be an efficient, effective complementary approach, since it addresses the difficulty in identifying relevant users in a cloudy supply chain trail.

The bottom line is that teamwork benefits all players.