The story so far …
The attack on SolarWinds, whereby a sophisticated attacker modified the company’s tooling to inject code that then was shipped, had a far-reaching impact across major technology companies and US Government departments.
While much that has been reported has been factual, the company states speculation and misinformation has also been reported online. For example, a separate and unrelated insecure FTP server with the password ‘SolarWinds123’ also came to light. While the company acknowledges this is truthful and states it should not have happened, this problem was not related to the attack, it held no Orion code, it was not part of the company’s domain (hence not subject to the complex password policy in place) and was shut down within the hour it was discovered.
Additionally, some reporting has conflated the SolarWinds attack with the SolarWinds attacker. The same attacker has sought to compromise other organisations - testimony from Microsoft noted 60 clients with similar experiences. In fact, 30% of affected organisations were not SolarWinds customers at all. It’s important to separate those organisations being attacked by the same entity and those who were affected by the tainted SolarWinds code.
Tim Brown, SolarWinds Chief Information Security Officer, and Vice President for Security spoke one-on-one with iTWire to provide a first-hand account of the incident and the actions SolarWinds has taken and continues to take to move towards a more secure future, along with its focus on protecting customers and collaborating with industry and government entities to investigate and respond to the attack.
Brown oversees internal IT security, product security, and security strategy. He previously served as a former Dell Fellow and Chief Technical Officer and has over 20 years of experience developing and implementing security technology. He is a member of the advisory board for Clemson University, holds 18 issued patents on security-related topics, and is a trusted advisor to the White House.
As background information, “SUNBURST” is the name that has now been given by the security community to the hack. It used a supply chain attack to insert malicious code into SolarWinds’ Orion Platform, creating a backdoor through which malicious persons could gain entry into the internal environments of organisations running SolarWind’s Orion software.
While malicious attacks occur every day through phishing, malware, and other means, this Sunburst attack is especially shocking by weaponising software in such a dramatic and extremely sophisticated, coordinated way. It means software development practices must change; this truly is an inflection point.
On December 12, 2020, SolarWinds was notified it had shipped tainted code in its product. “People question did we know before then,” Brown said. “Absolutely not. We were shocked and dismayed on December 12.”
The warning came from FireEye who saw, inside their own network, attempts to communicate with a command-and-control centre. They looked deep into their environment, saw where the traffic was coming from and made the discovery it was a SolarWinds product.
The internal investigation began immediately. That date was a Saturday, but a critical response team came into the office on Sunday and didn’t leave for three weeks. SolarWinds brought in CrowdStrike for a thorough investigation, checking every workstation and server for signs of compromise and if the threat actor was still present.
“There were no signs the threat actor was present, nor any signs they moved laterally to any other product,” Brown said.
“The motivation was specific; the attack was to insert code, to have us ship that code, and be able to activate that code in a certain subset of companies and entities,” Brown said, stating the company had evidence from a series of sources including the research community that indicates the code was not activated in every environment it was executed in.
"It's important to understand what the code did," Brown said. "The Sunburst code had to first talk out to the Internet to be activated. It tried to reach a control and command server, then issue commands on the Windows server hosting Orion, then go sideways. There was intelligence in the code so it would only run in certain places. It wouldn’t run in the SolarWinds domain or certain address spaces owned by Microsoft. The code was an entry point to try and move laterally.”
The Sunburst code “tried a few things, and if it could talk it identified the domain, contacted the control and command centre, said ‘I am from company XYZ’,” Brown said. “For most companies, it didn’t do anything more unless it was someone the attackers wanted to target.”
The attack was sophisticated, so much so that “Microsoft said they believe a thousand people were behind the attack group, because of the level of sophistication,” Brown advised.
“The code was unique in the way it tried to evade things, and where it would run and not run. It inserted code but didn’t weaponise or take action until time had passed. It was smart, patient, and prescriptive, and made very little noise in our back-end environment,” Brown said.
Specifically, the SUNBURST code was not injected into the Orion source code itself, but into the build environment, through malware called SUNSPOT that was specifically written to carry out this swap in the SolarWinds build environment. “We check code out of source code control, have a TeamCity environment to kick off the build, and here the attacker looked for Orion to be built and swapped a file. It was a transient virtual machine and that’s hard to detect,” he said.
SolarWinds is confident no other component or code has been compromised. “Absolutely not,” Brown said. “The attack was a specific component to do a specific function. We looked at source code, controls, control systems, and involved KPMG and others. We didn’t see any tainting of the source code environment. The point where the insertion occurred was specific. It was a very pointed, targeted attack, and the mission was to taint code in a specific product and get out. Our instrumenting with CrowdStrike looked at lateral movement outside those areas and supports that the mission was extremely targeted.
The question remains, who did it? "We're not in the business of that level of investigation,” Brown said. “A number of other parties have attributed it to Russia and that’s who the U.S. Government is pointing a finger at right now.”
This assertion of a nation-state attack is based on careful analysis of the models used, and how the attack ultimately was not doing harm but collecting information. It was not a ransomware, say, attack, but instead performed stealthy gathering of information on specific entities. “It was a stepping stone to other things, potentially,” Brown said.
“We know the build system was compromised, and Sunburst was inserted through a service account, accessed via an entry point in New Jersey,” Brown said.
SolarWind’s investigation into how the service account - “patient zero” - became compromised revealed three different models were used. A vulnerable component was running; a highly targeted spear-phishing attempt resulted in an account compromise; and a broader-based spray paint campaign was aimed at a certain individual, and ultimately compromise occurred via email.
Microsoft’s testimony reported they knew of 60 customers with the same compromise in their email systems, where the same attacker had used email for reconnaissance.
The well-reported “SolarWinds123” password was unrelated to the attack. “It was a system with a separate username and password and nothing to do with Active Directory - that’s why it didn’t meet our password policies,” Brown said. “It was a separate FTP site and nothing to do with Sunburst. It did happen, somebody posted a username and password in their own GitHub account that was public, a researcher reported it, we saw, fixed it, brought it under control and made sure it won’t happen again.”
“It was completely separate from the build system or the service account. It happened, and we fixed it within the same hour it was reported. It was a system outside of policy and had nothing to do with corporate identity systems,” he explained.
The threat actor was stealthy and SolarWinds has responded by placing far greater safeguards in place - things you would not ordinarily think any software company ought to need to do, but which has become the reality of life post-SUNBURST. And here, all organisations large and small need to consider their own security posture.
For example, SolarWinds now conducts a two-way build. The code is pulled from source code control, built, hashed, then decompiled and compared back to the original, to ensure what ships is what was coded and nothing more or less.
“We’re trying to help people understand this and publishing new build methods and how to do secure builds, to help the community put appropriate safeguards against this threat actor,” Brown said.
“Yes, it did happen to us,” he said. However, SolarWinds has come through to the other side and is committed to helping the global software development community learn from its experience. “We want to be the best software company we can be and bring everyone else along with us to improve and be better,” Brown said.
Importantly, it's time the technology industry took "zero trust" seriously in its own environment. “Gone are the days where you run a standard system from developers,” Brown said. “We have to remove flexibility from developers. You need to really implement a zero-trust model and say we’re not going to trust anyone.”
Yet, how do you implement and test in an environment where you don’t trust anyone? This writer knows from his own experience of companies where software developers have root access, where systems administrators re-use passwords, where key systems have default administrator credentials, where administrators don’t have separate logins. All these things are bad, and all these things are known to be bad. Any technical team will admit, “yes, we need to do it better” but flexibility and security are always in tension. Make a password policy too lax and user accounts get compromised; make it too difficult and people write their passwords on paper and stick them to their monitor. Similarly, give developers admin access to systems and they can deploy software easily but if they are compromised so too are those systems. On the other hand, don’t give developers admin access and burn expensive time waiting for infrastructure assistance.
“It’s hard, and it’s not natural,” Brown says. “But it’s where we’re going.”
“For me, I insert bad code and then test we detect the bad code during the whole build chain. Sunburst happened at the end of the chain; the next attack could be anywhere,” he said.
"You have to do more code reviews, secure your keys, do things like we're doing with our double-build model, make sure it needs collusion to insert bad code and no one party can do it,” Brown said.
SolarWinds’ experience is a wake-up call to software developers everywhere to triple-check their chain and to put controls around every component.
Ultimately, Brown says, “We will get to a point where we are much more resilient than most. Due to going through this, it allowed us to perform an inspection for three months. My environment is higher than most now, without question. A bar has been set and we need to increase that bar.”
“It’s a wake-up call for the industry,” he said. “Courses and books will be written because of this. It’s an inflection point for the software industry because it occurred, and it happened. It didn’t theoretically happen.”
“We all learned and will be asked a lot of hard questions by customers. We’re embracing these questions and end up being one of the safer bets because of all our inspection, changes and openness as a result,” he said.
The concept of a supply chain attack is not new, but "in practice, that it actually happened at this scale, is what shifts,” Brown said. “It can happen, and everyone has to work differently. There must be safeguards, more openness, there will be regulation, and more thought and inspection. It will be an inflection point in a number of things, and these will help us all be resilient.”
Where to from here
"If a thousand-person nation-state is after you, they will find a way," Brown said. “The motivation was extremely high, the amount of practice and stealth they utilised was extremely high, and the research companies equate this threat actor as one of the most sophisticated they have seen.”
"We've undertaken a full-on shift of not trusting the environment, of zero trust across the environment. It will be much, much more difficult for any attack to get into our environment. It won’t depend on a single tool or single set of people,” he said.
“Customers love our products and our responsibility now is to give customers and their leadership teams the confidence to go forwards,” Brown said. “It comes from openness and transparency, explaining the actions we’ve taken. All we ask is they measure our competitors in the same way they’re measuring us and ask for the same things and set the bar as high for everyone in the industry.”
"Our approach is to own this and say this happened to us and take responsibility for it,” Brown said. “And to show the world we’re doing the right things to come out of it. We’re sharing the lessons we’ve learned to be an open and exemplary software provider. We produce strong code, not just simple and powerful and affordable, but secure.”
"We will come out of this stronger because of the things that happened, and we will help the world be better at the same time,” Brown said.
Here is a SolarWinds video on implementing zero trust in your organisation: