It all began when the cyberattack on the company FireEye went public, in which already published Red Team tools used for penetration testing were stolen. It remains unclear if FireEye’s own software solution has been manipulated by the attackers. What also remains unclear is whether more FireEye Red Team tools used for pen testing were stolen or not. Research on the FireEye incident has revealed that the software solution, SolarWinds Orion, was manipulated. In what is called a “supply chain attack”, the source code for SolarWind’s Orion solution was manipulated and a backdoor, named Sunburst, was smuggled into the program’s code.

We’re talking about a worst-case scenario here, since over 18,000 companies and organisations worldwide use Orion in their networks and have hence become vulnerable to potential backdoor attacks where the attackers could gain full access to their systems. This is why SolarWinds customers leveraging Orion environments have most likely been penetrated by these attacks.

Based on our current information, we are assuming that this backdoor has been in place since Q2 / Q3 of 2020. Microsoft has released a statement saying that they found these malicious SolarWinds binaries in their systems and have isolated and removed them.

Update from 13/01/2021: Stolen data such as source codes, bugtracker dumps, customer portal data, binaries, internal documentation and private Red Team tools from FireEye, Microsoft, Cisco and SolarWinds are being sold on the dark web. The authenticity of these data have not been confirmed.

Many companies are now facing the challenge of taking the right steps to deal with this vulnerability, while, in the meantime, data forensics are still struggling to find out what exactly happened.

The following guide is intended as a crutch, and may even be helpful for companies that have already started the recovery process, as it might include a few points you haven’t thought of. Do not blindly follow all of the measures listed below. It’s up to each organisation to conduct their own test, to find out which effects the measures have and to make sure that these are compliant and will not cause any collateral damage.

Bechtle has a team of experts from incident response, IT forensics and security available for your support.

Research / investigation.

  • Make a forensic image of the system or systems (SolarWinds Orion), including RAM and hashing of the hard drive
  • Protect any network logs such as firewall and NetFlows
  • Export all relevant logs on the server

Concerning accounts / identities used by SolarWinds Orions for surveillance and administration and local accounts on SolarWinds systems, you should ask yourself the following questions:

  • Have accounts (successfully or unsuccessfully) been authenticated on systems or applications that they shouldn’t have?
  • Have accounts (successfully or unsuccessfully) received authentication attempts from external locations, especially from locations you never logged in from before?
  • Have accounts triggered any “impossible travel” alarms?
  • Have any suspicious OAUTH applications e.g. mail read / write permissions been granted inside Office 365?

Networks.

  • Have systems connected to any known command and control domains or IP addresses?
  • Have systems connected to any DNS sinkhole 20.140.0.1 IP addresses?
  • Have systems connected to any external domains or IP addresses?
  • Have systems connected to any internal systems they shouldn’t be connected to?
  • Have systems connected to any internal systems at unusual times?
  • Were new SAML/ADFS Federation Trusts added to your Azure AD tenants?

Endpoint.

  • Is the backdoor dynamic link library (DLL) still on the SolarWinds Orion system?
  • Is the backdoor.dll on any other systems?
  • Are there any indicators of suspicious activity on the SolarWinds Orion server?
  • Are there any hints of suspicious activity on systems or applications that have their login information saved on the SolarWinds Orion systems?
  • Are there any suspicious ASEPs on the systems or remote desktop tools?
  • Were accounts on the SolarWinds Orion server, systems or applications that are accessed by administrators added or reactivated?
  • Were all configuration data from network devices that SolarWinds Orion had access to validated?
  • Did the security software in use on the SolarWinds Orion systems detect any suspicious behaviour or malware?

Post-treatment.

  • Were any activities detected on a system after its exploitation?

- CobaltStrike

- Empire

- Including "c:\windows\syswow64\netsetupsvc.dll"

  • Has your organisation’s domain been leaked online in RDP SSL certificates?

Containment.

  • Isolate the SolarWinds Orion server containing the backdoor.dll. Options:

- Segment SolarWinds Orion server

- Null routing IP address

- Deactivate the SolarWinds Orion server virtual network card

  • Deactivate all accounts that are used by SolarWinds Orion to access other devices or the communication in your network. These accounts include:

- Orion platform service account

- SQL database service account

- Service accounts for monitoring applications, systems, WMI

- Service account for mail alerting

- SNMP

  • Block network communication to usual C2 domains and IP addresses

- DNS sinkhole

- IP addresses

- Domain and IP on the proxy

  • Alarms for any system that accesses known indicators of compromise (IoCs) by Sunburst or uses a deactivated user ID (endpoint or network)

Eradication/cleaning.

  • Change all passwords for accounts used by SolarWind Orion incl. service accounts, admin accounts, shared accounts etc.
  • Change all passwords or community strings that are saved on the SolarWinds Orion platform
  • Remove the SolarWinds Orion servers from the network

Recovery.

  • Backup SolarWinds configurations and database
  • Reinstall the OS using a clean, trustworthy image
  • Install SolarWinds version 2020.2.1 HF2
  • Ensure endpoint security and visibility on the new server
  • Recover configurations and database from the backup
  • Check cryptographic hash function of the file SolarWinds.Orion.Core.BusinessLayer.dll and make sure that the backdoor.dll is removed

IoC.

https://github.com/fireeye/sunburst_countermeasures

https://msrc-blog.microsoft.com/2020/12/13/customer-guidance-on-recent-nation-state-cyber-attacks/

Sources.

https://mp.weixin.qq.com/s/v-ekPFtVNZG1W7vWjcuVug

https://securelist.com/sunburst-backdoor-kazuar/99981/

https://www.bloomberg.com/news/articles/2020-12-15/fireeye-stumbled-across-solarwinds-breach-while-probing-own-hack

https://www.reuters.com/article/us-usa-cyber-breach/suspected-russian-hacking-spree-used-another-major-tech-supplier-sources-idUSKBN28R2ZJ?il=0

https://blogs.microsoft.com/on-the-issues/2020/12/17/cyberattacks-cybersecurity-solarwinds-fireeye/

https://www.nytimes.com/2020/12/08/technology/fireeye-hacked-russians.html

https://msrc-blog.microsoft.com/2020/12/21/december-21st-2020-solorigate-resource-center/

Statements.

Cisco:

https://tools.cisco.com/security/center/resources/solarwinds_orion_event_response

Immediate action.

Microsoft Defender:

https://www.microsoft.com/security/blog/2020/12/15/ensuring-customers-are-protected-from-solorigate/

 

The recommendations that we have introduced here should provide you with an overview of effective measures. If you are in doubt, please speak to us—we can help you with your individual blueprint and realisation of a plan of action.

Send any requests to it-security@bechtle.com.