On March 18th, an employee in Ireland’s healthcare system clicked on and downloaded a malicious Microsoft Excel spreadsheet in an email.
According to an official report by PwC issued this month, that mistake led to a series of compromises that spread ransomware across the healthcare system two months later.
Eighty per cent of the data in the healthcare system’s IT environment — including patient electronic health records — was encrypted. Many hospitals were forced to cancel outpatient
appointments completely while others ran with significant delays and reverted to using pen and paper to continue recording patient care. Over half of the hospitals in the country canceled at least some of their outpatient appointments.
Fortunately, the Conti ransomware gang, which was behind the attack, handed over decryption keys after a public uproar at its attacking a hospital network. But it cost the government about $600 million to cleanse all systems.
Crucially
— an antivirus solution was set to monitor, not block malicious commands sent through attack tools;
— reliance for defence was placed on a single antivirus product that was not monitored or effectively maintained with updates across the environment. For example, the workstation on which the attacker gained their initial foothold had not had its antivirus signatures updated for over a year;
— the Health Service Executive (HSE), which provides services and runs the IT network, only periodically backed up data to offline tape;
–the flat IT network made it easier for staff to access applications, but also easier for the attacker to roam;
–the attacker was able to compromise and abuse a “significant number” of highly privileged system administrator accounts and move laterally to both statutory and voluntary hospitals on the network. Many of the tools and techniques used (which included the use of basic and non-obfuscated malicious PowerShell commands), were well-known to be used by ransomware groups, says the report. “As such, they would have almost certainly been identified by modern security monitoring tooling and a security monitoring capability;”
–The HSE’s antivirus software identified a tool commonly used by ransomware groups (Cobalt Strike) on six servers on May 7 (a week before the ransomware was detonated) but these alerts were not appropriately actioned. The HSE did not identify these alerts until after their cybersecurity solutions provider flagged them on May 12 and 13. At that point, the retained third party ‘critical incident response service’ was not invoked.
“The response to these detections was not sufficient as the HSE did not invoke a cybersecurity incident, escalate the cybersecurity incident; identify the severity of the threat; or thoroughly investigate and contain the threat,” said the report. “This was a result of insufficient cybersecurity expertise to understand the significance of these detections and an absence of cyber response governance and processes to guide the response to cybersecurity incidents.”
–the IT environment did not have many of the cybersecurity controls that are most effective at detecting and preventing human-operated ransomware attacks, the report found;
–HSE had a CIO with a budget of over CDN$100 million, but did not have a CISO or a security operations centre;
–while the office of the CIO had a staff of 350, only 15 full-time equivalent staff had cybersecurity roles — “and they did not possess the expertise and experience to perform the tasks expected of them,” the report says;
–the HSE’s board knew from a November, 2020 report there were many areas of known cybersecurity weaknesses, including known issues with excessive privileges on accounts. Implementation of many of the cybersecurity controls mentioned in that report “would have been highly likely to have prevented or detected techniques used by the attacker and therefore significantly increased the attacker’s difficulty in compromising the HSE and achieving their objectives,” said the report.
“The low level of cybersecurity maturity, combined with the frailty of the IT estate, enabled the attacker in this Incident to achieve their objectives with relative ease,” the report goes on. “The attacker was able to use well-known and simple attack techniques to move around the NHN, extract data, and deploy ransomware software over large parts of the estate, without detection.”
HSE provides all of Ireland’s public health services, serving 5 million people through 4,000 facilities, including 54 acute care hospitals. The HSE’s national health network includes over 70,000 end-user devices, 4,500 servers and 1,000 applications.
At the time of the incident, the CIO had an operating budget of about CDN$118 million and an IT capital budget of CDN$173 million. Yet 7,000 PCs were running unsupported versions of Windows 7.
The public hospitals operate under the authority of the HSE. Voluntary hospitals, which receive state funds through the HSE, have their own IT teams and infrastructure but also use the national IT infrastructure.
A brief timeline of the incident:
–on March 18 an HSE staff person opened the malicious Microsoft Excel file, infecting their PC. (Investigation later found the user of this PC was targeted with phishing emails with the same email subject four times between Dec. 14, 2020 and February 9, but the workstation was not successfully infected with malware.); Not long after, a second attacker was involved, who appears to have taken over.
–by March 23 this attacker had created a persistence mechanism on that machine that ensured access to the IT network even if the PC was rebooted or powered off
–on March 31 antivirus software detected the execution of Cobalt Strike and Mimikatz communication and password-stealing tools on the PC. However, it was set to monitor-only mode and didn’t block malicious commands;
–between April 1 and May 6 the network’s incident response provider detected no significant attacker activity. The post-incident investigation found that on May 8 there was evidence two hospitals had been hacked, another on May 9 and two more on May 10.
On that day the AV at one hospital detected Cobalt Strike, but failed to quarantine the malicious files. That hospital asked its cybersecurity solutions provider if it should be concerned. It was told since the threat had been remediated the risk was low;
–on May 11 a sixth hospital was hacked after numerous failed login attempts. Investigators suspect the attacker likely exploited an unpatched vulnerability in an unnamed product to gain access to the hospital’s domain. AV did, however, detect and delete malware on several systems;
–on May 12 the attacker was seen browsing folders, opening files, creating archives and more in systems of several hospitals. One hospital reset passwords for 4,500 accounts and made firewall configuration changes. HSE’s cybersecurity solutions provider emailed the security team to escalate alerts on two servers and requested a full on-demand scan be done. It was.
–on May 13 attacker activity was seen on IT systems within the HSE. Again the security team was notified by the cybersecurity solutions provider, which also noted there were unhandled threat events since May 7 on at least 16 systems. The security team requested the server team to restart servers.
Meanwhile the HSE investigated a complaint of suspicious activity coming from HSE to a hospital. It wrongly concluded activity came from the hospital;
–on May 14 the ransomware was executed. The national Department of Health, which also used the HSE network, detected and defused the attack the day before in most of its environment by installing endpoint detection and response software.
Compromised 180 systems
The HSE’s post-incident response investigation found the attacker compromised 180 systems and an unspecified number of highly privileged accounts across eight organizations and 19 domains. Over 2,800 servers and 3,500 workstations across 15 domains were encrypted.
Among the problems the PwC report found was that there was no dedicated committee
that provided direction and oversight of cybersecurity and the activities required
to reduce the HSE’s cyber risk exposure. A cybersecurity forum had previously been
established within the CIO’s office, but was disbanded before August 2019 without
replacement. That hindered the ability for granular cyber risks to be discussed and documented, and for mitigating controls to be identified and rapidly delivered.
There was a process through which risks were raised to the CIO’s office, but there was no centralized decision-making committee to provide direction and decide on a suitable course of action to mitigate these risks, considering the cybersecurity capabilities and controls required.
The report recommended
–establishing clear responsibilities for IT and cybersecurity across all parties that connect to the national health network, including a code of connection that sets minimum cybersecurity requirements for all parties;
–establishing an executive-level cybersecurity oversight committee to drive continuous assessment of cybersecurity risk and push forward a cybersecurity transformation program
–establishing an executive-level oversight committee for IT;
–establishing a board committee to oversee the transformation, and a chief technology and transformation officer (CTTO) to create a vision and architecture for a resilient network
–appointing a CISO who reports to the CTTO;
–ensuring that the HSE’s managed defence service or an equivalent is maintained to detect and respond to incidents on endpoints, to provide protection to the entirety of the national health network;
–establishing an initial cybersecurity incident monitoring and response capability to
drive immediate improvement to the ability to detect and respond to cybersecurity
events;
–upgrading the national medical imaging system to allow an upgrade to Windows 10.
Finally, the report also includes broad lessons for all organizations:
–Organizations need IT governance and cybersecurity leadership. Governance must ensure risks associated with technology are properly understood.
Organizations also need a single accountable leader responsible for delivering a cybersecurity strategy;
–given the heightened threat of ransomware, organizations should perform a cybersecurity assessment specific to the threat of this attack;
–organizations must have effective security monitoring tools, including deploying endpoint detection and response software;
–regular testing of cybersecurity capability through simulated attacks is vital. So are incident response, business continuity and disaster recovery plans. As part of that, have contractual retainers with third parties to support crisis response.
