Ransomware: Lessons Learned From Losing Millions
In 2021, I want to do something a little different.
I’m going to share my hope for the future by telling you some harsh truths created by the global business of ransomware. And it is a business. But this article is about hope for protecting privacy, trust, and, ultimately, our faith in the world around us. But as many of you know, the path to success is not without its... multimillion-dollar failures?
And besides, the best way to overcome something is to understand it. And so today, I’ve set out to have a frank discussion about the business of ransomware so that together we might understand it. Not so you will be scared of the threat, but so you will leave these pages with a better understanding of the world around you. And through that lens, you will see the path forward.
Already this year, not a day goes past where I don’t read a headline about ransomware, how to beat it, what widget will make you safe, and how the next webinar will give you the answers. I’ve worked some of the largest ransomware attacks in the region as an emergency responder and incident commander. I’ve shared some dark days with you all. I’ve made some wonderful new friends and colleagues along the way. We’ve spent millions of dollars safeguarding ourselves, paying ransoms, and responding to events.
I write to you today having spent thousands of hours in combat with some of the world's most inventive threat actors.
I want you to learn the hard lessons I’ve learned so I’m going to share with you 10 “truths” about what a ransomware attack is really like when it goes sideways (and it usually does) and how to prepare for that when it happens. This isn’t the “we can save you” and “this new product will save you no matter what” line that sometimes dominates the discussion. Again, this isn’t to scare you or market to you. It’s just the side of the story that isn’t being told, the view from the eye of the hurricane so to speak.
The reality is that there is good news and bad news.
The bad news is simple: you are vulnerable and there is no silver bullet.
There is no software that will protect you all the time. There is no gear or product that will safeguard your data every time. There is no moat deep enough, nor cavern wide enough. Ransomware is just the next in a long line of emerging threats, so if it's not this, it will be something else. Understanding and planning--not technology--is your greatest ally. Put it this way, if there was a product out there that could solve the ransomware threat, most people would just buy it, and ransomware wouldn’t be one of the most profitable cybercrime enterprises on the planet. What is more, there is also no CISO that can completely safeguard you. I make my living as one and I can tell you, with more than 40,000 hours into my career, we still lose battles. Our successes go uncelebrated through day to day operations, and our failures are sometimes epic in size and make the front page. That’s the gig, love it or hate it.
What I mean is, the good guys have to be right every time, an attacker only has to be right once. There are days when it’s like holding the tide back with a broom. On an individual scale, we are making a lot of good progress. On a global scale, we are not yet successfully combating the threat. This is in no small part due to the fact that the same principles, regulations, and tools that protect our privacy also protect the privacy of those that mean us harm. The battle ahead is complex and we must enter it prepared.
The good news is also simple: vigilance and focus make all the difference.
It’s true that we aren’t in the business of risk avoidance, we are in the business of risk reduction. We cannot remove the terrible effects of violating the public trust or losing your data, productivity, clients, brands, and dreams. But together we can make that less likely to happen and, in most cases, dramatically less likely. It’s not a perfect strategy. It’s half art and half science.
In order to illustrate the principles I want to share with you, I’m going to walk through 10 lessons of ransomware attacks that you might not be familiar with, how they play out, and what we can do to make it better.
Lesson 1: The ransomware attack doesn’t always start when the first files are encrypted.
In many cases, a ransomware attack covers a data exfiltration. The attack starts days, weeks, even months before the first file is encrypted. The data is long gone by the time your servers go down. The idea here is actually pretty crafty. If you rob a store, then burn the store down, nobody really notices you robbed it. Or if they do, it takes so long to figure it out, the thief is long gone.
In this example, and yes, this is becoming more typical, data pooling and eventual exfiltration is the goal. The ransomware was a cover. When you go to pay the ransom you’ll either be ignored, or they will offer you encryption keys for a fee and promise not to publicly expose your data for a larger fee. You may need one or both of these. In many cases, especially if the attack went on long enough, a month or more of backups may be infected and require cleaning or be completely lost. This effort is significant and requires more expertise than you may have internally. The delay in restoring data may be measured in weeks. If you don’t go through things at a granular level you may miss a dormant copy of an infected payload, a foothold as we the marketers call them. The whole mess could start over again days or weeks later. This has happened to some of the best-managed services providers and IT organizations I know. If anything is more frustrating than going through a ransomware attack, it's going through it again.
Lesson 2: The threat actors have gotten wise that you have good backups, so they are changing tactics.
It’s true, most of the threat actors have realized you will pay a lot more money to recover data if you can’t recover it yourself for a long period of time. They are targeting backup files and devices. If they destroy your backups, you are more likely to pay. The holy grail is still data exfiltration. Most victims will pay a lot more to keep data from being released publically than they will to recover the data in the first place. The new goal is to take the data, see if there is anything private that will cost you a fortune to go through public notification and see if you’ll avoid it for a few hundred grand.
Lesson 3: If you are on-premise and get exfiltrated, your hardware will probably not be recovery-ready for at least a few days--and it could be a lot longer.
You will be down during an attack and if you’re not prepared, there will be nothing you can do about it. I work with internal IT teams all the time that think the recovery will start as soon as the incident response team arrives. They have promised 8 hour or 24 hour Recovery Time Objectives. If data exfiltration is involved, you’re not going to be back up that quickly.
The reason for this is that the environment needs to be forensically imaged and a team of specialists needs to be approved for this process. The insurance company you notified (hopefully) will want to mitigate their own risk by assigning an attorney, and if they have reinsurance, the reinsurance provider may also want to assign one. Realize that everyone in the situation, yourself included, is trying to minimize damages. Your insurance company is a vested partner in your recovery, but their risk process may conflict with your desired timeline. It’s a fact of the business. Experienced incident responders will help you successfully navigate this conflict to everyone's mutual benefit.
Let me lay out an example of what I mean. All of the team members assigned will need to meet and decide how to proceed after the initial event is contained. In general, the incident response team (if they are worth their salt) will take control of the environment, mitigate the immediate threats and begin preparing it for forensic imaging. Once images are taken for analysis, they must be analyzed at a lab, which may not be local. I recommend working with incident response teams that can perform all of the forensic analysis locally. It just saves time, but, in all honesty, it’s not always possible. If you have to ship drives to a forensics shop, that shop will have to be approved by the legal team. You may have to wait for them to confirm the package is intact and is all of the data they need to make an analysis before the hardware environment can be released for recovery. This may take days, not to mention shipping challenges in today’s world. You’re still offline and recovery hasn’t even really begun unless you have a separate network and some extra hardware laying around. If you’re lucky enough to have a disaster recovery (DR) site or a cloud environment, this will help you.
Note: Why are these steps critical? If you lose data of a type that requires public notification (like banking data, credit cards, social security numbers, drivers licenses, medical records), you will need to go through a notification process. In order to notify, you need to know who was affected. It’s going to be a long haul and there is a specialized process for doing this.
The lesson here? Always, always have a fast way to get a second set of gear. A DR site, a cloud provider, a few servers in boxes...whatever it takes. You will need to bring up a redundant environment while your primary is frozen. It’s no small feat.
Lesson 4: During recovery, self reinfection is one of the most common mistakes.
For anyone who has ever worked with me during an incident, you know I’m absolutely draconian about the recovery process. I see you pick up a network cable and plug it in somewhere without a detailed explanation of what you are doing and why you are certain it’s safe, we are going to have a little chat. There is a good reason for this. Discipline is your friend. When dealing with infected environments, maintaining quarantine is critical to the recovery. You cross the wires and you lose control of the infection. Your recovery may have to start over. It happens way more often than you think. Way. More. Often.
Lesson 5: Paying the ransom is almost always an option (except when the government says you aren’t allowed to).
The US Treasury Department's Office of Foreign Assets Control (OFAC) advises us when we’re not allowed to pay a foreign threat actor. It’s never a good thing when you’re paying the criminal extorting you and our good friends at the United States Secret Services (USSS) are a great source of advice on this. In general, you are allowed to pay threat actors and it may be reasonable to do so. If you’re looking at hundreds of thousands in breach costs, a $50K ransom may seem reasonable. That said, paying a ransom looks a little different than you might expect.
In all of my time performing cyber counterintelligence operations and threat actor negotiations, a few trends have emerged. They are:
The cost of the ransom can be negotiated down by at least half. Most threat actor groups will play hardball coming this far down in price, but we can get them there if you don’t play games, irritate them, or take too long to start and conclude the negotiations. Time is money, literally.
Most threat actor groups will provide proof of life files (if asked) to show the assets they exfiltrated from you. You can’t just assume a file was just locked and not exfiltrated. This is an important part of the negotiation.
Only contact a threat actor from an anonymous email address. Revealing who you are is a really great way to end up on the “to be attacked in the future” list.
Some threat actor groups use shame sites to reveal data. If they are threatening to reveal the data, ask for the shame site and check out who is on it. This will give you some important insights into how extensive the attacks generally are.
If you pay to not have your data revealed publically, your chances of being re-extorted are about 50/50. In our experience, certain groups are pretty good about not re-extorting victims, while others are hit or miss. This is a business after all. If all victims get screwed over, nobody would ever pay. Re-extortion is not always a primary concern, though. The initial payment will give you time to get in front of public relations issues or shareholders.
Lesson 6: If you suffer a reportable data breach, the cost may skyrocket to the point that the ransom starts to look tempting.
It’s not atypical to have two legal firms, a PR firm, an incident response/cyber firm, a data mining or digital forensics firm, a threat actor negotiator or financial accountability firm, and your local team and resources all wrapped up in the recovery. That means update calls become $2,000/hr endeavors. I’m not kidding. Do the math. Cyber insurance is very important. I can’t stress this enough. Very important. A decent size ransomware attack with public notification can easily reach $250,000-$500,000 in fees alone just supporting the basic logistics.
Lesson 7: Law enforcement is your friend, so know how and when to use their services.
I spend a lot of time maintaining contact with our local law enforcement agencies. While many of them are totally slammed with work, they are generally great people, fun to work with, and want to genuinely help. There are exceptions, but, overall, most of the people that are driven to law enforcement have a mind to help. That said, their cyber capabilities are very limited when compared to private firms--but don’t count them out. A good incident response team will bring in law enforcement in three cases.
1) If there is a chance that litigation may occur and forensic images are best taken by a federal agency. I do work under the Federal Code of Evidence Rule 702 as an Expert Witness and I can tell you from a point of observation, the courts like when the Department of Justice does its own forensic collection vs. private firms. It’s a bias--and we can use that to our advantage.
2) The second reason law enforcement may be brought into your event is if PR questions will be asked. It’s always nice to say you’ve involved all of the authorities required and are doing everything possible to remediate the event.
3) The third is probably the most important. If you are dealing with a well-known threat actor, there is a chance there is a larger federal case being built against them. Involving law enforcement may give them clues and evidence, fact patterns, or behaviors that aid the development of the case. You won’t know if you’ve really helped in any way, but it’s the right thing to do. I also strongly advise authorizing your incident responder to involve appropriate law enforcement at times when it won’t hinder the recovery effort.
Lesson 8: Your IT team or MSP may not be able to help with some aspects of the recovery.
Many legal opinions advise against allowing the team responsible for the infrastructure to perform incident response and forensic work either due to conflict of interests or capabilities. It’s pretty common for a managed services provider (MSP) to have a security team, but if they aren’t approved by insurance or do not have specific subject matter expertise in threat actor negotiations, forensics, counterintelligence, incident response, public relations, or data mining, then they may not be permitted to perform that task as part of the claim. Even competent firms often must maintain segregation of duties in order to eliminate conflicts. This means the people with the most knowledge of your environment may be taking a back seat to a new team at certain parts of the recovery process. This will slow you down, but there is actually a pretty good reason for it, so don’t fight it if it happens.
Lesson 9: Fatigue is going to slow you down a lot more than you think.
IT teams are forged in fire. If you have ever been part of a long-standing IT department, you know exactly what I’m talking about. It’s brothers and sisters in arms mentality. Ask anyone who has been in a lengthy technology or datacenter firefight. Just about any team can crank for 24 hours--but the returns diminish quickly. Food, coffee/energy drinks, and sleep all become an issue fairly quickly after that. By 36 hours, your team is cognitively impaired and no longer fit to work in production. During a recovery effort, the last thing you want to do is lose recovery time due to fatigue. However, mistakes are even more costly. The goal here is to plan and practice rotation of schedules and support services. If your DR plan does not include supportive services and rotation schedules, up your game and get that done. Fatigue leads to mistakes, plain and simple. A good incident responder will help monitor the team for fatigue and help you manage the throughput schedule.
Lesson 10: Ransomware is just the next in a long list of attack types. This is only the beginning and it will keep changing.
The key point here isn’t to focus too much on any one attack type. Preparation for cyber attacks is not about understanding one vector because it will continue to change. It’s about understanding and operationalizing three guiding principles:
Discipline
Discipline means understanding the risk, seeing it clearly. It means having processes and safeguards that are not paper tigers, but tried and true plays you’ve practiced. Yes, practiced them despite your overburdened schedules and constant firefighting. It means running those plays over and over to train your organization--both IT and non-IT--how to think and act.
Response
Response is the plan you put into place to respond to issues. Not all of them will be of the same type or size. The response is about whom you partner with, what tech you use. and how you train. Whom do you trust when things go sideways? Is your team ready to follow them into the fight? Have you done the work?
Overwatch
Overwatch is your ability to see the next threat coming before it arrives. This is the most difficult of the three. It requires experience, vision, and creativity. If your organization lacks any of these skills, you will see these threats only when they are on your doorstep.
With that, I’ll leave you with a few final thoughts.
Get cyber insurance, good cyber insurance. Meet with the legal team they will use. There are specialized firms that work on cyber breaches. They will play a key role in providing privilege and guiding the legal process. I could write a whole additional blog post (maybe I will) on the pros and cons of privilege cloaks but that’s for another day. Suffice to say that your insurance company, your incident response team, and the legal teams assigned are your best tools for a swift-ish recovery. Knowing them, the procedures they use, and the experience they have. Most of all, place a call to your incident response team, it’s probably a third party that works with your insurance company. Have a virtual coffee with them. Trust me, the next time you talk to them, it will be nice to see a familiar face. Put it this way, do you want to meet your lawyer on the first day of court? Planning and preparation make all the difference. Remember, leadership in the face of risks like ransomware is extremely hard, take care of each other.
If you have questions or comments or your own stories to share, drop me a line here or at jason@theatomgroup.com and I’ll be sure to get back to you. As we learn more lessons, we will be sure to update this post, accordingly.
Jason Sgro is a Sr. Partner for Security & Privacy at The ATOM Group in Portsmouth, NH. ATOM's Security & Privacy Practice specializes in Emergency Response, Proactive Auditing, Data Forensics, Human Privacy, Security Assurance, Asset Recovery and Software Modernization, for startups to the Fortune 500.
