Right in Sync is a blog series featuring interviews with customers and influencers from the OffsiteDataSync (ODS) community about their experiences and advice on timely topics. In this edition, we speak with Eric Woodard (EW), CEO and founder of Protek, a managed services provider (MSP) with a fascinating story about a ransomware attack and how transparency forged an even stronger bond with customers.
ODS: Tell us a bit about Protek.
EW: We’re Utah’s premier outsourced MSP, primarily working with companies throughout Greater Salt Lake City and some of their affiliates across the U.S. Basically, we’re their outsourced IT department, offering full IT support, strategy, systems and security. We are just about to celebrate our tenth year in business.
ODS: What does your typical client look like?
EW: Most are small- to mid-sized companies (SMBs) focused on construction, engineering and architecture, as well as mortgage and title companies. Usually they have anywhere from 10 to 200 computers and we handle their IT, soup-to-nuts, from phones and PCs to security and cloud apps.
ODS: What sets Protek apart?
EW: Customer satisfaction and committed personnel. We have very high client and employee retention rates. We matured a lot faster than our competitors, so our processes are proven and our people have a lot of experience. Satisfaction rates hover between 95 – 99% at all times; we publish the stats right on our website.
Every quarter we meet with clients. We ask if they’re happy, what projects they have planned, and what challenges they anticipate. We discuss the “b-word” that IT people fear most — budget — and look down the road to plan strategy and suggest technologies that’ll help them be more cost-efficient. As trite as it sounds, we do what we say we’ll do, and that means a lot in the world of IT.
ODS: How has client work held up during the pandemic?
EW: We previously would have close to 50 service tickets a day, but when quarantine restrictions started coming down, we were seeing 300-400 a day, a huge surge due the need to support a suddenly all-remote workforce. Within just a few days, all our clients’ people were up and running from home. In a week or two, the daily ticket count was almost back to normal, with the occasional call from someone asking how to set up their Zoom background with a picture of their cat.
We have a very strong team partly because we went through a serious business crisis last year.
ODS: What happened?
EW: In February of 2019, we were the subject of a ransomware attack that encrypted our customer’s data and distributed malware to more than 1,700 machines. For six months, I wanted to crawl in a hole; we pride ourselves on the strength of our security.
We were the first and one of the largest of hundreds of MSPs hit. While many had security shortfalls, our systems were very advanced. We used multi-factor authentication, next gen firewalls and anti-virus technology. All software was updated at least once a week. Security information and event management (SIEM) patrolled for odd behavior and threat hunting software searched for malware.
ODS: How did they get in?
EW: One of our vendors had an undetected hole in its software that a sophisticated and highly organized Russian cybercriminal operation was able to exploit. They were in and out of our systems within 30 minutes, leaving millions of encrypted files and a ransom note in their wake. Months later the hackers announced they had made $2 billion and were retiring. I kid you not.
ODS: What did your backup provider do?
EW: At first I wasn’t too worried. We contracted with a backup-as-a-service (BaaS) and disaster-recovery-as-a-service (DRaaS) provider to protect all the data. Their service level agreement (SLA) covered situations exactly like this one. But, when I called to ask them to spin up between 200 and 300 servers from backups, they said they could only do a dozen and it would take a week. It was clear they weren’t going to be able to help us, so we decided to wire the $92k ransom to the criminals.
Then, we waited five days for the encryption key and, when we received it, the key didn’t work.
ODS: You must have been horrified!
EW: Well, I emailed the cybercriminals’ equivalent of customer service, they resolved the issue quickly. They even apologized for the inconvenience.
EW: Oh, yes. Even so, only 1,000 files could be decrypted per hour. Protek had to write scripts to run 50 decryptors at a time per computer and work round the clock for two months to retrieve all the data.
ODS: Business took a big hit?
EW: Remarkably, we retained 95% of our customers.
ODS: That’s incredible. How did you do that?
EW: First, you couldn’t ask for better clients; our average customer has been with us at least five years. But really, it came down to transparency and communication. We met with each client in person, explained exactly what happened, the consequences of the attack and measures we were taking to retrieve their data. We followed up with nightly email updates and held frequent conference calls.
To be honest, we’ve got an even stronger relationship with clients now. They appreciated our transparency and know how committed we are to ensuring that this never happens again.
ODS: And the provider?
EW: A big part of our plan required finding a new BaaS and DRaaS provider. We evaluated software solutions and decided Veeam would work best due to its ease-of-use, reliability and flexibility. We reached out to our Ingram Micro rep for a provider, who offered options. But, soon after, we went to a Ingram Micro conference and met with Veeam, who promptly walked us over to you folks.
We had ODS demonstrate its ability to spin up 200 to 300 servers from backups within a week. We required a written guarantee and strong SLA. We insisted on reliable backups stored in a secure, dependable data center. All of these requirements were met, so we switched to Veeam with a backup repository at ODS, and, once our data was in the system, we did some additional testing of our own comparing our old solution to the new one.
ODS: What were the results?
EW: The combination of ODS and Veeam was 75% faster to restore, and it took just five clicks to complete versus 30 minutes of a highly skilled tech’s time. ODS is now backing up all our on-prem servers.
Having the peace of mind that backups are occurring behind the scenes, that we can recover quickly in the event of disaster and that ODS is standing behind us is a huge comfort.
And, of course, we were able to retain nearly all of our customers. Our security posture and satisfaction rates are stronger than ever, our team is the best in the business, and we continue to grow. So despite the current public health and economic crisis, we’ve got a pretty positive outlook about the future. We faced our worst nightmare and came out the other side in even better shape than before we went in.
Read the case study here.