The Equifax Hack Revisited

Hacking of personal information.  It’s certainly nothing new.  But the hacking of a credit reporting agency?  That’s more akin to breaking into Fort Knox.

Even years later, the story remains shocking for its business ineptitude, callousness, and technical ignorance.  And for those reasons, and avoiding similar outcomes, it’s worth revisiting.

The guilty parties were world class, but the actual compromise? No, neither unforeseeable, nor sophisticated subterfuge.  It was instead a compendium of errors – some human, some systemic.  But together, they brought the house down.

You can read my thoughts at the time of the breach in this Guardian UK article, as well as the response to the hack from then CEO, Rick Smith.

The Take

A $3.5 Billion business was the victim.  This was a company with both the revenue and the asset risk profile necessary to justify unparalleled security infrastructure.

An infrastructure that should have protected sensitive data – data widely viewed as the holy grail of financial information hacking.  Yet, the crux of the hack and the failure was so embarrassingly basic.  Equifax didn’t apply software patches.

163 million consumers + 3 months to trawl for all of it.

As a result, 147.9 million Americans had their sensitive data stolen.  And 15.2 million British citizens suffered the same fate.  This occurred not in a single swoop.  But over the course of three months of unfettered access to Equifax systems. Three months, that’s an eternity in terms of IT compromises.

Equifax lost the core of its business that day, an unspoken promise to secure consumer data.

And when we say “got taken” we are talking the core of its business services, collecting and retaining personal and financial information. Equifax lost the core of its business that day – an unspoken promise to secure consumer data on over HALF of the American populace.

Remember this is not ancillary, as we might see with a retailer losing such information.  Credit data is all Equifax does.  All day long.

Centralization of Credit Analysis

Why was this massive breach so…massive?  You can thank modernization and standardization for that.

The three major credit reporting agencies (CRAs) are the mother lode of sensitive personal financial data and identity.  Prior to the 1920’s, this information was highly fragmented with a variety of organizations sourcing and supplementing the data.

Until the 1970 Fair Credit Reporting Act, the information rendered via a credit report was also anything but transparent.  It could be deeply subjective, even outrageously biased.¹

So modern regulatory & technological developments have been a good thing.  Decisions regarding credit worthiness can now be made efficiently, with general uniformity of fact.  And these decisions can be made regardless of an applicant’s or merchant’s geographic location.

That means that when you want to buy a car or home, the analysis of your ability to borrow can be determined rather quickly.  If credit has been wisely used and built over time, this can be a very good thing for consumers.

It increases consumer transactional options – their ability to buy what they want, when they want, and under what conditions (borrowing terms & interest).  And if you are denied credit, you can now determine WHY and by extension, the validity of the decision.

New Problems Arise

These modern, streamlined processes to determining credit have yielded a downside too.  Because a great deal is possible for anyone who gains unauthorized access to them.

A hacker could dramatically increase their crimes of opportunity by knowing who the best credit targets are, with visibility to credit histories.

Even worse, most if not all of the sensitive personal data needed to apply for new lines of credit or create synthetic identities is often in credit reporting files.  Stealing that information is a very big deal.

When the news of this hack first broke in the press in summer 2017, there was understandable shock the breach had even been possible.  Media outlets pounced with big coverage and the public inundated Equifax call centers. 

Congress held hearings.  The blowback was palatable and universal.

About the only stakeholder that seemed in denial was Equifax itself. 

With executive sell-offs of stock made just prior to informing the FBI of the breach… an attempt by the company to secure lifetime indemnity from those impacted in exchange for one year of credit monitoring…  and the CEO’s plan to remain at the helm…the firm looked at best clueless and at worst, incompetent and calculating.

A Well Worn Blueprint

But even a cursory understanding of the Fortune 500 landscape, should have been more than enough for the Atlanta based firm to be ready for this.  There was simply no reason for the company and its C-Suite not to be prepared because these kind of attacks were not new.

This breach was 3 years after Target and 10 after TJX.

Even from a casual read of major news story headlines over the years, any management team dealing with sensitive consumer data should have been ready.

This breach occurred three years after the high profile failure at Target (Dec 2013).²  It occurred a full decade after TJX’s.  The largest ever breach of credit and debit cards in 2007 (45.7mm over 18 months).³

In both instances, the victim companies internal systems were co-opted by hackers to collect sensitive information. Said another way, there was a readily available blueprint as to how hackers acquire sensitive personal information – via third-party exploitation.  No competent executive could claim ignorance of this attack vector.

How It Happened

Information Technology.  Specifically, “one IT guy” to quote the former CEO.⁴  In essence, the Equifax defense is someone on the technology side of the house failed to notify whomever was responsible for patching.

Per the GAO report, this was apparently due to an out of date systems administrator “recipient list”, presumably an email list.  So a vulnerability had been identified by a vendor, a patch was available, but that patch wasn’t universally applied…for months.

While Equifax says the breach lasted 76 days, it apparently lasted longer when the timeline is examined.

The GAO report, based on data from Equifax, appears to focus on the period between data removal from the company (in May) and discovery (in July) – stating the breach lasted 76 days.

If we consider the full period of time that hackers were on Equifax systems, the period is much longer than that reported in the press (about 80% longer) – at approximately 139 days.

Post-Mortem Analysis

A post-mortem suggests the following timeline:

• March 8, 2017 – US-CERT (a division of the US government) releases a notice of vulnerability in Apache Struts – a program used to create java web applications.  A program Equifax uses.  Links are provided to update to a patched version.
• Competent IT personnel monitor such outlets, as well as those of software vendors their companies utilize.
• March 10, 2017 – State sponsored attackers affiliated with the People’s Liberation Army of China, scan the web for this disclosed vulnerability and determine Equifax web servers have an unpatched version of Apache Struts.
• Hackers begin running test commands, broadening their understanding of the servers and data.  No data is taken.
• March, April, May, June 2017 – Equifax repeatedly fails to apply the readily available patch to ALL of its systems.

Data Usurpation Begins

• May 13, 2017 – the hackers begin removal of sensitive personal information from Equifax. 
• An expired digital certificate on one of Equifax’s servers meant that the encrypted traffic was not being examined by Equifax personnel or systems.  This allowed the data to move without review.  Another failure by people and processes. 
• The certificate had expired a full year earlier: May 2016.
• May, June, July 2017 – using encrypted commands and 9,000 queries, the hackers locate 48 additional servers, identify unencrypted usernames & passwords, and continue their theft undetected by Equifax personnel or scanning software.
• The attackers remove the stolen data in small, encrypted batches over the course of three months. 
• July 29, 2017  Equifax personnel discover the hack during a routine scan.  Presumably ending the data thefts.  Total time of hack – approximately 139 days.

Here is a nice graphic courtesy of the GAO for how this hack developed.

How Do We Prevent It?

Identify Your Risks, Establish Controls For Those Risks, and Repeat.

System Reviews By Independent Folks

From a consultant’s vantage, system reviews exist to prevent this very outcome.  Corporate fail-safe measures involve multiple overlapping systems or responsibilities to catch what one person might miss. 

If you are responsible for patching System A, then someone else should have review oversight to insure that happens. We can think of the identification of what ‘could’ go wrong as risk analysis, and preventing it from happening as the procedures we establish as controls.

Serious Patching Procedures

In this case we are talking manual patching.  This is unfortunately still a reality in the world of IT.  But such an approach simply means at least weekly meetings of relevant personnel who go over required patches and confirm, in writing, their application of same or rollout to do so.  There is oversight in this process and seriousness in intent.  It’s not hard, just detail-oriented.

Some technology pros may scoff at that.  It is akin to a weekly grammar lesson among English PhD’s.

But in the Equifax case, the System Administrator responsible for the Dispute Resolution Server “didn’t get the memo” about the Apache Struts vulnerability.  Literally.

He or she therefore never applied the patch.  And a system scan for vulnerabilities executed by Equifax to self-identify this vulnerability failed to find the non-compliant servers.

Treating this stuff as unworthy of IT personnel’s expertise has its answer in the outcome above.  So, my response to the complaint that this is a poor use of time and resources?  Yes, it is.  But please block out your calendar for it anyhow.

Data Silos

Just what it sounds like.  Data should be firewalled on different servers.  In this case, after the hackers breached the original servers (Dispute Resolution), they were able to use commands to locate an additional 48 servers.  They were also able to locate more sensitive data, including usernames and passwords.  A real mess.

Keeping strong ‘walls’ between these servers would have prevented the chaos from spreading.  That would have limited the damage (albeit still serious) to the original three servers, not 51.

Monitor the Monitoring

Systems which involve access to critical servers (basically all mission critical Information Systems and any other IS that can touch them) should be automatically monitored for these kind of things – and that monitoring confirmed on an on-going basis.  Equifax claims a quarter of a billion dollars spent on such oversight didn’t do the job.⁴  It is hard to believe we are talking about the same things.

Part of the problem was traffic sent in small, encrypted, batches.  This made it easier to blend in with regular network traffic.

However, note that Equifax failed to keep a digital certificate updated.  By the time of the actual removal of data, that certificate had expired a year prior.

What that meant is that the encrypted data associated with that certificate wasn’t examined at all.  The size of datasets became irrelevant.  No one was at the switch.

Software is a Backstop

The balance of the problem may be a general assumption that security and scanning systems will catch issues and all significant threats.  But the  scanning security software should just be a backstop not a mainline of defense.

In my mind, it is like flying an aircraft on auto-pilot and assuming that system can handle every eventuality without human involvement.

In other words, automated systems should not replace competent independent analysis (such as patching reviews, system/process audits, & team reviews for gaps and security status).  All these seemed to have had issues here.

Some technology pros may scoff at patching meetings.  It is akin to a weekly grammar lesson among English PhD’s.

My response to IT pushback?  163 million consumers compromised, a billion dollar class action suit, a $575 million dollar FTC settlement. 

By Equifax’s own admission, the cost has been at least $1.7 Billion – and it warns the company may continue these financial costs into the future.

In short, this was a glaring gap in procedures that just hadn’t been exploited yet.  It is no different than a company blaming the loss of all trade secrets on the company founder getting hit by a bus.

Basic guidelines establishing institutional memory and processes preclude this defense.

There appear to have been minimal ramifications for Equifax for their failures here.  Rick Smith retired with a $90mm payout.  Or as Fortune fittingly states, a 57 cent premium for each consumer record that was compromised during the breach.  And yes, a handful of Equifax top execs lost their jobs.

Congressional outrage went from volcanic…to a steady simmer, to nothing of note.  There have been no regulatory changes because of the company’s failures.⁵

Yes Equifax is on the hook for over a billion, something paid for by shareholders.  But even there.  I question shareholder assessment of actual business practices.

The company’s stock price is considerably higher than it was.  Importantly, this is for a company with a fairly static model and limited offerings.

In fact, it’s up more than 150% from it’s low (as of this writing), shortly after the news of the hack went public, (see graphic above).  So this seems less a case of sea change, and more a case of let’s “move on”.

Conclusions

Quite a bit of what has been discussed here is low hanging fruit.  And even then, some of those easy wins haven’t been discussed because they’re so ancillary.  But it is helpful to remember the lesson to focus on the fundamentals first.

A corporate server with a username & password of “admin” in 2017.

Krebs on Security noted that an Equifax server focused on the Argentinian consumer market had a username & password of “admin”.  Yes, that’s a username of “admin” and a password of “admin”.  In 2017.

So, Equifax?  Yes, I think they’ve earned the bad press.

Businesses would do well to start with the simple before the complex.  I’m sure it is easy to pile on here.  And yes, it is easy to see mistakes with the benefit of hindsight.

It’s a complex landscape out there.  It is not uncommon for me to hear pitches from software vendors for very advanced security focused solutions.

Everything from investigative platforms to aid staff in disguising their whitehat work via geographic, timestamp, and language modification, to full-on Security Operations Centers (SOCs) or the outsourcing of same.  That stuff is great, and absolutely has a place.

A company doing everything right AND having those resources?  That is one impressive operation.

But here, again, our attack vectors are pretty revealing.  We are talking email distribution lists, poor practices concerning software patches, and outdated digital certificates.

This at a billion dollar company.

Keep It Personal

So as for parting advice for the rest of us, what can you do given these realities? I recommend you maintain a credit freeze on your personal credit accounts with the three major reporting agencies.  This will limit the damage to your credit and future financial self considerably.

Second, check your credit report regularly. By staggering your free annual reports from the three agencies on a quarterly basis, you are always only a few months away from an accurate read on what is recorded in your file.

And lastly, limit the extent of your personal information dissemination whenever possible. Does that big box retailer really need the information requested to activate a warranty or complete a purchase? Ask before you offer it.

These steps may seem extreme, but given the power of credit information in our society and the fallout from abuse, it also is a smart approach. Just in case someone else, “doesn’t get the memo”.

(1) https://time.com/3961676/history-credit-scores/
(2) https://www.nbcnews.com/business/business-news/target-settles-2013-hacked-customer-data-breach-18-5-million-n764031
(3) https://www.computerweekly.com/news/2240080607/TJX-hack-the-biggest-in-history
(4) https://www.nbcnews.com/business/consumer/former-equifax-ceo-blames-one-it-guy-massive-hack-n807956
(5) https://www.politico.com/story/2018/01/01/equifax-data-breach-congress-action-319631