Digital immune systems

“How can you go for a walk at a time like this?”

My woman didn't expect to see me putting on my waterproof coat. A few minutes earlier I’d gone into my home office to take a quick look at some unusual activity I'd noticed on the server logs. She knew something was wrong the minute I walked back into the kitchen. My face what white with shock. I’d discovered that criminals had attacked the servers that power my business.

An automated army of computers attacked the MarketingMotor servers on Boxing day. It wasn't personal. They attacked tens of thousands of other businesses in the same way.

Their automated attack installed a hidden back door for their human masters to use later. They’d picked Boxing day because that'd give them plenty of time before people got back to work in the new year.

I’d planned to spend the week between Christmas and New Year tinkering in my man-cave, instead I’d be fighting for my business. But before I could start the fight I needed a battle plan - and there is nothing like a stroll in the sleet to sharpen the mind.

There were more than 100 000 separate computer files on the server. I have no idea what 99 000 of them do. The computer code I wrote to build MarketingMotor is only the tip of the iceberg. It's build on foundations of millions of lines of computer code written by other people. Any of those foundation files could have been compromised.

The safest course of action was to dump everything on the server and start with afresh.

I’ve got good backups so I wasn't too worried about losing data. What did worry me was keeping the core MarketingMotor lead delivery engine running while restoring. My clients depend on the sales enquiries they get from it.

Can I share some lessons I learned from this with you?

Lesson 1: Nothing is safe anymore.

Hackers will break your website. Your email, your social media, your user names and passwords etc will be compromised. Dropbox, Elance and lost control over sensitive information they stored about me. (You can check if your personal information isn't so personal anymore here: )

Am I being pessimistic? Consider my reasons below and decide for yourself.

Criminals have access to military-grade cyber weapons.

There are two constants:

  1. Governments race each other to build better weapons.
  2. Those weapons end up in criminal hands.

The Thompson sub-machine gun was developed at great cost to give American troops an advantage in trench warfare. It ended up being far more famous as the "Chicago typewriter" - the weapon of choice for Al Capone and his mobsters.

The AK47 was the Soviet answer to German firepower after the second World War. They now feature in robberies every day.

Are governments going to be better at looking after computer code than they are at keeping tally of the 75 000 000 AK47s they've produced?

No business is too small to be worth attacking.

Past cyber attacks were often robberies. Criminals stole credit card details, email addresses, usernames and passwords because there is a market for such things. Bigger targets were more attractive.

  • 1 000 000 000 accounts were stolen from Yahoo in 2013 and another 500 000 000 in 2014.
  • Ebay asked 145 000 000 users to change passwords after hackers stole customer names, addresses and dates of birth in 2015.
  • Hackers revealed 157 000 TalkTalk customer’s details in 2015
  • etc etc.

The attack on my business wasn’t robbery, it was kidnapping.

The criminals plotted to copy the databases and then delete them. They'd have sent a badly punctuated ransom email asking for $2000 - $3000 in untraceable Bitcoin. Many businesses without backups pay rather than lose every customer record, email, invoice etc.

My web hosts told me the attack originated in Ukraine. The automated part of such an attack can be ordered for less than $10. The human part runs from a sweatshop. It's a viable business model.

All the foundations are cracked.

All web sites and computer programs are built on foundations of computer code. Usually many foundation layers. Your web site or program might be bullet proof but attackers can often get in through a weakness in one of those foundation layers.

Wordpress is a good example. It powers 1 in 4 websites today. Most of the businesses with a Wordpress website don't care about or understand the Wordpress plumbing. They just build their website with it.

Wordpress has been under constant attack since 2003. You'd think by now they would have figured out and fixed every chink in the armour. Not so. 20 previously unknown vulnerabilities were discovered in 2016 and 16 have been identified this year.

Wordpress - like every other program - rests on foundations of computer code. Those foundations have been around even longer than Wordpress.

The most common foundations are:

  • Linux had 218 new ways to break in discovered in 2016 and 136 in 2017.
  • Apache had 103 vulnerabilities discovered in 2016 and 2 so far this year.
  • MySql which had just 1 hole in 2016.
  • PHP - 107 previously unknown ways to break in in 2016 and 16 this year.

Nobody is too powerful to beat or too small to ignore. So what’s a girl to do? My thoughts if you will...

Lesson 2: You need a digital immune system.

The old way of thinking about computer security came from medieval castles. A big strong wall around the outside with a safe space inside. As long as you could build a bigger wall than your enemy could tear down your chattels were safe inside. Your guards watched the one gate and poured boiling oil on any trespassers.

But you couldn’t make Cape Town or London safe today by building a wall around it. There are too many roads, pipes and cables coming into the city to defend. But the real problem - you’re more likely to be robbed by someone living in the city than by a barbarian bashing at the gate.

Your digital life is more like a city than a castle. You can never defend every entry and exit point. You’ll never build a wall stronger than the attackers can break down. You'll never build one higher than they can fly over.

Instead we need to learn from our bodies. They live in a world filled with constant threats. At any given second millions of bacteria and viruses are trying to colonise us. Our immune system deals with this all day, every day. We rarely notice it backing up and restoring. The process is part of normal life. It's invisible apart from the occasional bout of man-flu.

Lesson 3: Redundancy.

Our bodies have redundancy - two eyes, ears, lungs, kidneys etc. It won’t be pleasant but you'd survive the loss of an eye, an ear or a kidney.

It doesn’t take a lot of set up or even much interruption in your workflow to set up many redundant backups. A combination of Dropbox and a portable hard drive will get you 90% of the way there. Storing stuff online will take you the rest of the way. I keep a month’s worth of daily backups on Amazon Web Services S3. Last month’s bill was $0.34.

Lesson 4: Practice restoring backups.

Restoring everything from scratch in January wasn't the desired routine invisible process. I'd never done it before (mea culpa) so it took several dummy runs before I was confident to do it live.

I'd recently read a great book called The Checklist Manifesto by Atul Gawande. He talks about how airline pilots have checklists of what to do in case of every emergency. A cabin door falls off at 30 000 feet? They don't try to figure out what to do in the stress (understatement) of the moment. Instead they flip to the "cabin door fell off" checklist and start working through the steps. The checklist discipline brought huge improvements to safety in air travel and surgery. It has application everywhere.

I made a checklist of the steps I'd have to take to restore from scratch while keeping the business running. I won’t bore you with the details. It took a shedload of work to prepare and test everything and about 15 minutes of downtime to restore from scratch and restart the core engine again.

I now practice restoring backups every Monday morning. I work my way through the checklist. It's not much fun but it's the only way to be sure that the immune system is working.

© Peter Bowen 2018 | Isle of Wight