Bash Remote Exploit & Amazon AWS Emergency Maintenance: Not The Best 24 Hours For DevOps Teams

by Pere Hospital  September 25, 2014

Many Ops teams have their heads spinning and descended into a frenzy of activity with two unrelated but major news that have broken in the last 24 hours:

  • A serious flaw was discovered in Bash (a very popular UNIX shell) that is remotely exploitable. Exploit is also known as “Shellshock”.
  • And, few hours later, Amazon announced that they are going to reboot a large swathe of their instances in next few days due to an undisclosed security patch that needs to be applied to hosts.

Secure-Servers-Banner

As said, this is hell of a work for DevOps teams!

 

Bash Exploit (aka “Shellshock”): What it is and what have we done?

There is no need to get into technical details, as there is a lot of information available already.

The attack vector is via HTTP/CGI scripts and also through OpenSSH and is related to how environment variables are processed. The breath of the attack surface means that you need to patch all your Linux servers NOW. There is already Metasploit code to exploit it, so time is clearly running out.

At Cloudways, we have already patched all our customers infrastructure as well as our own. DevOps comes handy in  such a situation and mcollective is an outstanding tool when it comes to orchestrating deployments like this on distributed networks with thousands of servers and that must be done at full speed.

AWS Emergency Maintenance: What it is and what have we done?

As if the above incident was not enough, yesterday most (if not all) AWS customers received an email from Amazon about an upcoming emergency maintenance. The email says that an emergency maintenance was needed for many of the hosts where their instances are running and that this would start to happen from as soon as this Saturday.

This implies the reboot of the host, so of all the instances on it. As usual, no specific information about the patch to be deployed was released by Amazon. This is all we have been able to find out from an answer of an AWS representative in a forum post:

“Unfortunately there is no way to reschedule this, these are very timely security and operational patches that need to be applied to a portion of our EC2 instance fleet.”

What makes the situation worst is this bit in the email many of you would have received:

“We will need to do this maintenance update in the window provided. You will not be able to stop/start or re-launch instances in order to avoid this maintenance update. “

This is not like the situations of the past. Typically, when a host needs a patch and your instance is on it, you can just reboot it and the instance will start on a different already patched or unaffected host. This means that you can control when and how your instances will be rebooted.

The sentence in the email implies that in this particular situation, such a practice is not likely. If you reboot your instance now, there are no warranties that it will land on a safe host and so it will be likely rebooted all the same in assigned maintenance window. Truly bad for people with hundreds of AWS instances, which have been scheduled to be rebooted over the weekend and who have two days to scramble people and reorganize shifts to be able to ensure everything comes back normally after the reboots. It must be very serious stuff for AWS to risk the backslash that this will certainly bring.

As of now, the best course of action seems to try to do a controlled reboot of critical instances (that you can’t afford to be rebooted in the AWS assigned maintenance window) and after 12 to 15 hours, check if they appear again in the Events List in your AWS Management Console.

It seems that AWS is trying to reduce this 12 to 15 hours period that you need to wait before knowing if your instances have landed on a safe host to something more reasonable. You can follow this thread for updates on it.

And, this is precisely what we are doing.

So, let me get back to work now! But, do enjoy your weekend. We’re here to handle these technical troubles for you.

Debian Bash Update: Fri 26/09 09:05 UTC

I wanted to put seconds in the update title as the situation is extremely fluid. 🙂

Yesterday, we rolled Debian Bash patch 4.2+dfsg-0.1+deb7u1. However, this seems but I was already reading yesterday evening that, as more information is known about Shellshock and new attack vectors appear, it was highly likely that new patches would be needed. Alas, this morning another patch 4.2+dfsg-0.1+deb7u3 was released and we have just completed another roll-out process to all our systems.

From what I hear in the InfoSec community, an in-depth review is being done of the affected packages, and it is highly likely that more patches will come over the weekend. It seems we’ll be busy this weekend.

For more information, keep checking the following pages:

Debian Security Tracker  https://security-tracker.debian.org/tracker/CVE-2014-6271

Cloudways Status Page http://status.cloudways.com/

Start Creating Web Apps on Managed Cloud Servers Now

Easy Web App Deployment for Agencies, Developers and E-Commerce Industry.

About Pere Hospital

Pere Hospital (CISSP & OSCP) is the CTO and co-founder of Cloudways Ltd. He has over two decades of experience in IT Security, Risk Analysis and Virtualization Technologies. You can follow Pere on Twitter at @phospital and read his blog at www.perehospital.cat

Stay Connected:

You Might Also Like...