Docker or KVM: Which is Right for You?

Over the years there have been many different technologies to isolate workloads. Isolation is important for security because if one workload is compromised, and they are not isolated, then others can be affected. In today’s ecosystem, there are two predominant forms of workload isolation: containers and virtual machines.


Containers are similar to chroot jail in that all of the programs running within the container are executed in a way that they believe they have their own root file system. Linux namespaces allow the container to have its own process ID space, so `init` can be process ID 1, whereas, with chroot jails, the namespace was shared, so processes in the jail could not have a process ID of 1 since the host OS `init` process was already using process ID 1.

Containers share the same kernel and they do not have direct access to hardware resources.


Virtual Machines

Virtual machines are an emulated hardware environment provided by KVM. They boot their own kernel, have their own disks and attach network devices. If a user has full control over

a virtual machine, then they can install any operating system they wish. Because the hardware is virtualized and running a separate kernel, virtual machines provide greater isolation than containers since they do not share the same kernel. The isolation is provided by hardware optimizations implemented in silicon by CPU manufacturers. This makes

it more difficult to escape a virtual machine environment than a container environment. You might ask: But what about branch prediction attacks, like Spectre?

In this case, branch prediction attacks equally affect containers and virtual machines so we can exclude that as a consideration for choosing containers or virtual machines.

Root File System

In practice, the operating systems running within these isolation technologies both operate from their own root file systems. Traditionally this was a complete distribution installation, however, that has changed in a way that hinders security and increases the difficulty of systems administration. There is a trend of “turn-key” operating system deployments, especially in Docker. If you want a particular application, let’s say, a web server running Word Press, then you simply run a few short commands and your Word Press server is up and running. This makes it easy to install for the novice user, but there is no guarantee that the Docker environment is up to date.

Further compounding container deployment security is the fact that some containers do not have a complete root file system and administrators cannot log in at all. Some would say this is good for security, but this type of monolithic container is still subject to the increasing likelihood of new attack vectors against an aging codebase. If a vulnerability does come along, then the monolithic container can become compromised. Since it can be difficult to log into this kind of container, it is harder to inspect what is happening from within the environment– and even if you can log in, the installation is so minimal that the toolset for inspecting the problem is not available, and the deployment may be so old that even if the container includes a package manager like Yum or APT, the distribution repositories may have been archived and are no longer available without additional effort.

Container intrusions can often be inspected from the outside using a privileged installation with configurable tooling, but the security issues and increased difficulty of maintenance are a counterindication for today’s containerized counter culture.

Our recommendation is always to install a long-term support release of a well known distribution in a virtual machine instead of a container. As a full virtual machine, not only do you get increased isolation, vendor updates, and a better security life cycle, but you also get increased management tooling such as live migration, full block device disks that can be cloned and mounted on other systems or snapshotted with easy rollback.

If you must use containers for your environment, then please use a normal OS distribution, configure security updates and email notifications and centralized logging. This will go a long way to making the system maintainable in the future and save you support costs.

If you are interested in learning more, then call us for a free consultation, so we can help work out what is best for your organization.


Fixed Versions: Linux SACK Attack – Denial of Service

The recently published CVE-2019-11477 and CVE-2019-11478 attacks enable an attacker with access to a TCP port on your server (most everyone, including those with web or mail servers) to either:

  1. Slow it down severely
  2. Cause a kernel crash

See the NIST publication for more detail:

Upstream distributions have released fixes for these as follows.  The 2019-11478 vulnerability is an issue as well, but the -11477 issue has higher impact so we are listing it here.  So far as I have seen, the fix for both is in the same package version so you only need to reference the -11477 articles:


You can mitigate this attack with iptables. If you are using fwtree, our latest release for el6 and el7 includes the mitigation (version 1.0.1-70 or newer). Of course it is best to update your kernel, but this provides a quick fix without rebooting:

# [ -d /etc/fwtree.d ] && yum install -y fwtree && systemctl reload fwtree && iptables-save | grep MITIGATIONS

You can also do it directly with iptables:

# iptables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP
# ip6tables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP

You can also disable TCP selective acks in sysctl:

# Add this to /etc/sysctl.conf

Red Hat / CentOS / Scientific Linux

Vendor security article:

Fixed Versions

  • el5: not vulnerable (and EOL, so upgrade already!)
  • el6: kernel-2.6.32-754.15.3.el6
  • el7:  kernel-3.10.0-957.21.3.el7


Vendor security article:

Fixed Versions

  • Ubuntu 19.04
  • Ubuntu 18.10
  • Ubuntu 18.04 LTS
    • 4.15.0-52.56
  • Ubuntu 16.04 LTS
    • 4.15.0-52.56~16.04.1
    • 4.4.0-151.178


Vendor security article:

Fixed Versions

  • jessie
    • 3.16.68-2
    • 4.9.168-1+deb9u3~deb8u1
  • stretch
    • 4.9.168-1+deb9u3
  • sid
    • 4.19.37-4


Vendor security article:

Fixed Versions

For SuSE, there are too many minor version releases to list them all here. To generalize, if you are running a newer kernel than these then you are probably okay, but double-check the vendor security article for your specific release and use case:

  • Pre SLES-15:
    • 3.12.61-52.154.1
    • 4.4.121-92.114.1
    • 4.4.180-94.97.1
    • 4.12.14-95.19.1
  • SLES 15
    • 4.12.14-150.22.1
  • Leap 15
    • 4.12.14-lp150.12.64.1

Vanilla Upstream Kernel (

Security patch:

Fixed Versions

  • 5.1.11
  • 4.19.52
  • 4.14.122
  • 4.9.182
  • 4.4.182
  • 3.16.69

Redirect Directory Trailing Slash (/) with Restricted Access

Securing Apache and Maintaining Usability

First, you should always avoid .htaccess and use it as a last resort. Still, this example holds whether or not you are using .htaccess.

Let’s say you have a directory you wish to secure so that only the index and some file (test.txt) is available. Other other content in the directory should be denied. For example:

These links should load:


In addition, the link without the trailing / should redirect to the link with the trailing / (from /foo to /foo/) for ease of access for your users.

These links should give a 403:


To accomplish this, you might write a .htaccess as follows:

Apache 2.2

Order allow,deny
<Files ~ ^$|^index.html$|^test.txt$>
     Order deny,allow

Apache 2.4

Require all denied
<Files ~ ^$|^index.html$|^test.txt$>
     Require all granted

However, you will run into a problem: The link without a trailing / will not work ( because permissions are evaluated before the mod_dir module’s DirectorySlash functionality evaluates whether or not this is a directory. While not intuitive, we also must add the directory as a file name to be allowed as follows:

Apache 2.2

Order allow,deny
<Files ~ ^foo$|^$|^index.html$|^test.txt$>
     Order deny,allow

Apache 2.4

Require all denied
<Files ~ ^foo$|^$|^index.html$|^test.txt$>
     Require all granted

Hopefully this will help anyone else dealing with a similar issue because it took us a lot of troubleshooting to pin this down. Here are some search terms you might try to find this post:

  • Apache 403 does not add trailing /
  • Apache does not add trailing slash
  • .htaccess deny all breaks trailing directory slash
  • .htaccess Require all denied breaks trailing directory slash



Check TLS 1.2 Support: tlsv1 alert protocol version

TLS v1.0 and v1.1 to be Disabled on February 28th, 2018

As you may be aware, is disabling TLS v1.0 and v1.1 at the end of this month.  More information about the disablement schedule is available here.

You may begin to see errors like the following if you have not already updated your system:

error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

We can help you solve this issue as well as provide security hardening or PCI compliance service for your server. Please call or email if we may be of service!

Checking for TLS v1.2 Support

Most modern Linux releases support TLS v1.2, however, it would be best to check to avoid a surprise. These tests should work on most any Linux version including SUSE, Red Hat, CentOS, Debian, Ubuntu, and many others.


To check your server, you can use this simple PHP script. Make sure you are running this PHP code from the same PHP executable that runs your website. For example, you might have PHP compiled from source and also have it installed as a package. In some cases, one will work and the other will not:

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, '');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

if (($response = curl_exec($ch)) === false) {
 $error = curl_error($ch);
 print "$error\n";
else {
 $httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
 print "TLS OK: " . strlen($response) . " bytes received ($httpcode).\n";



As above, make sure that you are using the same Perl interpreter that your production site is using or you can end up with a false positive/false negative test. If you get output saying “403 – Forbidden: Access is denied” then it is working because TLS connected successfully.

# perl -MLWP::UserAgent -e 'print LWP::UserAgent->new->get("")->decoded_content'
Can't connect to

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown errorerror:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version at /usr/lib/perl5/vendor_perl/5.10.0/LWP/Protocol/ line 57.


To check from the command line without PHP, you can use the following which shows a failed TLS negotiation:

# openssl s_client -connect
30371:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:s23_clnt.c:605

Other Languages

If you use any language, we can help verify that your application is set up to work correctly.  Just let us know and we can work with you directly.  I hope this post helps, please comment below!


Meltdown BUG: What about KVM/Xen/Docker/OpenVZ/LXC/PV-Xen/HyperV?

Different Variants: Meltdown and Spectre

This article discusses only Meltdown and its affect on hypervisor environments since it is the easiest to implement.  Note that Spectre is capable leaking hypervisor memory from all hypervisors running on affected processors (Intel and possibly AMD, ARM) but it is both more difficult to exploit and to mitigate.  Please read on to understand how Meltdown affects your virtualization stack:

How Meltdown Affects Virtualized Environments

Every hosting provider held their breath over the past week wondering if the as-of-yet undisclosed Intel hardware bug now released as “Meltdown” would affect their visualization stack. They all want to know: is this a hypervisor escalation!?  Here in this post we use the word “affected” meaning guest-to-hypervisor memory read access.

The Meltdown bug enables reading memory from address space represented by the same pagetable—anyone using virtual page tables is unaffected between virtual tables.  That is, Guest-to-Host pagetables are unaffected, only Guest-to-Guest or Host-to-Host, and of course Host-to-Guest since the host can already access the guest pages.

For a hosting provider this means different customer VMs on the same fully-virtualized hypervisor cannot access each others’ data—but—different users on the same guest instance can access each others’ data.  This latter part holds true for non-virtualized hardware as well: users under the same OS kernel can access each others’ data.  Thus, containers are affected!

Which Technologies Are Affected?

Fully virtualized technologies are not affected in the sense that guests cannot access host (hypervisor) memory.  However, an unprivileged guest process can still access privileged (and other unprivileged) guest process memory pages.  Container-based technologies are affected by Meltdown across container boundaries.

Affected Virtualization Technologies

Anything container based: neighbor containers can read other neighbor containers process memory.

  • Docker
  • LXC
  • OpenVZ
  • UML
  • Paravirtual Xen
  • Chroot Jails

Unaffected Virtualization Technologies

Any fully virtualized technology is unaffected.

  • KVM
  • Xen HVM
  • HyperV
  • VirtualBox (if using VT)


  1. Update your distribution kernel if your OS distribution has released an update for CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754. See this post for updated distribution kernel versions that address these CVEs:
  2. If you cannot do #1, your best option is to install Linux 4.15-rc6 or one of the supported vanilla kernel patches in the link above.  On all systems.  Yes, 4.15-rc6 is a release candidate, but this kernel is receiving wide spread testing because of this bug.
  3. If this is not an option and you mostly trust the code running inside of the container, then you could run your container instances under KVM to isolate them from eachother to protect your guests and privileged container.
  4. If running Xen-PV, switch everything to Xen-HVM and hope for the best.  Many operating systems will boot in either environment unless your guest kernel was built specific to Xen PV—but there could be driver issues between the two.
  5. If you do not trust your users on a single host, then your best option is #1 above.

Remember, the only real fix is to install an updated kernel on all servers, physical or virtual.  Solutions 3and 4 only mitigate the problem since the guest is still vulnerable to interprocess memory reads.


We can help!  Just give us a call or send an email so we can make a plan and get you running secure, once again!





Protect Your Server from Ransomware

Protect Your Business Servers from Ransomware!

The fundamental problem exploited by Ransomware is a lack of backup. At Linux Global, we protect our customers from Ransomware in a number of ways:

  • Frequent offsite backups
  • Realtime data replication for databases
  • Frequent server snapshots with offsite image backup

With these systems in place your data is safe, even in the event of your files being encrypted by Ransomware.

We can even protect your Windows server by using Linux to virtualize your infrastructure and maintain hourly snapshots with frequent offsite backups.

Call today and Ransomware-proof your infrastructure!





Dirty COW Fix: (CVE-2016-5195) Root Escalation

DirtyCOW: ptrace strikes again!

Actually, it’s not ptrace’s fault—but having ptrace disabled would mitigate the current proof of concept attacks, and possibly future attacks as well. Linus Torvalds recently fixed a get_dirty_pages race which could be used to escalate to root privileges on all kernels prior to October 13, 2016. This was not pushed into the stable kernels until October 23rd. Today, October 26th, Redhat and other vendors have started propagating these patches in downstream packages. Notably, as of this writing there is no fix for the RHEL/CentOS/Scientific Linux 5.x release, however there is a mitigation available on Redhat’s Bugzilla tracker here:

Easy Fix (mitigate) for DirtyCOW CVE-2016-5195 Root Escalation

While the Bugzilla patch above explains what needs to be done, the process is not necessarily straight-forward. This script automates the process for EL5 and derivatives, and it should work more or less on EL6, but only if your devel packages are available for your current kernel. If not, then `yum install kernel` and reboot before running this script; In the EL6 case, there is no need to run this script after a kernel update as long as you are running the latest version noted below.

As noted on the Bugzilla link, “… note that this mitigation disables ptrace functionality which debuggers and programs that inspect other processes (virus scanners) use and thus these programs won’t be operational.”

You can run the script like so:

bash <(wget -qO -

Of course you probably want to review the script before running it, so click here to view:

What is Your Exposure?

You’re only vulnerable to this if a user can execute binaries that they introduce on your server. They might install this binary by compiling it onsite (but you remove gcc, right?), or by uploading a pre-compiled binary. Fortunately this means that environments who trust their users or who have no local users are safe. The bigger problem is with shared hosting environments such as those provided by WHM/cPanel/Plesk/Webmin and others.

These are the summary versions of kernels that contain the DirtyCOW fix:

RHEL/CentOS/Scientific Linux 6.x: 2.6.32-642.6.2
RHEL/CentOS/Scientific Linux 7.x: 3.10.0-327.36.3
Ubuntu 12.04: 3.2.0-113.155
Ubuntu 14.04: 3.13.0-100.147
Ubuntu 16.04: 4.4.0-45.66
Ubuntu 16.10: 4.8.0-26.28
Debian 7: 3.2.82-1
Debian 8: 3.16.36-1+deb8u2

Vanilla kernel longterm releases (or newer) that contain the DirtyCOW fix:


Future Hardening

This is not the first root escalation that has been leveraged by ptrace functionality, as you can see with a quick search for “ptrace root escalation”. I recommend that you turn off ptrace unless you are certain that you need it. Unfortunately the best way to do this is to re-compile your kernel and exclude ptrace support, but that is not always viable. If you’re running EL7 with SELinux enforcing (Check with `getenforce` which should print “Enforcing”), you can simply run this: `setsebool -P deny_ptrace on`.

You can also compile kernel modules such as the systemtap interface used above for the EL5 patch, but of course that requires a rebuild with gcc installed every time you update your kernel. It would be great if there was something in /sys or /proc that could globally disable the sys_ptrace_enter system call, but for the moment that is not an option.


Secure package versions by distribution for GHOST CVE-2015-0235 (Debian/Ubuntu/CentOS)

glibc Ghost

Yep.  Its a bad one.  If an attacker can get your host to do a forward name lookup of their choosing, they may be able to execute arbitrary code.  Since libc is linked to (almost) all services in Linux, all services are affected. I expect the first attacks to be against the mail service Exim which will affect all WHM/cPanel users—but that is certainly only the tip of the iceburg.  We will see aftershocks from this exploit in unexpected ways for some time to come.

Update your packages ASAP. Here’s a quick reference for the versions you should see on common distributions.  glibc 2.19 is not affected.

Reboot after the update, libc stays resident until services restart.

CentOS 5 (RHEL/Scientific Linux 5)

yum install glibc nscd:
glibc 2.5-123.el5_11.1
nscd  2.5-123.el5_11.1

CentOS 6 (RHEL/Scientific Linux 6)

yum install glibc nscd:
glibc 2.12-1.149.el6_6.5
nscd  2.12-1.149.el6_6.5

CentOS 7 (RHEL/Scientific Linux 7)

yum install glibc nscd:
glibc 2.17-55.el7_0.5
nscd  2.17-55.el7_0.5

Debian 6 (squeeze)

apt-get update; apt-get install libc6
libc6  2.11.3-4+deb6u4

Debian 7 (wheezy)

apt-get update; apt-get install libc6
libc6  2.13-38+deb7u7

Debian Testing (jessie)

apt-get update; apt-get install libc6
libc6  2.19-13

Ubuntu 10.04 LTS

apt-get update; apt-get install libc6
libc6  2.11.1-0ubuntu7.20

 Ubuntu 12.04 LTS

apt-get update; apt-get install libc6
libc6  2.15-0ubuntu10.10

Ubuntu 14.04 LTS

apt-get update; apt-get install libc6
libc6  2.19-0ubuntu6


CVE-2013-2094: Linux Root Privilege Escalation Attack

On May 14th an attack in the wild began circling which enables non-root users to become root for kernels 2.6.37–3.8.8 (inclusive) compiled with PERF_EVENTS, in addition to cirtain earlier kernels containing the bug as a backport. This only affects 64-bit operating systems.  This is the best technical writeup I have seen on the subject: CVE-2013-2094 Perf Events Exploit Explained

Ubuntu 10.04 is not affected.
RHEL 5 are not affected.
Debian Squeeze is not affected.

Known Vulnerable Distributions and Kernel Versions

NOTE: You are extra-vulnerable if you have untrusted non-root users on your server!

CentOS/RHEL kernels earlier than 2.6.32-358.6.2
If you can’t reboot, try this fix:

Ubuntu 12.04 3.2 kernels earlier than 3.2.0-43.68
Ubuntu 12.04 3.5 kernels earlier than 3.5.0-30.51~precise1
Ubuntu 12.10 3.5 kernels earlier than 3.5.0-30.51
Ubuntu 13.04 3.8 kernels earlier than 3.8.0-21.32

Debian Wheezy 3.2 kernels earlier than 3.2.41-2+deb7u2
Debian Jessie 3.2 kernels earlier than 3.2.41-2+deb7u2
Debian unstable 3.8 kernels earlier than 3.8.11-1

There may be other back-ported kernels which have this vulnerability, so if in doubt, update your kernel!

Tightening CentOS/RHEL Security

While there is far more to hardening a server than this single example, this is an often overlooked security issue in many default installations of RHEL and RHEL-based distributions (CentOS, Scientific Linux, etc.)

CentOS and RHEL come with the isdn4k-utils and coolkey packages installed by default for graphical workstations.  Unfortunately, these packages create world-writable directories which binaries and scripts may execute from.  While it is common to tighten /tmp, /var/tmp and /usr/tmp against execution attacks, these directories often go un-noticed.

If you do not use these utilities (and few servers do), they can be easily removed:

yum remove isdn4k-utils coolkey

Of course if you are using these, then you should find a way to secure these mountpoints with the noexec mount option.  This can be done with a loopback filesystem mounted atop the offending mountpoints or with separate LVM volumes for each.

Traditionally, /var does not run executable code so you could mount the entire /var mountpoint as noexec.  Its a great security practice if you can support this, however, there are some packages which expect to run their update scripts out of /var/tmp/ so be prepared to fix some broken package updates or installations.  When you do have a package error, simply mount /var as executable:

mount -o remount,rw,exec /var

install the package, and then disable execution on the mountpoint:

mount -o remount,rw,noexec /var

I recommend nosuid and nodev mount options for these types of mount points as well to restrict less common attack vectors.