librsync error: “RS_DEFAULT_STRONG_LEN” undeclared

We needed to compile an old version of rdiff-backup on CentOS 7 but got the following error:

 _librsyncmodule.c: In function â_librsync_new_sigmakerâ:
_librsyncmodule.c:63:17: error: âRS_DEFAULT_STRONG_LENâ undeclared (first use in this function)
 (size_t)RS_DEFAULT_STRONG_LEN);
 ^
_librsyncmodule.c:63:17: note: each undeclared identifier is reported only once for each function it appears in
_librsyncmodule.c:63:9: error: too few arguments to function ârs_sig_beginâ
 (size_t)RS_DEFAULT_STRONG_LEN);
 ^
In file included from _librsyncmodule.c:25:0:
/usr/include/librsync.h:370:11: note: declared here
 rs_job_t *rs_sig_begin(size_t new_block_len,
 ^
error: command 'gcc' failed with exit status 1

The librsync library changed the colling convention of `rs_sig_begin` so if you get an error like that, then a patch like this might help:

]# diff -uw _librsyncmodule.c.ORIG _librsyncmodule.c
--- _librsyncmodule.c.ORIG 2006-11-11 23:32:01.000000000 -0800
+++ _librsyncmodule.c 2018-02-20 11:22:06.529111816 -0800
@@ -59,8 +59,8 @@
 if (sm == NULL) return NULL;
 sm->x_attr = NULL;

- sm->sig_job = rs_sig_begin((size_t)blocklen,
- (size_t)RS_DEFAULT_STRONG_LEN);
+ sm->sig_job = rs_sig_begin((size_t)blocklen, 8,
+ (size_t)RS_MD4_SIG_MAGIC);
 return (PyObject*)sm;
 }

-Eric

Check Authorize.net TLS 1.2 Support: tlsv1 alert protocol version

TLS v1.0 and v1.1 to be Disabled on February 28th, 2018

As you may be aware, Authorize.net is disabling TLS v1.0 and v1.1 at the end of this month.  More information about the disablement schedule is available here.

You may begin to see errors like the following if you have not already updated your system:

error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

We can help you solve this issue as well as provide security hardening or PCI compliance service for your server. Please call or email if we may be of service!

Checking for TLS v1.2 Support

Most modern Linux releases support TLS v1.2, however, it would be best to check to avoid a surprise. These tests should work on most any Linux version including SUSE, Red Hat, CentOS, Debian, Ubuntu, and many others.

PHP

To check your server, you can use this simple PHP script. Make sure you are running this PHP code from the same PHP executable that runs your website. For example, you might have PHP compiled from source and also have it installed as a package. In some cases, one will work and the other will not:

<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'https://apitest.authorize.net');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

if (($response = curl_exec($ch)) === false) {
 $error = curl_error($ch);
 print "$error\n";
}
else {
 $httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
 print "TLS OK: " . strlen($response) . " bytes received ($httpcode).\n";
}

curl_close($ch);
?>

Perl

As above, make sure that you are using the same Perl interpreter that your production site is using or you can end up with a false positive/false negative test. If you get output saying “403 – Forbidden: Access is denied” then it is working because TLS connected successfully.

# perl -MLWP::UserAgent -e 'print LWP::UserAgent->new->get("https://apitest.authorize.net")->decoded_content'
Can't connect to apitest.authorize.net:443

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown errorerror:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version at /usr/lib/perl5/vendor_perl/5.10.0/LWP/Protocol/http.pm line 57.

OpenSSL/Generic

To check from the command line without PHP, you can use the following which shows a failed TLS negotiation:

# openssl s_client -connect apitest.authorize.net:443
CONNECTED(00000003)
30371:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:s23_clnt.c:605

Other Languages

If you use any language, we can help verify that your application is set up to work correctly.  Just let us know and we can work with you directly.  I hope this post helps, please comment below!

-Eric

Spectre / Meltdown: Security Updates by Distribution

Secured Kernel Versions by Distribution and Version

CentOS/RHEL/Scientific Linux

  • Version 7.x: 3.10.0-693.11.6
  • Version 6.x: 2.6.32-696.18.7

Debian

  • Sid: 4.14.12-2
  • Stretch: 4.9.65-3+deb9u2
  • Jessie: 3.16.51-3+deb8u1
  • Wheezy: 3.2.96-3

Ubuntu

Testing kernels are available here: ppa:canonical-kernel-team/pti

  • Trusty 14.04: 4.4.0-108.131~14.04.1 (linux-lts-xenial)
  • Xenial 16.04: 4.4.0-108.131
  • Artful 17.10: 4.13.0-24.28

SUSE

Arch Linux

  • 4.14.11-1
  • 4.9.75-1

Vanilla from Kernel.org

These are per stable branch release.  Eg, 4.9.75 is protected, but that doesn’t mean 4.14.0 is.  The minor ‘y‘ of 4.x.y is the important number here:

  • >= 4.15-rc6
  • >= 4.14.11
  • >= 4.9.75
  • >= 4.4.110
  • >= 3.2.98

Meltdown BUG: What about KVM/Xen/Docker/OpenVZ/LXC/PV-Xen/HyperV?

Different Variants: Meltdown and Spectre

This article discusses only Meltdown and its affect on hypervisor environments since it is the easiest to implement.  Note that Spectre is capable leaking hypervisor memory from all hypervisors running on affected processors (Intel and possibly AMD, ARM) but it is both more difficult to exploit and to mitigate.  Please read on to understand how Meltdown affects your virtualization stack:

How Meltdown Affects Virtualized Environments

Every hosting provider held their breath over the past week wondering if the as-of-yet undisclosed Intel hardware bug now released as “Meltdown” would affect their visualization stack. They all want to know: is this a hypervisor escalation!?  Here in this post we use the word “affected” meaning guest-to-hypervisor memory read access.

The Meltdown bug enables reading memory from address space represented by the same pagetable—anyone using virtual page tables is unaffected between virtual tables.  That is, Guest-to-Host pagetables are unaffected, only Guest-to-Guest or Host-to-Host, and of course Host-to-Guest since the host can already access the guest pages.

For a hosting provider this means different customer VMs on the same fully-virtualized hypervisor cannot access each others’ data—but—different users on the same guest instance can access each others’ data.  This latter part holds true for non-virtualized hardware as well: users under the same OS kernel can access each others’ data.  Thus, containers are affected!

Which Technologies Are Affected?

Fully virtualized technologies are not affected in the sense that guests cannot access host (hypervisor) memory.  However, an unprivileged guest process can still access privileged (and other unprivileged) guest process memory pages.  Container-based technologies are affected by Meltdown across container boundaries.

Affected Virtualization Technologies

Anything container based: neighbor containers can read other neighbor containers process memory.

  • Docker
  • LXC
  • OpenVZ
  • UML
  • Paravirtual Xen
  • Chroot Jails

Unaffected Virtualization Technologies

Any fully virtualized technology is unaffected.

  • KVM
  • Xen HVM
  • HyperV
  • VirtualBox (if using VT)

Solutions

  1. Update your distribution kernel if your OS distribution has released an update for CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754. See this post for updated distribution kernel versions that address these CVEs: https://www.linuxglobal.com/spectre-meltdown-security-updates-distribution/
  2. If you cannot do #1, your best option is to install Linux 4.15-rc6 or one of the supported vanilla kernel patches in the link above.  On all systems.  Yes, 4.15-rc6 is a release candidate, but this kernel is receiving wide spread testing because of this bug.
  3. If this is not an option and you mostly trust the code running inside of the container, then you could run your container instances under KVM to isolate them from eachother to protect your guests and privileged container.
  4. If running Xen-PV, switch everything to Xen-HVM and hope for the best.  Many operating systems will boot in either environment unless your guest kernel was built specific to Xen PV—but there could be driver issues between the two.
  5. If you do not trust your users on a single host, then your best option is #1 above.

Remember, the only real fix is to install an updated kernel on all servers, physical or virtual.  Solutions 3and 4 only mitigate the problem since the guest is still vulnerable to interprocess memory reads.

Help!

We can help!  Just give us a call or send an email so we can make a plan and get you running secure, once again!

-Eric

 

 

 

Recovering from a Cloud Server Failure

If you have received an email or a support ticket from your provider like the message below, then chances are your data is no longer available. We may be able to work with your provider to recover data and would be happy to help if this could be done. Cloud Servers fail for a number of reasons, and not all failures cause permanent data loss.

Your cloud server could not be recovered due to the failure of its host. You have the option to rebuild your server from your most recent server image or from a stock image … We apologize for any inconvenience this may have caused you.

If there is permanent data loss, then your only option is to plan for the future and move ahead—and we can help with that too.

Backups are certainly useful for recovering data, but they take a long time to put back into service. As you deploy your new environment, consider redundant designs for your infrastructure and let us know if we can help prevent this in the future!

-Eric

Protect Your Server from Ransomware

Protect Your Business Servers from Ransomware!

The fundamental problem exploited by Ransomware is a lack of backup. At Linux Global, we protect our customers from Ransomware in a number of ways:

  • Frequent offsite backups
  • Realtime data replication for databases
  • Frequent server snapshots with offsite image backup

With these systems in place your data is safe, even in the event of your files being encrypted by Ransomware.

We can even protect your Windows server by using Linux to virtualize your infrastructure and maintain hourly snapshots with frequent offsite backups.

Call today and Ransomware-proof your infrastructure!

 

 

 

 

CentOS/RHEL/Scientific Linux 5 404 Not found

Problems Updating Packages

You may have landed here because of errors like the following:

 ~]# yum upgrade
Loaded plugins: fastestmirror, security
Determining fastest mirrors
 * base: mirrors.kernel.org
 * extras: mirror.chpc.utah.edu
 * updates: mirror.steadfast.net
http://mirrors.kernel.org/centos/5.11/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.

This is because on March 31st, all of the EL5 derivatives are now end of life. It is time to update to EL6 or EL7.

If you are unable to update immediately, but would like to get your OS as up to date as possible, then CentOS users can update their repositories from mirror.centos.org to vault.centos.org in /etc/yum.repo.d/CentOS-*.conf . For Scientific Linux, you will want to use the following link in your repository files: http://ftp.scientificlinux.org/linux/scientific/obsolete/511/$basearch/SL/.

We have migrated many CentOS/RHEL/Scientific Linux systems and would be happy to assist if you need help!

-Eric

Issues Upgrading CentOS/RHEL/Scientific Linux 7.2 to 7.3

If you’ve been a systems administrator for awhile, then you know it’s best practice to have security updates to install automatically—and you also know that this breaks things from time to time. This happened to use when EL 7.3 came out a few months ago and caused unexpected issues with systems running KVM, libvirt, and LVM2 with large quantities of snapshots (4,480 and counting!).

The first issue that we discovered was virtual machine lockup during live migration. This is related to an MSR_TSC_AUX update that Redhat pushed into 7.3, but for which the Linux 4.1.y stable branches had not yet merged the kernel update to support this. While I’ve not yet tested 4.1.39, it appears to have those patches. Most users will not experience this particular bug if they are using the vendor provided EL7 kernel—but if you are using 4.1 in order to have stable bcache support, then you might run into this. You can read more details on the patches here: https://patchwork.kernel.org/patch/9538171/

Shortly after we discovered the first issue (but before we had time to fix it), we discovered that LUKS passthrough crashes libvirt unless you are using libvirt’s keystore. Since we pass encrypted volumes directly into the virtual machine and let the virtual machine unlock the volume, this was causing endless segmentation faults of libvirtd as systemd restarted it after failure. After much troubleshooting and inspection with GDB to figure out where the problem actually was, we discovered that libvirt was assuming that all LUKS volumes have a key in their keystore. This has been fixed in the latest version, and more information about this is available here: https://bugzilla.redhat.com/show_bug.cgi?id=1411394

Not to be outdone, the 7.2 to 7.3 upgrade was also causing segmentation faults of dmeventd. At the time, we did not know that it was a bug in LVM2—but having a third issue compounded with the two above, it was time for more drastic measures: Revert the packages! After installing the EL7.2 version of libvirt, KVM, and LVM2 (and their dependencies), we were back up and running.

Feeling brave, we decided to try the 7.3 upgrade again today since the first two issues were fixed. At the time, we didn’t really know the third issue was an issue independent of the others, so this was our first opportunity to investigate. This issue is still outstanding, and the actual problem is unclear. We have found the first bad commit (9156c5d dmeventd rework locking code) in LVM2 and posted to the lvm-devel list, so hopefully this will be fixed soon. For the moment we are holding back LVM2 updates which seems to be working fine with the rest of the system packages upgraded to 7.3. You can read more about the beginning of this fix here: https://lvm-devel.redhat.narkive.com/xxKNaNG6/bisect-regression-segv-9156c5d-dmeventd-rework-locking-code

So is it time to 7.3 from 7.2? Yes! But only if you hold back LVM2. The easiest way to do this is to add the following to your /etc/yum.repo.d/CentOS-Base.repo in the [base] and [updates] sections:

exclude=lvm2* device-mapper*

Update: Tue Apr 4 16:25:39 PDT 2017

The LVM problem was related to the reserved_stack value in /etc/lvm/lvm.conf being too high on our system. Somehow this introduced a regression in LVM2 since it certainly worked before in EL7.2 .

So, if you get an error like this, shrink your reserved_stack and see if it fixes the problem:

kernel: dmeventd[28383]: segfault at 7f9477240ea8 ip 00007f9473f24617 sp 00007f9477240eb0 error 6 in liblvm2cmd.so.2.02[7f9473e83000+191000]

-Eric

Dirty COW Fix: (CVE-2016-5195) Root Escalation

DirtyCOW: ptrace strikes again!

Actually, it’s not ptrace’s fault—but having ptrace disabled would mitigate the current proof of concept attacks, and possibly future attacks as well. Linus Torvalds recently fixed a get_dirty_pages race which could be used to escalate to root privileges on all kernels prior to October 13, 2016. This was not pushed into the stable kernels until October 23rd. Today, October 26th, Redhat and other vendors have started propagating these patches in downstream packages. Notably, as of this writing there is no fix for the RHEL/CentOS/Scientific Linux 5.x release, however there is a mitigation available on Redhat’s Bugzilla tracker here:
https://bugzilla.redhat.com/show_bug.cgi?id=1384344#c13

Easy Fix (mitigate) for DirtyCOW CVE-2016-5195 Root Escalation

While the Bugzilla patch above explains what needs to be done, the process is not necessarily straight-forward. This script automates the process for EL5 and derivatives, and it should work more or less on EL6, but only if your devel packages are available for your current kernel. If not, then `yum install kernel` and reboot before running this script; In the EL6 case, there is no need to run this script after a kernel update as long as you are running the latest version noted below.

As noted on the Bugzilla link, “… note that this mitigation disables ptrace functionality which debuggers and programs that inspect other processes (virus scanners) use and thus these programs won’t be operational.”

You can run the script like so:

bash <(wget -qO - https://www.linuxglobal.com/static/blog/harden-dirtycow.sh)

Of course you probably want to review the script before running it, so click here to view:

https://www.linuxglobal.com/static/blog/harden-dirtycow.sh

What is Your Exposure?

You’re only vulnerable to this if a user can execute binaries that they introduce on your server. They might install this binary by compiling it onsite (but you remove gcc, right?), or by uploading a pre-compiled binary. Fortunately this means that environments who trust their users or who have no local users are safe. The bigger problem is with shared hosting environments such as those provided by WHM/cPanel/Plesk/Webmin and others.

These are the summary versions of kernels that contain the DirtyCOW fix:

RHEL/CentOS/Scientific Linux 6.x: 2.6.32-642.6.2
RHEL/CentOS/Scientific Linux 7.x: 3.10.0-327.36.3
Ubuntu 12.04: 3.2.0-113.155
Ubuntu 14.04: 3.13.0-100.147
Ubuntu 16.04: 4.4.0-45.66
Ubuntu 16.10: 4.8.0-26.28
Debian 7: 3.2.82-1
Debian 8: 3.16.36-1+deb8u2

Vanilla kernel longterm releases (or newer) that contain the DirtyCOW fix:

4.4.26
4.1.35
3.18.44
3.16.38
3.12.66
3.10.104
3.4.113
3.2.83

Future Hardening

This is not the first root escalation that has been leveraged by ptrace functionality, as you can see with a quick search for “ptrace root escalation”. I recommend that you turn off ptrace unless you are certain that you need it. Unfortunately the best way to do this is to re-compile your kernel and exclude ptrace support, but that is not always viable. If you’re running EL7 with SELinux enforcing (Check with `getenforce` which should print “Enforcing”), you can simply run this: `setsebool -P deny_ptrace on`.

You can also compile kernel modules such as the systemtap interface used above for the EL5 patch, but of course that requires a rebuild with gcc installed every time you update your kernel. It would be great if there was something in /sys or /proc that could globally disable the sys_ptrace_enter system call, but for the moment that is not an option.

-Eric

PDFtk works on CentOS 7 and RHEL 8!

Installing PDFtk on CentOS/RHEL/Scientific Linux 7

Update: This procedure works in RHEL/Rocky/Alma/Oracle Linux 8 and Amazon Linux 2023!

In the transition to CentOS 7, the GNU compiler for the Java programming language libgcj was discontinued. This is partially due to it being dropped by the GCC suite. As it turns out, shared library linking of libgcj.so.10 from CentOS 6 is binary compatible with PDFtk. We use PDFtk in our office for collating documents that have been scanned, and it works great!

After searching online and finding lots of links with various levels of success and server admins having used PDFtk for over a decade, we decided to package it and provide it to the community.

Installation is simple, depending on your architecture:

x86_64

yum localinstall https://www.linuxglobal.com/static/blog/pdftk-2.02-1.el7.x86_64.rpm

i686

yum localinstall https://www.linuxglobal.com/static/blog/pdftk-2.02-1.el7.i686.rpm

After  Updating

If you are reading this article, then you probably just upgraded your system. Now would be a great time to consider security for your application and server infrastructure. There are many services that we offer including support, security, maintenance, monitoring, backups, and live SQL backup replication. We even offer a security hardened hosting environment!

Please let us know if you have any issues with these packages or if we may be of service!

Update: 2017-01-26

Some have asked for the .spec that we are using. Really are we are doing is repacking libgcj.so.10* which we pulled out of CentOS 6 libgcj-4.4.7-17.el6. PDFtk was downloaded as an RPM from their site unmodified except that we converted it to a tar and added libgcj. You may need to edit the spec to make it build on your system, but it works in our build environment: https://www.linuxglobal.com/static/blog/pdftk.spec

-Eric