Server Security Update Best Practices

Securing the Server with Update Patches

In any environment it is important to keep systems up to date with security patches that fix vulnerabilities. In large deployments with many use cases, you may have application requirements that depend on certain versions of packages being installed and upgrading those packages could create undesired side effects. There are 3 basic ways to manage updates in this case in order to balance security patching with usability.

Before you start

In all cases it is a good idea to prioritize based on severity. Vendors typically publish how important the vulnerability is and how broad its exposure may be, and by reviewing the security notes for the update you can decide whether or not the vulnerability affects your implementation.

Always test deployment of the update in an environment intended to replicate your application requirements before applying them to production systems. If there are any problems, solving them in the test environment will make it easier to apply to production and minimize downtime. It is a good idea to schedule downtime or a maintenance window to keep you from surprising end-users with server interruption.

Keep a complete backup of your operating system or use snapshots to roll back to an earlier version in case something breaks during an update.

Update daily and follow the latest release

Updating all packages is the easiest thing to do. Unfortunately this can have problems if specific versions of software are needed, so it could break functionality.

Exclude critical packages from being updated and install all updates

This assumes that you know which packages are critical and should not be updated. By excluding those packages from the update process, the rest of the system can remain up to date. However, excluding a package from update could cause package dependencies to break. If this happens, it is likely that no package will install at all because it will not be able to install dependencies, so you will need to watch your update logs to make sure they are successful or alert on failure.

Review release information and only install packages that require security updates

This requires additional administrative overhead, but allows you to precisely update the packages that you need to maintain security while keeping all other packages at their current version. Sometimes an updated package requires a dependency; dependency chains can be quite long if the updated package is from a newer minor release of the operating system (for example, applying a version 7.6 patch to a 7.4 system). In these cases, it is sometimes possible to download the .src.rpm package and rebuild it on the earlier release platform (7.4) so that the newer rebuilt package will install cleanly in an older environment.

 

 

Scan Servers For SSH Password Authentication

Make Sure Password Authentication is Off

Sometimes it is useful to find out what hosts on your network allow Password authentication so that you can turn it off. Here is a simple script to facilitate that. Create a file called “iplist” filled with each IP that you wish to test and then run the script below. Optionally, you can set ‘password’ to something you expect to work and it will tell you if it authenticated, or if it only asked for a password but failed to authenticate:

#!/bin/bash

password=mypassword

while read HOST; do
echo "echo '$password'; echo >&2 '$HOST asked for password'; rm \$0" > /dev/shm/askpass
chmod 755 askpass
export SSH_ASKPASS="/dev/shm/askpass"
export DISPLAY=x
setsid ssh -o PreferredAuthentications=password,keyboard-interactive -o StrictHostKeyChecking=no -o ConnectTimeout=3 -o GSSAPIAuthentication=no $HOST "echo $HOST logged in" < /dev/null
done < iplist

-Eric

Debug spinning PHP script on a WHM/cPanel Server

Getting A Backtrace

Sometimes PHP will spin at 100% CPU and it is difficult to figure out why. The `strace` command is too noisy, and without knowing where in the code there is a problem, you cannot insert your own backtrace. The newer version of WHM has support for multiple PHP versions, so make sure you run this for whatever PHP version the site is using. In our case, this is using php-fpm.

First, install xdebug:

/opt/cpanel/ea-php72/root/usr/bin/pecl install xdebug

After that, follow the instructions here: https://stackoverflow.com/questions/14261821/get-a-stack-trace-of-a-hung-php-script#53056294

Basically you just need to run the following:

gdb --batch --readnever --pid=$pid --command=/tmp/dumpstack.gdbscript

And the content of dumpstack.gdbscript is:

set $xstack = ((long**)&xdebug_globals)[2]
if ($xstack !=0) && ($xstack[0]!=0)
set $pcurrent = (long*)$xstack[0]
while $pcurrent
set $xptr = (long*)$pcurrent[0]
set $xptr_s = (char**)$xptr
set $xptr_i = (int*)$xptr
set $filename = $xptr_s[4]
set $funcname = $xptr_s[1]
set $linenum = $xptr_i[10]
if ($funcname!=0)
printf "%s@%s:%d\\n", $funcname, $filename, $linenum
else
printf "global@%s:%d\\n", $filename, $linenum
end
set $pnext = (long*)$pcurrent[2]
set $pcurrent = $pnext
end
else
printf "no stack"
end

Fix LVM Thin Can’t create snapshot, Failed to suspend vg01/pool0 with queued messages

Fix LVM Thin Snapshot Creation Errors

From time to time you might see errors like the following:

~]# lvcreate -s -n foo-snap data/foo
Can’t create snapshot bar-snap as origin bar is not suspended.
Failed to suspend vg01/pool0 with queued messages.

You will note that foo and bar have nothing to do with each other, but the error message prevents creating additional thin volumes. While the cause is unknown, the fix is easy. Something caused LVM to try to create an LVM that it was unable to complete, so it generates this in its metadata:

message1 {
create = "bar-snap"
}

The Fix

  1. deactivate the thinpool
  2. dump the VG metadata
  3. backup the file
  4. remove the message1 section
  5. restore the metadata.

The Procedure

  • vgcfgbackup -f /tmp/pool0-current vg01
  • cp /tmp/pool0-current /tmp/pool0-current-orig # backup the file before making changes
  • vim /tmp/pool0-current # remove the message1 section in vg01 -> logical_volumes -> pool0
  • vgcfgrestore -f /tmp/pool0-current vg01 –force

Hopefully this works for you, and hopefully whatever causes this gets fixed upstream.