Sparse-file Support for rdiff-backup

Massive LVM snapshots use lots of space on your backup destination. Virtual machine volume images are (often) mostly empty, especially if more disk has been allocated than the VM is currently using. In such a case, it makes sense only to store nonzero blocks of data. 

This is a patch to rdiff-backup 1.2.8 to add sparse file support.
More info is available on the rdiff-backup wiki.

UPDATE [updated Sun Jan 2 19:49:50 PST 2011]
This is an updated (more efficient/faster) patch to support sparse files
I’ve also written a patch that aligns rdiff-blocksizes for files >1GB on “Globals.blocksize” boundaries (currently 1024*128). This works much better for RAID devices than the “square-root” approach for smaller files, as reads are aligned on 128k boundaries instead of 16-byte-aligned boundaries. See the patch for details.

-Eric

BlockFuse to the Rescue: rdiff-backup of LVM Snapshots and Block Devices

Over the years I have used rdiff-backup as an incremental backup solution. It works really well on many platforms, supports files >4GB, ACLs, and much much more.

Unfortunately, rdiff-backup does not support backing up block device content; instead, it will replicate the block device inode’s major/minor numbers on the destination system (doesn’t backup the internal content). If you are backing up all of your root filesystem (/), this is probably what you want. But, what if you’re backing up large virtual machine LVM snapshots?

Not finding a solution on the web, I wrote my own using the Linux FUSE filesystem.

BlockFuse takes two arguments:

	# ./block-fuse
	usage: ./block-fuse /dev/directory /mnt/point

For example:

	# # Take an LVM snapshot:
	# lvm -s /dev/vgBoot/asterisk -n _snap-asterisk -L 1G
	   ...
	# # Mount /dev/mapper as /mnt/block-devices:
	# ./block-fuse /dev/mapper /mnt/block-devices
	# ls -l /mnt/block-devices
	-r-------- 1 root root  10G 2010-12-21 16:07 vgBoot-_snap--asterisk
	-r-------- 1 root root 1.0G 2010-12-21 16:07 vgBoot-_snap--asterisk-cow
	   ...
	# # Perform your backup:
	# rdiff-backup --include '/mnt/block-devices/*_snap*' --exclude '*' \
		/mnt/block-devices \
		/mnt/backup/lvm-snapshots
	# #
	# ls -l /mnt/backup/lvm-snapshots/
	drwx------ 3 root root         4096 2010-12-21 16:19 rdiff-backup-data
	-r-------- 1 root root  10737418240 1969-12-31 16:00 vgBoot-_snap--asterisk
	-r-------- 1 root root   1073741824 1969-12-31 16:00 vgBoot-_snap--asterisk-cow

Thus, rdiff-backup is able to backup block-device content, including LVM snapshots using BlockFuse. BlockFuse is quite simple: it enumerates the content of the mount-source directory, and exports all block devices with non-zero size as a file with 0400 permissions, owned by your fuse user (probably root for this).

Notes:

  • BlockFuse does not support writing, so your data is read-only-safe. In a catastrophic recovery where you cannot restore a snapshot and must recover from rdiff- backup, just use rdiff-backup’s –restore-as-of argument, and ‘dd’ the recovered “file” back onto the original block device.
  • BlockFuse uses the mount-time as the modification time (st_mtime) for the mounted filesystem. This will force rdiff-backup to scan the block devices for changes. Therefore you must unmount and re-mount your BlockFuse filesystem after updating your snapshots. If you do not, rdiff-backup will skip the “files” because their modification timestamp had not changed since the last backup. (It would be easy to write a SIGHUP handler for this, so send me a patch if you do!)

Incidentally, I have this working in production, backing up snapshots as large as 350GB, so this is well tested. Still, this software is TO BE USED AT YOUR OWN RISK! Patches are welcome if you have a novell idea or change to add to BlockFuse.

Wed Dec 22 15:19:06 PST 2010: BlockFuse v0.01 initial release
Tue Dec 21 16:39:41 PST 2010: BlockFuse v0.02 now uses mmap’ed IO!
Tue Jan 14 10:53:54 PST 2014: BlockFuse v0.03 now follows symlinks and supports i386 architectures.

Download BlockFuse v0.03.

2014-01-14: Thank you for your patience waiting for the current version to be uploaded.  If someone would like to maintain BlockFuse and open a public git repo to maintain the package I would greatly appreciate it.

Cheers,

-Eric

Perl script-fu: Downloading UPS Invoices

UPS provides their shipping invoices and tracking history via their website, and you can even download .CSV and .XML files. Unfortunately, they do not have an easy way to automate this process. After scouring the web for something someone else has written, I decided to write my own: 

pull-ups-invoices.pl

Its a terse 100 line program, and your config options are at the top of the script. It depends on WWW::Mechanize, so make sure its installed. Oh, and be forewarned: this script does not validate UPS’s SSL certificate, so I hope you trust your link to UPS 😉

-Eric

A call—and hang-up, from a root-signing CA

Fri Oct 8 13:34:11 PDT 2010

Today I received an interesting call from a well-known Signing CA. One of my SSL certificates was soon to expire, and they called looking for my business. I spoke with them briefly, genuinely interested in their service, and the sales rep hung up on me when I asked a specific question.

You see, certificate authorities (CAs) have “sales spiders” that crawl the web looking for soon-to-expire certificates to renew using a competing CA. There are many out there, and I expressed some of the challenges with Trust on the Internet in my previous post.

When the CA called the first time, I barely missed the call, and immediately called the number back, not knowing who had called. I introduced myself politely and was greeted politely by the the CA sales rep. I could hear the call-center chatter of keyboards and whisper of voices in the background. He explained to me how certificates work, why you would want one from the CA, and I learned something interesting: It is much easier to get a root-signing certificate than I had thought.

There are (depending on who you ask) three “levels” of trust in certificates:

  • Domain Validation (DV)
  • Organization Validation (OV)
  • Extended Validation (EV)

The Padlock: Domain Validation simply verifies that the certificate presented by a server to which you connect. This is the least intensive validation, and also the least expensive. They simply make sure that an email address at that domain (like admin@example.com) can validate that the domain exists, and that the person requesting the SSL certificate actually manages the domain.

The Gold Padlock: Organization Validation is a bit more intense, and you must submit your business entity record (eg, Articles of Incorporation) for verification. The sales rep informed me that some phishing sites have managed to get the “Gold Padlock OV validation” and that one really needs an EV certificate to be trusted on the Internet.

The Green Bar and Padlock: According to the the CA Sales rep, for extended validation, they actually pull a Dunn & Bradstreet report (which costs $45 from D&B) and verify your entities existence, external to the business registration papers that you might submit (depending on the CA) for the OV version of a certificate. This is the certificate the the CA rep wished to sell.

So I asked if the CA issues root-signing certificates, to which he did not understand and began talking about code signing certificates (which are similar, but completely unrelated). When he paused for a moment, I asked again, explaining what I meant: “No”, I said, “I mean—do you offer a root-signing services such that you would offer a signed intermediate ceritificate for large organizations to sign domain names for their organization?”

There was a pause.

He mumbled something to pass time and I heard the clatter of fingers on a keyboard which could only be an uncertain sales rep quickly looking for an answer by posting a question on a sales-rep “IRC” channel for “silently” asking questions of the guys that know the answers when he does not have one himself.

After another moment, he said that intermediate root-signing certificates are available for a $20,000 deposit.

“Great!”, I said, “and what form of validation do you perform to guarantee that the recipient is who they say they are? Is it a more intense validation than the EV validation?” “No, no”, he said: “Intermediate root-signing certificates are only audited at the OV level, perhaps a bit more—but not as intense as the EV”.

So I said: “doesn’t that mean that anyone with $20,000 can get a root-signing certificate and sign what they wish?” He agreed, but said “if they have $20,000 they must be a serious business”. I pushed on: “So, any institution could ‘deposit’ $20,000 for an intermediate signing certificate—that the CA would verify—and that institution could then sign any certificate with any name, effectively enabling a man-in-the-middle attack for any domain?—–
hello?—–hello?—-”

Not even a click. He was gone. I dialed the number back (remember he quickly answered last time) and found some pretty string quartet hold music, but no recording except after 5 minutes, a place to leave your phone number; I did, and I do not expect a call back. Fascinating!

A thought to ponder for the reader: Can EV domains be trusted more than OV domains when an OV-validated intermediate-signed certificate can forge EV certificates? I suspect they can, but I would like to be proven wrong.

-Eric

Circumventing Privacy, the “legal” way.

I strong recommend reading, or at least skimming:

I have known this attack is possible for some time—and have even performed this attack in consentual environments without an intermediate certificate.

These types of attacks are simple to implement in principle, but difficult to execute because root certificates are well controlled by root certificate authorities. This is changing, however, since well-known certificate authorities will provide signed intermediate certificates for a price—or—perhaps a writ from a judge. I will compile CAs providing this service on this blog entry as I find them. For now, this is a start:

With that, I recommend the use of Certificate Patrol to be aware of the problem; Certificate Patrol is also available here.

Update Thu Sep 23 22:53:25 PDT 2010 

I find it interesting that Google is now a CA that my browser trusts:
http://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crt

Click here and view the certificate rendered:
https://sandbox.google.com/checkout/view/buy?o=shoppingcart&shoppingcart=

This is not good, bad, or otherwise. It simply shows the state of trust on the Internet. Google can sign any “common name” into existence and have it trusted by all modern browsers.

-Eric

armel distcc+ccache farm for low-powered devices

The FriendlyArm mini2440 and Nokia N900

Both the FriendlyArm 2440 and Nokia N900 run armel-compatible CPU’s, but neither are very powerful in of themselves. Today I wrote a tutorial on setting up an armel distcc+ccache compile farm under the Qemu VM environment. Give it a swing if you need more compute power than your cute-lil’-device can provide.

-Eric

Digium Multi-link PPP Support: DAHDI T1 Bonding

Getting started

I have had quite the challenge recently using Digium’s multi-port T1 cards for link bonding. The plan is to have multiple links from multiple cards go to multiple locations and provide aggregate bandwidth and fault tolerence. This means I wish to unplug any wire from any bonded link and—except for reduced bandwidth from the missing link—the network should continue to operate as if nothing happened: telephone conversations must continue and open TCP sockets must stay open.

Prior Work

I found this guide in my search for multi-link PPP, however, it was written for the Zaptel device driver, which was renamed to DAHDI due trademark issues back in 2008. This means the Zaptel documentation is at least 2 years old and DAHDI has come a long way since then.

Still, this guide is nearly sufficient to provide redundant links. The challenge one might experience in using this Zaptel guide on modern DAHDI drivers is the lack of detailed documentation. Thus, the motivation for this article.

Sangoma, a competing T1/E1/J1 card manufacturer released a modified version of pppd to manage multi-link PPP under Linux in a more reliable way. From reading the code, it appears that Sangoma modified pppd such that it will exit if it looses its multilink bundle—and uses a wrapper script, pppmon, to restart the daemon upon failure.

This is not ideal, since we would like the pppd daemon to keep the pppX interface up even if the multilink bundle drops in order to keep routes in place, as the Linux kernel will drop all routes through an interface when that interface is removed. As it turns out, this is a much more difficult challenge than it sounds.

The Case of the Missing Route

There are two complete-failure scenarios (that is, multilink-bundle-failure complete) that we would like to seamlessly recover from:

  • Both remote servers stay up, but all multipoint links drop (ie, unplug/replug).
  • One pppd daemon drops (perhaps the endpoint reboots), and the other side stays on.

The first does does not need LCP negotiation, and existing PPP state can continue upoon recovery. Simply using the pppd persist argument is sufficient here after removing Sangoma’s “exit-on-error” logic, however, this presents a challenge for the latter scenario:

If the side that remains available (ie, turned on) keeps its state, then the remote side will attempt to LCP handshake when it reboots (or re-launches pppd). Since the existing side is assuming existing state, it does gets confused by LCP request frames coming down the PPP link, and the link becomes inoperable.

Thinking, “well, why not just re-initialize the link whenever a multilink-bundle fails” I added new_phase(PHASE_INITIALIZE) at the point that the multipoint bundle is lost; this is nearly the same as re-executing pppd, but it keeps the associated pppX interface—and its associated routes—alive and kicking. This worked well when the remote-end reboots complpetely but, then, the first failure scenario of unplug/replug does not recover: The DAHDI pppd plugin attempts to re-initialize the master channel carrying the PPP link and throws “device or resource busy” errors.

The “Solution”

It turns out the easiest “fix” for this is to write a wrapper shell script around pppd with no-persist and no-fork. You can background the scripts and manage them in a hand-crafted way or perhaps a SysV script. I added the config to the end of /etc/rc.local using the “hub server” to assign IP address, allowing all of the clients to dynamically pick up IP addresses. This means any tech can plug a unit in and it will train up with the proper addressing no matter which port the T1 lines are plugged into. I also used the watchdog daemon to reboot the PPP router if it hasn’t gotten a ping response in X seconds, X*2 seconds after having booted.

This is perhaps not ideal since it drops routes when pppX is downed, but the watchdog will reboot the system if access is really lost.

Future Work

It would be great if someone were to patch the mainline pppd to support graceful recovery from the two failure scenarios listed above—without bring the pppX interface down and losing the network routes. Email me if you’re curious and I will point you in the right direction. After a few hours poking at the pppd code, this change may not be trivial since multi-link PPP apears to be a hack into pppd at the moment.

-Eric