BackBurner stream based archiving software

News

April 5, 1999: Rev 0.60 VERY stable version. Cleaned up a whole mess of problems. I was mixing my paradigm betwen sysopen type operations and open type operations, which was causing some very subtle bugs. Also, I was assuming that perls write function was the opposite of its read function... no such luck. Cleaned up these problems and my blocking problems also went away, so defaults are now set back for reasonable efficiency. Perl's greatest asset (you just type what you want and it does it) is also it's greatest liability (when you are doing something you think you want but you shouldnt really want, Perl will just do it anyway instead of complaining about being asked to do nonsense).

April 5, 1999: Rev 0.51 No blocking factors besides 512 work, I am not sure why, I am looking into it. In the mean time, I changed all blocking factors to 512. This hurts performance, but increases reliability.

April 4, 1999: Rev 0.5, Initial beta release.

Download

Download It! (Only 36K bytes, Gotta love those Perl scripts)

More Information

This is a rough early beta version, please let me know any successes or failures you may have with this software, I would like it to be usefull to as many people as possible. You can reach me, Bill Kilgallon, at killbill@one.net

What Is Backburner?

At the highest level, backburner is a collection of PERL scripts that leverage existing parts of the Unix system and allow users to backup and restore verified and compressed images of any mountable file-system, network drive, or even a disk partition. This should allow even unrecognized operating system partitions to be ghosted and restored via your Unix system, and allow recognized file-systems to be backed up and restored in whole or in parts.

These backups can be done with virtually any Unix supported media. Existing wrappers support floppy disks for the poor but brave, and CD-Recordable and CD-ReWritable disks for the rest of us. Also, a wrapper for any mountable media allows the use of Zip disks, NFS mounts, or any other accessible locations (including even FTP transfers). Extensions to support virtually any media are straightforward to create as simple modifications of the existing scripts.

For the experienced Unix programmer, BackBurner is a collection of tools that allow a Unix stream to be captured in stasis to any media, then reconstituted at a later date as a playback of an exact duplicate of the original stream. This capability, tied together with powerful stream oriented Unix utilities such as dd, tar and gzip allows tremendous functionality and flexibility.

Credits

BackBurner was written by Bill Kilgallon, killbill@one.net, in March of 1999. It is my second serious attempt at a PERL program, so be merciful in any critiques. I am sure there is a better way to do just about everything I did. Feel free to submit it to me, and I will likely incorporate it into any future versions.

Credits also go to my longtime friend and technical mentor, Jim Weirich. He is as patient as he is knowledgeable... every programmer should have a Jim at their disposal. Mark Grosser has also been a strong friend and mentor, and taught me discipline and tenacity as an engineer.

Backburners current state

Backburner is currently a beta release. This is not so much due to a lack of stability of the software (it works great on my system), but due to it's young age, lack of easy customization, and lousy user interface. As others give me feedback as to functionality on other systems with other hardware, we will move towards a real release. Also, as people write graphical wrappers around backburners functionality, it will become much easier to use, and may become a full featured archive and restore package. If someone were to put an X-CD-Roast flavor wrapper around backburner, it could very easily become a full featured backup and restore package, easily rivaling many of the available Windows backup packages.

Backburner Validation Tests

Here are Validation tests and their current status. I currently lack the disk space (or the nerve) to perform all the tests. As others verify their functionality, email me and I will update this page. My system is currently a dual booting AMD-K6-2 333 with the VIA chipset, 64 MB PC100 ram, Red Hat 5.2 Linux, and a Philips 3610 CD-Rewriter. I have the latest versions of cdrecord, but any reasonable version should work.

Licensing

BackBurner is released according to the LGPL:

Copyright (C) 1999, Bill Kilgallon

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Library General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public License for more details.

You should have received a copy of the GNU Library General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

Requirements

BackBurner requires PERL at a minimum, probably any version you have will work, I used pretty basic constructs.

Backburner generally works by creating a temporary image of the media to be created on a mounted file-system, then transfers that image to the target media . This means that if you wish to back up to CD-Recordable or CD-ReWritable disks (650 MB), you must have at least 650 MB free somewhere on your file system. This is a reflection of some of the warts left over as a result of inheriting a digital data specification from a digital audio specification. If you have a CD-Recorder or ReWriter, you probably know all about this already, and likely have a 650 MB scratch area somewhere that you use for disk images. Note that if the media can be mounted and written too directly, there is no need for a temporary image. The mountable device type does not use a temporary file.

If you want to write CD-Recordable or CD-Rewritable disks, you will need a suitable utility, such as cdrecorder (distributed as part of XCDRoast), to be able to fixate the image to disk. The current implementation of BackBurner requires cdrecorder, but you could substitute your own utility with very little work.

Getting a CD-Recorder/Rewriter to work with current Linux distributions is a non-trivial task. Set aside about 4 hours to get it done, read up on building new kernels, and gather the necessary information before you start. If there is an interest, I will try to post a summary of instructions here in the future. In reality the process is not that bad, but lack of a single up to date HOWTO makes it look pretty intimidating. The cdrecorder home page has the best information I have found yet on getting it wrorking.

If you have a LOT of time and patience, and not much money (or perhaps common sense), you might choose to use the floppy version of backburner. It will back up a typical 500 MB Linux system partition with about 200 floppies. Yes, this sounds dumb to me as well, but I remember building my own printer cable when I was in college because I had the $5 for the parts, but not $30 for the cable. Now you know what to do with all those AOL floppy disks everybody kept giving you. This also might be the only reasonable approach to getting some ancient laptops functional as well, you could build a functional partition up on your main system that is small enough to fit your laptop drive, and use this utility to transfer the image or file-system. If anyone tries this, let me know if it works, I gave away my last 386 Linux laptop a while back.

Installation

Well, for now just download the perl scripts, make any necessary changes for your device files (mainly in the bbcapture scripts, the location is obvious), and copy the files to an appropriate (/usr/local/bin) directory.

The section of bbcapture that you may need to change is pretty intuitive. Any changes should (hopefully) be pretty obvious. In future (non-beta) versions of the software, this will be driven from a configuration file. The existing sample setup should work for most users, as long as your CD-Recorder or CD-ReWriter is the first SCSI (or SCSI emulation) device in your system. The floppy version (though why you would want to use it is beyond me) should work for anyone who has a 3.5" high density floppy as the first floppy drive on their system.

How does it work?

At it's heart, backburner is simply a collection of utilities that captures and restores Unix data streams to a variety of media. This probably means little to the Unix newbe, but means just about everything to anyone experienced in the Unix paradigm.

If you are a Unix newbe, welcome. The learning curve is pretty steep, but the result is very powerful, stick with it. Here is a crash course in streams to help just a little.

Backing up and restoring a raw disk partition

Use the dd command for converting between disk partitions and streams. For example, to back up the first partition on the first hard drive to a CD-Rewriter, use the following command:

dd if=/dev/hda1 | gzip | bbcapture --tempfile /mnt/bigdisk/tempburn/backup.tmp --device cdrw --attended

To later extract this archive back to the same disk, you would use the command:

bbextract --device cdrw | gunzip | dd of=/dev/hda1

In both cases, the meaning of the arguments is as follows:

/dev/hda1
The device file for the partition to be archived (see the dd man page).
--tempfile /mnt/bigdisk/tempburn/backup.tmp
The temporary file that bbcapture will use to construct the image to be sent to the specified media. Must have space for one media's worth of data. See the bbcapture --help command for the exact sizes of each media.
--device cdrw
The type of device to archive too. Currently supported types include CD-Recordables, CD-ReWritables, Floppies, and Mountable file systems. Other types can be easily added.
--attended
The attended flag indicates if there will be a person available to easily swap media. Mainly only useful for retry of corrupted writes or for the first media written. If --attended is set, the user is prompted before any writes to media are attempted, including the initial write. If the flag is not set, the first write attempt and all retries (in the event of verification failure) are performed without waiting for media change or insertion (whenever possible).

Backing up and restoring a mounted file-system

Use the tar command for converting between mounted file-systems and streams. For example, to back up the current root file-system to a CD-Recorder, use the following command:

tar -clvzf - / | bbcapture --tempfile /mnt/bigdisk/tempburn/backup.tmp --device cdr --attended

To later extract this archive back to the same disk, you would use the command:

bbextract --device cdr | tar -zxvf -

The options are the same as in the previous section. You can use any tar flags you want, the specified ones are a sane example. For Unix newbes, here is a description of what the tar options mean to help just a little.

When to use what

Well, backburner will capture and restore any stream, so it is none of my business what you do with it. If it can be a stream (which just about all of Unix can be), it can be captured and restored.

The dd command makes (with gzip) a compressed bit for bit mirror image of a partition (commonly called ghosting a drive). This is great for complete disaster recovery or mass production of disk images. It would also be the only way to back up partitions of other operating systems that cannot be mounted on your Linux system (precious few systems, in my experience). It is a strong tool, but a blunt instrument. It can only restore the captured image, individual files can not be extracted.

The tar command is probably a better solution for any mountable file system, as it can selectively back up and restore particular files or hierarchies to or from selectable locations. Heck, tar was written for exactly this sort of thing. It has tons of options for managing permissions, locations to read from, locations to write from, archive searching, archive verification, including files, excluding files, etc. If the file system can be mounted, it is probably better to use tar.

The emergency recovery shell script

Emergency recovery shell script

Frequently Asked Questions

Actually, this is the first release, so there have been no frequently asked questions, but here are the questions I would ask if I were you.

Why are you writing a raw file to the disk, instead of using an ISO9660 data format?
Great question... I wanted to create an iso9660 (normal CD-Rom) data format, but the current tool for generating these disk images (mkisofs) builds the file-system and populates the data all in one step. You tell it the file-system hierarchy you want to have included, and it will include it in a freshly created iso9660 format. This means that to use it with streams we would have to create a 650 MB file of our data, then tell mkisofs to create a second file that is a iso9660 disc image that contains our file. This means we now need a 1.3 Gig temporary space just to write a CD-Rom, and this sounds crazy to me. I agonized enough about having to create the 650 MB temporary file, and only caved in and allowd that because I suspect that anyone using a CD-Recorder or CD-ReWriter has already had to set aside 650 MB worth of space in order to get reliable writes.
Could you write to a CD-Recorder or CD-ReWriter without a temporary file?
Sure, but you need an uninterrupted data stream. I am using a K6-2 333 Linux system with PC100 dram, not the fastest system but no dog either. I could not create, compress, and fixate a stream in real time and maintain it uninterrupted and suitable for 2x writing to my drive with a 1MB internal buffer. Porting the existing backburner package to eliminate the intermediate file would be pretty straightforward, feel free to have at it. Send me the resulting port, and I will try to include it with the next release. I did not do it because the need was low, the likelihood of working reliably seemed low, and the potential payoff was low.
Why not use mkisofs (or XCD-Roast wrapper) directly to back up data?
I tried, it failed. It was missing directories, crashing, and otherwise behaving badly. I also have concerns that it would not correctly handle funny permissions, links, device files, hidden files, etc. Further, it does not allow data to be stored in a compressed format, which seems like a waste of good archival media to me. Also, it gives no ability to handle raw partitions for un-mounted or un-recognized file-systems. By choosing to use streams, I get all the benefits of the dd command, the gzip command, the tar command, the crypt command, and all the rest of the power of Unix, all without doing any extra work. tar, gzip and dd may not have a slick graphical user interface, but if you look at the total functionality of the three working together (see the man page of each for options), I can think of few commercial offerings that can rival their combined functionality.
How can I keep from backing up my temporary file during my archive session?
The easiest way is to make sure your temporary file is on a different file system then the one you are backing up. If this is not possible, then the --exclude option to tar will save you if tar is your data source. If the incoming data is from dd, and you have to have the temp file on the same partition the data is coming from, then you may just have to grit your teeth and hope for the best. Hopefully, a scandisk (windows) or fsck (Unix) of the restored disk will clean up the mess. Ghosting a mounted partition is probably asking for trouble anyway (in theory), but in practice, I bet it would work.

Items still on the todo list

If you are looking to contribute to backburner, here is the list of things still to be done. Feel free to submit implementations for these items, or other suggestions to be incorporated into this list.

  1. Get a few brave souls to install it on their system and tell me all the things I missed so I can update this document.
  2. Add a non-interactive logging capability for status and errors.
  3. Implement user friendly wrapper for doing compressed tar archives.
  4. Implement user friendly wrapper for doing compressed partition images.
  5. Add some signal handlers to make program easier to kill if things go wayward.
  6. Develop simplified minimal restore scripts that use sh to restore from a rescue disk. Perhaps make the backup wrapper create a rescue script on a floppy during the backup. It would be nice if this script did not need perl, so it could be used with the Red Hat rescue disc.
  7. Build up the initial installation routines.
  8. Build up man pages.
  9. Allow defaults to be driven from configuration file (.backburnerrc).
  10. Add real time status updates to long duration procedures to give user warm fuzzies that things are working. Use curses for the gee-whiz factor.
  11. Test the various backup and restore scenarios listed in this document, and develop detailed documentation with screenshots demonstrating each step.
  12. Put a fancy shmancy graphical wrapper around backburner (like X-CD-Roast is to cdrecorder) that makes backup and restore user friendly and nice and flashy.