dparrish.com

There’s been enough actual interest in Rollout for me to move it to a community site. I chose Google Code because I work there, and it’s much faster than Sourceforge.

The new URL for Rollout is: http://code.google.com/p/rollout.

Unfortunately they don’t support git, so I have to deal once again with Subversion.

The new checkout instructions are:
svn checkout http://rollout.googlecode.com/svn/trunk/ rollout

The manual is online there, and an issue tracker, so submit bugs!

Inspired by Alex King’s recent post on backups, I hacked together this short Perl script that will backup MySQL databases to Amazon’s excellent S3 storage system.

You can download it here. You will need to edit a few configuration items at the top of the script, to include your Amazon Web Services IDs and MySQL details. You will also need the Amazon::S3 and DBI perl modules. You can get them from CPAN.

$ wget -Omysql_s3.pl http://dparrish.com/files/mysql_s3.pl
$ vi mysql_s3.pl (enter your details)
$ chmod 755 mysql_s3.pl
$ ./mysql_s3.pl

It’s probably best to add it to cron so it will run daily. Something like this works nicely:

0 3 * * * perl /home/dparrish/mysql_s3.pl

I’d also suggest using S3 Firefox Organizer to manage your S3 account. It’s a Firefox plugin.

I’ve decided that after 8 months of using Rollout on a daily basis, I should publish a “stable” 1.0 release.

You can get the 1.0 stable release at http://dparrish.com/files/rollout-1.0.tar.gz.

Although this is stable and shouldn’t destroy any of your systems, it does still (and will always) require configuration before it will work for you.

As always, the latest version is available by git:

git clone http://git.dparrish.com/rollout

Comments / bug reports / flames are more than welcome.

Tags: rollout

I’ve been mucking around with Comet, and ran into a situation which I couldn’t seem to find a solution on the lazyweb for. The problem was that none of the streamed JavaScript code blocks would be executed until the entire page was loaded.

I tracked it down to the gzip encoding module, which waits for all the data to be output, compresses it and sends it on to the browser.

You can disable gzip encoding for a particular directory in a .htaccess file (with Apache) like this:

RemoveOutputFilter DEFLATE
SetEnv no-gzip

I can’t stand it when people release open source software, expect people to use it, and don’t provide any sort of documentation. I’m not the sort of person who enjoys writing doco, but I do think it’s required.

So here, in it’s entirety: the Rollout Manual
Read the rest of this entry »

Announcing a new Open-Source project: Rollout!

Rollout is a system developed to automate system administration on UNIX servers. It is primarily focused towards Linux, but could be adapted to Solaris, HP-UX, AIX, etc.

It is written purely in Perl, and the configuration is also a Perl source file. Some Perl knowledge is required to edit the configuration, but copy-and-paste may suffice.

System Administration can generally be reduced to a set of steps, which must be completed in order. Some example steps include:

* Copy files, create symlinks
* Edit configuration files
* Start / stop services
* Add / modify users & groups
* Install applications

The idea behind Rollout is to automate all these steps in a configurable way. Rollout is by design idempotent - it can be run many times on a single server and produce the same results.

There is a certain amount of overlap between this and other projects such as puppet and cfengine. They all have similar goals. Rollout is designed to be very easy to extend (as long as you can cope with Perl) and very fast.

You can get rollout from my git repository at http://git.dparrish.com/rollout. If you’re using Cogito, you can run:

cg clone http://git.dparrish.com/rollout

This is the first public release, and as such is not expected to be everything to everybody. However it is in active production use and hasn’t destroyed anything yet. Use it at your own risk. The documentation included contains instructions on setting up a server and configuring clients.

Also, I’m enabling comments on this post (and I feel dirty doing it) so that people can submit bugs/patches/requests/complaints.

If you’re one of the lucky few using dbackup, and want to upgrade from 1.1.0 to 1.1.1 so things work (I really shouldn’t use a minor version number increase to fix such a major bug), you’ll probably want to run the following command in your backup storage root:

rename 's/^([a-z./]+).(d+).(d+).(d+-d+-d+-d+-d+-d+).(meta|data)$/
      sprintf("%s.%03d.%03d.%s.%s", $1, $2, $3, $4, $5)/ex' */*

This should be run as all one line. It requires the rename perl script, which is in the perl package on Debian/Ubuntu at least. If you don’t have it, grab it here.

I’ve released a new version of dbackup today, version 1.1.1. This version fixes a sorting problem, which you would see if you ever get above level 10 in a backup run.

You can download the new version at:

Along with this release, I’m also opening up my git repository, so you can grab the latest source yourself. If you’re using Cogito, which I highly recommend, you can run:

cg clone http://git.dparrish.com/dbackup

For instructions on using dbackup, see my original post.

For the last couple of years - since blogs have become common, I’ve been telling myself and anyone who’d listen that I’ll never have a blog.

The single reason I don’t want a blog is that I hate them. People generally don’t have anything interesting to say, and I generally don’t want to read what they do have to say. There’s enough junk on Slashdot, Fark and Digg to keep me amused for the rest of my life.

So, the reason I have a blog now is so I can have my own useful OpenID for use on other supporting sites. OpenID is cool.

Comments will be forever disabled on my homepage. If you want to comment on anything I say, do so on your own blog and send a TrackBack ping.

dbackup is a disk-based client-server backup system for Linux or other UNIX systems.

It works on the principal that disks are cheaper and more reliable than tapes.

Backups are started by cron probably on a daily basis by the client. The client backs up individual filesystems / directories with tar and sends the result to the server, which stores them in a simple tree-based directory structure.

Restores are trivial, either by using the supplied restore client, or by simply copying the appropriate tar files off the server and uncompressing them.

Features

• Backup very large filesystems as long as you have the disk space
• Works with very large numbers of servers / filesystems
• No state is kept on the client, it’s all on the server
• Backup to multiple backup servers in round-robin
• Configuration can be global or per-server
• Authentication for client restores (optional)
• Supports optional gzip or bzip2 compression
• Supports client-side gpg encryption
• Trivial procedure for restores
• Daily / Full incremental backups

Read the rest of this entry »