Blog-Shmog.com explodes!!
An error in Blog-Shmog.com's auto-backup script sends its database down in flames...

One of the "gotcha's" that comes with any database-driven website such as Drupal (or wordpress or moveabletype) is that you essentially have TWO content sources to backup: your database and your webroot. Lose either and be prepared to spend lots of late nights doing emergency repairs!

Blog-Shmog.com explodes!!

5 weeks after upgrading to Drupal 6.x, I stumbled upon (got shitkicked by) a small but disastrous mistake. Back when I first started the upgrade process, I duplicated my database and set up a development environment (so I could perform the upgrade without disrupting anything). Once the update was complete and debugged, I copied everything back to my production server BUT I completely forgot to update my auto-backup script to reflect the new Drupal 6 database name. So every week, my custom backup script was backing up the old, unchanged (and outdated AND INCOMPATIBLE) drupal 5.x database!! I discovered this fatal error one day when I decided to roll back my database by 24hrs to fix a small development glitch. I unpacked my latest tar file, cleared my database and restored from backup. Oops! Blog-Shmog.com died.

I instantly lost over 5 weeks of content, comments and edits- everything since the Drupal 6.x upgrade. All thanks to a missing "_6" in my backup script. grrrr!!

Yelling, Screaming, Boozing & Restoring

After pouring myself 2, three, four stiff drink(s), I went to work rebuilding... I threw my old Drupal 5.x webroot back online, loaded up my old database and went about upgrading to Drupal 6. again.

Restoring the Drupal core and contributed modules actually went a lot faster than I expected. I guess I learned a few tricks during my first 6.x upgrade that served me well the second time around, but getting the framework back was the easy part. --- And then it hit me- Google probably cached my website!! ...Sure enough, by searching for specific keywords relating to my missing pages, I brought up Google's (only 4 day-old, by the way) cache of Blog-Shmog.com and copy/pasted everything back into Drupal. The process was simple, but reeeaaaaally slow. Each comment had to be re-entered manually. one by one. in the proper order.

I manually re-entered over 25 comments, 3 page nodes, a user profile and a custom contact form, all thanks to Google's cache! Try THAT pre-Google (sorry waybackmachine, no cookie for you.)

Lessons Learned

No0Oo0Ooo!!!

Learn from my misfortune:

Always, always, always backup your shit.
Obviously, this doesn't just apply to your website. But you alrady knew that...

Make sure to backup your webroot AND your database.
There are backup modules floating around the drupal community, but most of them either only backup your database, or have been abandoned since drupal 5.x. Choose your backup method carefully!

Once your site is backed up, CHECK THE BACKUPS!!
If I had just spent the time to go through my sqldumps more carefully, I would have noticed my Drupal 6 tables were missing. dumb-dumb.

Enjoy your backups, but don't trust them.
If for whatever reason you need to roll-back your database, do a manual backup first, just to be safe. Paranoid: probably. Worth it: definitely.

If you DO lose your data, GOOGLE-IT FAST.
Don't wait! If you wait too long to check Google cache, it might re-cache your busted-ass website!! If you need more time to do the repairs, copy/paste all the Google cache you can get your hands on and safeguard it for later.

Restore cached content IN ORDER
After restoring your content, you can manually adjust each comment's (or node's) datestamp, so you can match it to the original post date. However, you can't re-order your posts once they're created. I found this out the hard way, and had to create a custom Views block to show my recent comments block in the right order.

Manual Backup

If you have shell access to your webserver, you can easily create an instant backup of your website from the command line. sign into your webserver and cd to your webroot, then type:

tar cf webroot-backup.tar ./

This will create a compressed tar file named webroot-backup.tar in your webroot that you can then download via FTP or keep elsewhere on your webserver for safekeeping.

You can create a mysql dump of your database by typing:

mysqldump --user=[USERNAME] --host=[SQL-HOST] --password=[PASSWORD] --add-drop-table [DATABASE-NAME] > database.sql

This will create a file named database.sql. (If you don't change directories between the two backup commands, this file will also be located in your webroot!) Remember to swap in your own values into the fields: [USERNAME], [SQL-HOST], [PASSWORD] and [DATABASE-NAME] (You can find them in your Drupal settings.php file, if you need...).

It's a very bad idea to leave the backups in your webroot, so make sure to either download & delete them, or move them elsewhere, outside your webroot, mmmkay?!

Shmackup v1.0

Shmackup.sh v1.0
click to download...

After adding a stupid little underscore and a stupid little 6 to the [DATABASE-NAME] field in my auto-backup script, everything was up and running again, and my site was being backed up as it should. Here's what I use to auto-backup my site... introducing: Shmackup.sh. Shmackup will export your webroot and database and pack them both into a gzipped tar.gz file with a date-stamped filename. No selective backup features are available in v1.0, but v1.1 is currently under development and will support selective backups, allowing you to cut down on file size by skipping cache and session tables.

Configure and upload Shmackup.sh to your server (outside your webroot, dumdumb!), chmod +x it, and schedule a cron job to run it at a your desired backup interval. I usually set it to perform weekly backups for normal use, and up the interval to daily backups when I'm doing development or playing with settings. Just play with the crontab value to adjust the backup interval to suit your needs.

No matter which backup solution you decide to use, always double triple check your backups before ever blindly accepting that your website content is safe. And if you don't (yet) have any backup methods in place for your website, at the very least SSH to your server and run the command line tools (above) or sign into phpmyadmin (or whatever other mySQL front end you use) and export yourself a fresh database dump.

The single most important thing to remember is: if you DO lose your data, don't panic. Head over to Google and immediately re-download your data from Google cache. It won't do much for deleted files from your webroot, but it CAN help you rebuild your ENTIRE database!