Infinite Disk, integrated file migration/backup

Disclaimer

Please do not email me about this project without reading the disclaimer above.

This scheme also acts as backup. All files can be backed up. To speed backup, before they are sent to the server they can be super compressed and only changes sent via deltas.

The backup server can conserve disk space by noticing that, for example, two customers both have identical copies of MS Word For Windows DLLs (Dynamic Link Libraries) installed. The server only needs to keep one copy of each DLL (Dynamic Link Library). It has to be careful. Customers may have identically named files that are not identical.

This is very old idea. Univac 1106 mainframes and DEC (Digital Equipment Corporation) PDP-10s with far less than a megabyte of RAM (Random Access Memory) used to integrate file migration with backup to tape.

Before you can restore, you need to free up space. This is done by dropping infrequently used files already backed up. Happily, you don’t have to take time out to back them up.

You could also implement this with backup to CD (Compact Disc) ROM (Read Only Memory) burner. To consolidate backups, so you don’t have do shuffle a zillion discs, you may need to periodically re-backup files that have not changed, (which could entail temporarily restoring them.).

You probably would want to use 64-bit checksums for end to end assurance files were backed up and restored correctly. These also act as cookies for almost uniquely identifying files.

The only tricky technical challenge is the hook into the operating system. It only has to intercept file open. You don’t have to intercept directory reads or close. You leave tiny stub files behind in the directory as proxies for the file. I would tackle NT first. The hooks are much more formal with no ways around them. By the time you are ready, Windows 2000 will be out with I hope similar bulletproof hooks. You could then leave Win95 and Win98 on the trash heap of history. I must admit I have not studied the open hooks available, but I have studied the defragger interface. It is quite bulletproof and stable. I would hope the open hooks would be done similarly. In contrast, in Win95/98 there are many ways around the hooks since low level sector I/O is not restricted. You can cannibalise the Filemon vxd.

Backup should be imperceptible, a low priority background process. It might even stop completely when the computer or Internet connection were busy. You might simply monitor throughput and if it is slower than some threshold, temporarily shutdown. Backup would have relatively low overhead on the cpu, disk and Internet connection. It soaks up the upload side of the Internet channel which is not normally very busy. Unfortunately, with ADSL upload is typically not nearly as fast as the download side. However, that works to your advantage when it comes time to restore.

You would do a manual recall on a directory to revive a cold project so that all files are ready to go rather than dribbling in as you open them. You could even do this on a different machine if you knew the password. See how this project evolves into a scheme where you can sit down at any machine in the world and your logical desktop is sitting there ready for you.

For a simplified initial version, you could handle migration but not backup. You keep at most one backup of each file on the server. The ISP (Internet Service Provider) then needs to track files only by cookie.

For the full backup version, the ISP needs to maintain a list of what backups exist on mass backup organised in a directory structure matching the customer’s, so the user can selectively restore individual files by date. Further, the user should be able to say, "Put this directory back the way it was as of March 26 1999".

This scheme is not going to work without a 24-hour ADSL or faster connection. Restore would be too slow. You want backups to be scheduled at any time at the convenience of the ISP. He is typically backing up the least recently used files. If a user wanted to use one of these file, the backup would be automatically abandoned. You thus have very little interaction between the backup and the files the user is working on.

What about a purely local version where the user is responsible for making the backups? You can get about a gig per CD with compression. ZIP drives are too tiny to bother with. Tape drives are too slow to search to bring back files.

I think it wisest to tackle this first as an Internet service. That gets rid of many headaches.

• Incompetent, lazy, busy users are not responsible for backups.
• The user does not need a CD-ROM (Compact Disc — Read Only Memory) burner.
• You don’t need code that runs on all brands of CD ROM burners.
• Offsite backup, rotation, etc is handled by pros with a single brand of industrial strength hardware.
• The user does not need to find the appropriate media to restore. The user can’t lose his backups. Thieves can’t steal the only copy. Fire can’t burn the only copy.

Stavros Macrakis macrakis@alum.mit.edu hopes to find someone to implement this for under . See the Automatic File Update project for hints on implementation details.

standard footer

	This page is posted on the web at:	http://mindprod.com/project/infinitedisk.html
	Optional Replicator mirror of mindprod.com on local hard disk J:	J:\mindprod\project\infinitedisk.html
	Please read the feedback from other visitors, or send your own feedback about the site. Contact Roedy. Please feel free to link to this page without explicit permission.
	Canadian Mind Products IP:[65.110.21.43] Your face IP:[3.138.69.101]
Feedback	You are visitor number