Automatic File Updates

Disclaimer

Please do not email me about this project without reading the disclaimer above.

There are already some push tools that update files on the client sites. These include Marímba, Java Web Start, DMP, Funduc Patch and Symantec LiveUpdate. However, they are not suitable for two simple applications I have:

1. Keeping people’s expanded copies of cmp*.zip up-to-date. These are giant files containing all the *.html, *.gif, *.jpg and *.mp3 files on my website in downloadable zipped format.
2. Keeping a JavaHelp.jar file up-to-date on a client site.

• Every time even so much as a comma changes in a file, the client must download the whole thing.
• Some tools require running custom software on the server. ISPs (Internet Service Providers) don’t like you doing that, at least not without paying them a fat fee.
• The tools may be too expensive for the hobbyist.
• You can’t FTP (File Transfer Protocol) upload the master copy of the files on the server because people are always busy downloading them.

How are Automatic Update Files Stored on the Server?

How are Automatic Update Files Stored on the Client?

The master copies of the files created by the programmers are also maintained in this conventional way. No one is aware of the *.upd files. They are a transport mechanism only. The conversion to and from the *.upd files is fully automatic.

How Does Automatic Update Work

You need a tool to help you prepare your *.upd files for uploading to the server. It starts with a list of directories and files to process. It detects file and member changes via file and member dates or possibly with the checksums, or even with comparison with your current set of *.upd files. It is probably easiest to use file dates exclusively in determining which *.jar files to create and have a checksum verify routine you run periodically. If you get a failure, you manually redate the affected files with a touch utility to force a correction.

How do you handle a file whose date has changed, but whose contents have not? You could:

• Avoid changing file dates on your master files unless you really change them.
• Propagate the redated file as if it were a truly updated file.
• Fix the original file date back to the previous date.
• Ignore the problem. Leave the file with the old date on the client site. Don’t worry that its date does not exactly match.
• Treat this as a special case of delta compression. See below.
• Invent a special propagation mechanism that changes file dates, without actually transmitting the corresponding files.

What if the client refreshes so infrequently that the necessary Delete entries are no longer present?

• You might not sweat it and just leave the ancient deleted files lying around on the client site.
• You might tell the client to start from scratch.
• It is probably easiest to just keep all Delete entries present forever. Deletes are fairly rare in comparison with Replaces.
• Treat this with multiple state-of-the-union files. See below.

Extending Automatic Update

1. You get some eager beavers who check every ten minutes if there have been updates. The way the scheme works now, they would download the entire rather fat state-of-the union *.upd file just to discover nothing had changed. You get around this by maintaining several state-of-the-union files. You might have a yearly, monthly, weekly, daily and hourly version. The hourly version just has changes made in the last hour. The root file points to all of them. In addition the root file tells you the low and high sequence number each state-of-the-union file covers. The yearly version may be completely up-to-date, or it may not. By looking at the ranges, the client can figure out which of these files it needs to download and process, if any. It may need to process more than one or none. You could have as many of them or as few of them as your wanted, spanning any range of sequence numbers.
2. Getting an install started from scratch is rather inefficient since the client downloads a zillion tiny files. It is also rather inefficient to do a massive update, since a large number of individual files/members would have to be downloaded. Therefore each upd file might live also in one or more lump files, where a number of upd files are consolidated. The client downloader can then decide the most efficient way to get the individual files it needs. The lump files can be retired just like upd files.e They can be updated, to agglutinate groups of upd files in different ways, or to drop retired/replaced upd files. The restriction is, the client must download the entire lump file if it wants even one upd file in it. As a last resort, all upd files are always available individually. Ironically, each lump file is also an upd file, using the same sequence number naming scheme. The client would look at the lump files available and decide which ones have the most stuff they need and the least stuff they don’t. If there is too much unwanted stuff, then it would pay to download files individually. In practice you might have a lump for updates up to the first of this year, one for updates from first of the year to the first of this month, one for the first of the month until yesterday and one for today’s updates. Note that ideally you rebuild all the lumps each day to prune them of deadwood and add any new files to them. A client coming in cold would need to download all four lump files. A client who updated daily would need to download only one. A client who updated hourly would download individually. The server is free to update the lump files at any time even when clients are in the middle of downloads because of the way updates are done by always creating new upd and lump files.
3. You need to do your updates to the server’s copy of the *.upd files in batches. You don’t update the root file until all the upd files in the batch are complete. The client does not start using his system again until all the upd files mentioned in the root file are downloaded and installed. You don’t want the client using his system when only a few of the files of the update have been installed.
4. Some files can’t just simply be plopped on the client site. They need to be installed, e.g. inserted into the registry or specially processed, e.g. to set special attribute bits, reboot to replace a DLL (Dynamic Link Library), etc. the way Java Web Start does. You need a way to specify custom installers. See Installer and the Installer Project.
5. This scheme still redownloads large non-jar files even if so much as a comma in them changes. See the Delta Creator project for how to tackle that problem. The same problem applies to large members where only a tiny part of them has actually changed.
6. Automatically notify clients when there are changes. This could be done by email, or by a tiny UDP (User Datagram Protocol) or TCP/IP (Transmission Control Protocol/Internet Protocol) probe to the running application. This just probes them to consider doing a refresh cycle. It does not actually send them any update data. If there are very frequent updates to the master files, you have to avoid pestering your clients more frequently than they want to be pestered. You also have to consider they may have many indendent applications using this scheme. The probe had better identify the application and how to unregister email probes. You eventually have to give up on notifying clients that never bother to update or respond to probes. The prober is not necessarily the server where the client gets updates.
7. If you have a great many clients, you need a way to clone your server files and have clients use all the mirrored servers TuCows-style, picking one close, functioning, up-to-date and not too busy. Ideally you want mirror site selection automatic. Further, you want propagation to the various mirror sites automatic. You want seriously out of date clients to first use the less-up-to-date servers to avoid overloading the up-to-date ones.
8. Other projects that could be based on such automatic update include Bulk file distributor, HTML Glossary Presenter, On-Line Books, Sanity Checker, Infinite Disk, Prebranded Software rental with auto updates.

Baby Steps Toward Automatic Update Nirvana

1. Download entire applications as a lump, e.g. with an installer, or with a giant zip file.
2. Download just the files that have changed, using Java Web Start.
3. Download just the members of jars and zips that have changed using the Automatic File Updater described here.
4. Use lumping to avoid downloading separate upd files most of the time.
5. Download just the chunks of files or members that have changed using the Delta Creator.
6. Use the bulk file distributor project so that you can efficiently use multiple client-based distributed servers. This lets you distribute to millions of customers using only a small server.
7. Instead of using simple HTTP file transfer protocols, use custom server software to let the client grab all the updates in a single TCP/IP session.
8. Use instantaneous update so that applications can use the up to the second information, even information that becomes available after the app has started. This requires storing data in specially structured files, usually an SQL (Standard Query Language) database. Ironically this type of update is much more evolved than the simpler types described above. See Oracle Distributed Databases and Oracle Replicated Databases.

standard footer

	This page is posted on the web at:	http://mindprod.com/project/autoupdate.html
	Optional Replicator mirror of mindprod.com on local hard disk J:	J:\mindprod\project\autoupdate.html
	Please read the feedback from other visitors, or send your own feedback about the site. Contact Roedy. Please feel free to link to this page without explicit permission.
	Canadian Mind Products IP:[65.110.21.43] Your face IP:[3.21.46.129]
Feedback	You are visitor number