Link Replacer

Disclaimer

Please do not email me about this project without reading the disclaimer above.

The Problem

Imagine you are the webmaster of some large website, e.g. the CBC (Canadian Broadcasting Corporation). A listener points out that there are hundreds of obsolete links on the website to the old url for the home page of some show. They need to be updated. The site is so large it would take an unacceptably long time to do a global search and replace. Further other people are working on all these pages. You might not be able to get immediate access to a particular page. Most of the pages in the website don’t even need to be revised.

Your Mission

Your mission, if you choose to accept it, is so find a way to rapidly update all the links and generate 303 permanent redirects to redirect hits from outside the website to the old pages.

Implementation

This is similar to the way a search engine works. You periodically spider the website, avoiding respidering documents that have not changed since the last spider. You create a giant index. The key is the absolute URL (Uniform Resource Locator) linked to, the value is the list of pages where that link appears.

When you get a request to update, You look up that URL in the index, then search each of those pages in turn and replace the URLs (Uniform Resource Locators) with the new one. If you cannot get access to a page, you put in queue to do later.

After the next spider, you need to automatically repeat all the updates since the last spider, but only on the newly spidered pages.

Wrinkles

• Ideally every URL, relative or absolute, is enclosed in "…". If you are using macros, they might also be enclosed is some other delimiters such as {…}
• You might also have to deal with:
  <a class="strawberry" id="ABC" href=http://abc.com>
  which would require regexes or parsing.
• When a tree of documents has moved, sometimes you redirect each page to the corresponding document. Sometimes you redirect them all to a common consolidation document. In any case you want to generate the minimum number of redirect statements to define the changes. You need a way to tersely specify which way you want it handled.
• Large websites such as newpapers will likely use some authoring system to manage simultaneous access to the documents. Your code is going to have to integrate with that. The way to handle that is to dynamically load a custom class that locks and unlocks a document. The default version would simply use File.canRead

Selling

After you get this working on your own small website, find some broken links on the website of some potential customer. You might use BrokenLinks. Use your program to generate a list of the pages that need to be fixed, and a list of redirects. Send these results, innocently, as if you had discovered them manually. Repeat. Eventually they should ask you how you know all this. If they don’t, try a different potential customer. This way you will not be rebuffed as a salesman and you will have already proved the value of your program.

standard footer

	This page is posted on the web at:	http://mindprod.com/project/linkreplacer.html
	Optional Replicator mirror of mindprod.com on local hard disk J:	J:\mindprod\project\linkreplacer.html
	Please read the feedback from other visitors, or send your own feedback about the site. Contact Roedy. Please feel free to link to this page without explicit permission.
	Canadian Mind Products IP:[65.110.21.43] Your face IP:[3.141.29.202]
Feedback	You are visitor number