Broken Links

• You are here :
• home
• Java Glossary
• Java Applications
• Broken Links
•  

©1996-2012 Roedy Green, Canadian Mind Products

Introduction

• XENU often tells you a link is broken when it really isn’t. BrokenLinks retests broken links and often removes links that are not really broken from your consideration. BrokenLinks typically discovers that only 1% of the links XENU reports as broken actually need to be repaired. 99% of them are just down temporarily or fail XENU’s quick probe, but pass BrokenLink’s more patient test which more accurately simulates what a real browser does.
• XENU gives you a report of the status of the universe at a given instant in time. A link may be broken simply because the website it points to is down for a few hours for maintenance. BrokenLinks maintains a history of when a link was last found good and last found bad. It then removes from your consideration links that may be only temporarily not working.
• There are links that any automated link checker, including BrokenLinks, believes to be broken, but when you try them manually they, for all practical purposes work. They may include pages that require a password or certificate override. Sometimes these involve multiple layer of redirection, problems with one of the intermediate steps. With BrokenLinks, you can provide a list of such URL s, to treat, at least for now, as good, to take these out of consideration. BrokenLinks retests this list itself and prepares a list of them in a form that you can manually retest them too, to make sure they truly are good, (or irrelevant). XENU’s similar feature simply ignores all such links for now and all time. Out of site, out of mind.

Why use Xenu?

• XENU is extremely fast compared with the competition.
• It is free.
• It does the spidering work (link chasing and finding) that BrokenLinks does not yet handle.

How to Use Xenu

First you spider your local copy of your website with Xenu. Read the Xenu documentation on how to do that. You first have to be sure XENU is working properly before BrokenLinks will work. Use XENU directly to find orphans.

Once you are pretty sure you have XENU configured correctly, run it on your local website, with external link checking turned on.

Configuring Brokenlinks

Download and install a free copy of Brokenlinks.

The first time you use BrokenLinks you must configure it by creating a text file with a text editor. It will look something like this:

The properties are all pretty straightforward except for brokenForgivenessDays=7.

1. If you have only a handful of broken links, and you religiously run XENU/BrokenLinks every day, you might set brokenForgivenessDays=2, though I still set it to 6. One advantage of running every day is you stay on top of researching and repairing broken links. You are never faced with large numbers of them to fix all at once. I personally run BrokenLinks twice a day so that I test sites at different times of day, avoiding treating them as dead when they are just temporarily down for backup. Further, that way I rarely have more than a couple of links to research at any one time.
2. If you have only a handful of broken links, and you religiously run XENU/BrokenLinks twice a week, use brokenForgivenessDays=5
3. If you don’t want to think about brokenForgivenessDays, leave this property out, and accept the default: brokenForgivenessDays=7
4. If you have only a handful of broken links, and you religiously run XENU/BrokenLinks every week, use brokenForgivenessDays=8
5. If you have hundreds of broken links, and you run XENU/BrokenLinks only every once in a while, use brokenForgivenessDays=14
6. You can experiment setting it to various values. The smaller the brokenForgivenessDays number, the sooner and the more broken links will be revealed to you. However, you will be pestered with more temporarily broken links. If you are feeling overwhelmed by broken links, increase the value to show you only the deadest links. The minimum value that makes much sense is 1. XENU itself effectively uses 0.

Running Brokenlinks

You will get a report of the critical broken links to research both in text and html form. Embed the html in a web page somewhere. Here is my list of broken links for mindprod.com. The layout is designed so make it easy to research the problems. You can click to get the page where the broken link is, or click to where it was trying to go.

Then research the broken links and fix them. The run XENU again, click Export Page Map to TAB-separated File and run BrokenLinks. Run this cycle at different times of the day, since some websites shutdown part of the day for maintenance. You want to catch them when they are up. Run the cycle after repairing a batch of links to see how you did. After you get the list whittled down to none, run the cycle weekly, twice weekly or daily to stay on top of the broken links. I find running it daily works best since you never get overwhelmed with work, and thus are not tempted to postpone the work.

If you are pressed for time, you an also rerun BrokenLinks without a new XENU run. This will catch most of the problems you would rerunning XENU, but not all.

If you erase the history.bin file, it will automatically start over from scratch collecting history.

It is best to run BrokenLinks at various times of day so that you won’t think a site is down that is just offline for an hour each day for backup. I am a bit compulsive. I run it twice a day.

Presumed Good File

Sample Text Report

Sample HTML Export

Broken Links Sorted by Error Code

There are 5 links that have been broken for at least 6 days yet to be fixed. Last revised: 2010-02-02

Links Presumed Good

Xenu claims the following links are broken, but they have been manually found to be good. They should be manually rechecked from time to time. The problem may be an unknown SSL certificate authority which needs to be OKed manually, (a missing/unknown/uninstalled certificate root authority) or it may be the website sends the data, but with not-found status.

There are 13 links marked as presumed good despite what Xenu says. Last revised: 2010-02-02

SSL (Secure Sockets Layer) certificate authority which needs to be OKed manually, (a missing/unknown/uninstalled certificate root authority) or it may be the website sends the data, but with not-found status.

There are 13 links marked as presumed good despite what Xenu says. Last revised: 2010-02-02

Repairing Broken Links

• The more often you run XENU/BrokenLinks, the better you odds you will catch a website when it is up, and thus have fewer false broken links to deal with.
• If an entire website goes down, procrastinate fixing any links. It will usually come back within 5 days. If the website itself is up, check to see that most of it is working before investing time fixing links. They may just be having temporary server problems.
• Email the author or webmaster telling them that a certain link is not working and ask if the material is still available and where. Often it is a technical problem they are unaware of e.g. a file accidentally deleted or a website down. They fix it and the link will come back to life within a day or two. They are typically embarrassed and they thank me profusely for bringing the problem to their attention.
• Go to the home page of the target website, and use the local search to see if you can find the document.
• Use Google’s site search, e.g.site:mindprod.com to get Google to look only on one particular site.
• Go to the home page and try to find what you want by using the menu system.
• Look in the google caches. The original document may be there. The date on the cache can be a clue too.
• Look in http://web.archive.org, aka the wayback machine to look in the old snapshots of the site.
• If you know the title of a video, Google will almost surely find it posted somewhere else.
• When you first insert a link, and have duplicate sources, record them. They may come in handy later.
• Make sure you label links (perhaps in the comments) with what they are, and perhaps quoting a little content. Having a precise quotation of some content will make it easier to find the document if it moves.
• Scan the XENU export document for any unrecognised links. Chase them back to the original link you put on you web page. These complicated chains occur when a website keeps referring you over and over to a replacement of a replacement before it finally tells you it does not have the document. This is tricky to do, so I do it as a last resort. I hope eventually to get BrokenLinks to automate this for me.

Automatically Repairing Redirects

BrokenLinks can automatically repair permanently redirected URLs. Websites often reorganise, and leave behind tombstones on the old page that describe where the information is now. Your browser will automatically follow these chains to find the new information. You know this has happened when the URL displayed when the page in found does not match the original. It is best to update your web pages with the new link since they browse faster by going direct to the link, and because they will continue to work if the tombstone is deleted.

BrokenLinks has a feature to automatically maintain these changes for you. BrokenLinks automatically exports a redirects.csv CSV file that gives the old URL, the new URL, and the pages where the old URL appears. It is best to manually examine this list to prune any changes you don’t want to apply, e.g. Yahoo’s replacement links that go preposterously on and on and one. Then use replaceURLs to process that file and apply the changes to your local website mirror. Best take a backup before you try it out. If you generate URLs with code, import them from databases, ReplaceURLs will correct your website and its HTML macros embedded in comments, so the your changes will not will be undone the next time your regenerate your HTML. ReplaceURLs can deal with & encoded in the replacing URLs as either & or &, but it expects & to be encoded as & in the website. It also works when one URL has a trailing / and the candidate match does not.

You can use the CSVPatch utility to automatically replace URLs in CSV files as well.

Here is the TakeCommand script I use to run BrokenLinks, automatically discard some of the redirects I won’t apply, let me edit the list of both permanent and temporary links, and also use them to update two CSV files, hassle.csv and air.csv.

I also scan the temporary redirects looking for redirects to pages with names containing words like error or suspended. I then manually check these out. Usually it means the website owner has not paid his ISP (Internet Service Provider) bills and the account has been suspended. Sometimes sites have died, or not paid bills and the owner or ISP redirects them to another living site, sometimes the ISP ’s or someone else’s parking site. He should use a permanent redirect, but uses a temporary one instead. I can catch these by eyeballing the list. The list is mostly just internal housekeeping junk, so I don’t scan it carefully every day. It sometimes contains broken links masquerading as temporary redirects or permanent redirects masquerading as temporary redirects.

ReplaceURLs

rem run replaceURLs to update all the redirected URLS on a website
java.exe replaceurls.jar  E:\redirects.csv

You might want to repair some of the links manually. You want to make sure the new link truly points to the original information, not some parking page. Just prune the ones you want to ignore or handle manually, and feed the remainder to replaceURLs

ReplaceURLs presumes all your URLs are pure lower case. It won’t find them if they are mixed or all upper case, (except for the tail end path part). Some validator programs will complain about URLS not in all lower case. You can condition your website to use all lower case URLs by running TidyURLs.

TidyURLs

TidyURLs will clean up the links on your website, making sure they are lower case (just the host part). They will put quotes around URLs that are missing them. It will replace spaces in URLs with %20. There are many other cleanups and validations. l It is a command line utility that allows the switches -s for subdirectories too, -q for quiet, -v for verbose, -dry for dry run (does not actually change your files, just tells you what it would do if the -dry option were not there. It allows you to specify which files or file trees you want to process. It automatically ignores all files except *.html files. Here is how you typically use it:

Futures

1. Vastly improve the speed of rechecking links by checking 30 of them time simultaneously the way XENU does.
2. Convert to Java Web Start. This will make the program easier to use by novices since it will not require configuration. The Configuration properties file will be replaced by a GUI (Graphic User Interface). The user will not have to manually allocate a directory for the history file.
3. Remove the dependence on XENU. Handle everything it does in BrokenLinks. This will as a side effect make BrokenLinks notice local links that are in the wrong case. Wrong case links work under Windows and XENU, but fail after you upload to a Unix-based webserver.
4. Avoid checking links that recently checked OK to vastly speed up link checking. You could then afford to do it daily or even before every upload. XENU rechecks everything from scratch every time you run it.
5. Tools to insert warnings styles on broken links so they will have an icon next to them warning your visitors of the problem and letting them know you are aware of it.

Acquiring HtmlMacros

Package	Version	Released	Licence	Language	Notes
Brokenlinks	2.5	2012-02-28	free	Java	❘ precis ❘ manual ❘ screenshot ❘ browse source repository for the current version of Brokenlinks. find and track broken links on your website. Back end to Xenu Link Sleuth. Also includes utility to tidy URLs on a website, and updates redirected links. 4.1MB zip for Brokenlinks Java source, compiled class files, jar and documentation to run on your own machine as an application. Runs on any OS that supports Java e.g. W2K/XP/W2003/Vista/W7-32/W7-64/Linux/Ubuntu/Solaris/OSX. First install the most recent Java. To install, extract the zip download with WinZip, (or similar unzip utility) into any directory you please, often J:\ — ticking off the use folder names option. To check out the corresponding source from the Subversion repository, use the TortoiseSVN repo-browser to access brokenlinks source in repository with [Tortoise] Subversion client on wush.net/svn/mindprod/com/mindprod/brokenlinks/. After you have installed the jar, you can run it as an application. Type: java -jar J:\com\mindprod\brokenlinks\brokenlinks.jar parms adjusting as necessary to account for where the jar file is. download ASP PAD XML program description for the current version of Brokenlinks. Brokenlinks is free. $489.00 US donated so far. If the CMP utilities solved your problem, please donate a buck or two, or donate to one of the charities featured in the footer public service ads throughout the website and get a tax receipt. Full source included. You may even include the source code, modified or unmodified in free/commercial open source/proprietary programs that you write and distribute. Non-military use only.

Links

	You can get the freshest copy of this page from:	or possibly from your local J: drive (Java virtual drive/mindprod.com website mirror)
	http://mindprod.com/application/brokenlinks.manual.html	J:\mindprod\application\brokenlinks.manual.html
	Please email your feedback for publication, letters to the editor, errors, omissions, typos, formatting errors, ambiguities, unclear wording, broken/redirected link reports, suggestions to improve this page or comments to Roedy Green : . If you want your message, your name or email kept confidential, not considered for public posting, please explicitly specify that. Unless you state otherwise, I will treat your message as a letter to the editor that I may or may not publish in the feedback section. After that, it will be too late to retract it. If you disagree with something I said, please quote it and cite the web page where you found it, tell me why you think it is wrong, and, if possible, provide some supporting evidence. Threatening to kill me or spouting obscenities has yet to persuade me to change my mind.
	Canadian Mind Products
	mindprod.com IP:[65.110.21.43]
view Blog	Your face IP:[38.107.179.211]
Feedback	You are visitor number 11.