Xenu : Java Glossary

You can also get it to spider just a subtree. It works very rapidly, spidering your site using multiple threads. It now understands Applets and jars and checks links to jar entries. It does not understand anchors. It only checks that the file exists, not the target. It spiders very quickly, but is extremely slow preparing reports since it uses linear algorithms that do not scale well. Other than that, it is pretty good. You can speed it up by chopping out everything you don’t need from the links report. The by-links is much faster than the by-page. It does not just sort, it does a linear sweep for links in each page. Ouch! With a website the size of mine, I have to turn off the by-page report or the program never completes.

Xenu can also produce an orphans report, a list of files nobody links to. These are most often discarded images, or stray files. Just because something is an orphan does not mean it should be discarded. It may be the home page of a ring or a download you posted not linked into your site.

Xenu puts most of its configuration information in X:\Program Files\Xenu\xenu.ini. You may find it more convenient to edit the xenu.ini file than use the menus if you have a lot of repetitive information to enter. It lets you store the broken link data in user files such as xenu.xen which you can save anywhere you please.

For each constellation of checks you will see two sections like this:

Putting something on the exclusion list blocks both checking links on that page and links to that page. Thus they may show up on the orphans report.

You can use File ⇒ Export to TAB-separated File… or File ⇒ Export Page Map to TAB-separated File… for further computer processing, e.g. with the BrokenLinks utility.

You can configure a list of URLs (Uniform Resource Locators) not to check only when you click File ⇒ Check URL.

While spidering, it appears to stop everything to ask you about some site with a certificate, but actually it is continuing to check links in the background.

It has a number of bugs and problems including:

• Beware, Xenu is extremely case-sensitive. Make sure you use lower case drive letters and / instead of \ in your URLs. It won’t complain. It just won’t work. If you have trouble, manually patch the Xenu.ini file to correct your URLs.
• Many false broken links. Reducing the number of threads helps.
• It used to ignore style sheets. This has been fixed.
• It used to ignore Applet class references or jar references. That has been fixed.
• It often gets confused by redirects and fails to tell you the document where the problem originated.
• It is extremely slow if you ask for the full set of reports.
• Sometimes it stalls. Just clicking the Xenu window often prods it back to life.
• It tends to overwhelm you with links that are not really broken, just slow, temporarily down, ones that don’t support HTTP (Hypertext Transfer Protocol) HEAD, or that want a you to lie about what sort of browser you are.
• It rechecks very link from scratch every time. It does not maintain history. If it did, it would not need to recheck links that were known good recently, except local links, which are fast check anyway.
• Be careful to verify the check external links option at the very last moment before you start the spidering.
• Its progress report can be disconcerting. It tells you the percentage of links that it currently knows about that it has already checked. It can be 99% finished then drop back to 90% as it discovers more links to check. The most practical way to determine progress is to record the time it takes to do a complete spidering and then on subsequent runs compare the time taken with the expected time to complete the whole job.
• Every local file:/// exclude directory or file must exist. If it does not, it ignores all subsequent excludes. This is a bug I have reported.
• It matters if you specify http: or https: on your String statements. Sites are constantly converting from http: to https: and you have to keep your xenu.ini up to date.
• It does not matter if you use / or \, but you must be consistent. Trailing / on directory names are also optional.

Here is what a Xenu configuration file typically look like.

I find it sometimes easier to edit the xenu.ini file with a text editor than to use Xenu’s GUI (Graphic User Interface). It certainly easier to proofread. I tidy xenu.ini with INI before and after each edit.

When you specify the url you want to check, e.g. file:///E:/mindprod/index.html it must exactly match the [file:///E:/mindprod/index.html-Exclude] in the *.ini file including whether you use / or \.

You can find the changelog in Program Files\xenu\whatsnew.txt.