HTML Splitter & Boilerplate Refresher

Disclaimer

Please do not email me about this project without reading the disclaimer above.

1. splitting up giant .html documents into manageable pieces.
2. replacing your standard headers and footers on a set of HTML (Hypertext Markup Language) files, with macro-style replacement to customise them.
3. organize your scattered thoughts into categories.
4. Replace computer generated tables automatically sandwiched inside hand-generated HTML.

HTML Documents, such as this one, tend to grow fat and unwieldy. At some point they need to be split into smaller documents. Have a look at methods.html. It has been split up, with a menu put on the front to link to the various pieces. Breaking up even one big document manually is at least a day’s work, by the time you get all the minor HTML adjustments done and proofread.

I would like to automate, or at least semi-automate the process. Here is how it would work. You take your monster document and insert magic tags in it showing how you want it split up. Then you run a utility that creates the new files.

There are features to deal with boilerplate, such as standard headers and footers and for customising those standard headers and footers.

The utility also generates you a menu that can be used to jump to all the different documents created.

SPLIT Tags

• TO=file : The name of the file to be split off, without the .html extension. You may have several chunks directed to the same file. They will just be tacked one after the other. You can also specify a list of files separated by commas. Note that the quotes are mandatory. That way there is no confusion between the HERE option and a file called HERE.
  TO=HERE : means leave the chunk where it is, in the mother document.
  TO=ALL : means insert the fragment in all the files created, e.g. a header or footer or other standard boilerplate.
  TO=NONE : means the chunk is discarded.
• FROM=file : The name of the file where to find the fragment to include, without the .html extension. You always include entire files. You may also specify a list of files separated my commas. This would typically be some boilerplate header or footer.
  FROM=HERE : means you can find the chunk immediately following. The end of it will be marked by the next SPLIT tag or EOF (End Of File) whichever comes first. This is the default. This chunk will be directed wherever the TO parameter directs it.
  FROM=MENU : means you generate a menu with links to all the child files, using their titles and icons and insert this where specified by TO.
• TITLE : description to use in a generated menu for the TO file, or metatags, or H1 headers.
• ICON : optional icon descriptor to be used the represent the fragment in the master menu. The height, width and name can all be generated from the icon file name or the icon file contents directly. See the Image Amanuensis Project.

However, when you propagate boilerplate you want it slightly customised. You can arrange to get it customised by embedding magic comment tags that will be expanded when you run the REFRESH utility.

Customising your boilerplate: REFRESH Tags

• <!--! THIS -->

  The name of the document often appears in the document itself. You might want it left the same to continue to refer to the mother document or you might want it changed to refer instead to the name of the child document. You mark such places with a <!--! THIS -->. The text <!--! THIS --> will be replaced by swim in the child fragment swim.html. This lets you generate strings like http://mindprod.com/swim.html or swim.cnt. The utility looks at the current directory to determine if extra levels of directory need to be included in the generated file name.

• <!--! ICON -->

  This works just like THIS so you can make references to the fragment’s icon in the standard headers or footers, or even the fragment body.

• <!--! MOTHER -->

  This works just like THIS so you can make references to the fragment’s mother file in the standard headers or footers, or even the fragment body. The file from which this was split off, or the file just above it in the document hierarchy.

Regenerating Boilerplate

<!--! BEGIN REFRESH FROM="fred" -->...
<!--! END REFRESH FROM="fred" -->

You thus have the option of either boilerplate that is regeneratable with embedded REFRESH tags and that which should not be regenerated later, without such tags, presumably because you intend to hand customise it and don’t want your customisations overwritten.

You might invent a shortcut

<!--! INCLUDE FROM="header2" -->

that acts like a BEGIN REFRESH / END REFRESH pair. You would use it to get the boilerplate included in the first place, when you are not using SPLIT. It would be deleted after processing.

During SPLIT processing, all tags except SPLIT are ignored. They are just treated as ordinary text. They can be expanded/refreshed later with a REFRESH run. The SPLIT process leaves behind a line like this in each file which is useful in expansion of tags:

You can manually edit the INFO tag generated by SPLIT or insert it manually. This enables you to use the REFRESH utility without ever using the SPLIT utility.

THIS, MOTHER and ICON not in boiler plate text, are expanded once and cannot be refreshed since there is no tag left behind. If you want them refreshable, they must appear in boilerplate text, enclosed in REFRESH tags. In a pinch you could create a boilerplate file to include that had almost nothing in it but a THIS or MOTHER tag. It will be expanded in the context of the file where it finally appears.

Your head is probably hurting by now. The basic problem is how to run REFRESH multiple times. You want to get rid of what was included/expanded earlier, before you re-expand the tags. You need a way of identifying where earlier expansion material started and ended. You also must leave embedded notes around about how to re-expand. For your first cut, only worry about running REFRESH once and leave no trace of the tags. Once you have that working you may feel ready to tackle the problem of running REFRESH multiple times to freshen your boilerplate expansions.

Dealing With Broken Links

• The process generates a global script for Funduc Search and Replace. You run this script and it fixes all the <a href=FILE#NAME>…</a> references, not only in the files you have just split, but in other related files that used to refer to spots in your old giant mother document. If you left out this step, the dangling links would all just take you to the top of the new mother document.
• Better still it would produce an old:new file for the HTML Disturbed Link Patcher utility.
• On your first cut, you might leave this out and just search and replace all references to the mother document or internal references manually. The catch is, you are bound to make errors if you do it manually, unless there are just a handful.

Manual Touchup

HTML Validator

AutoGlue

I wrote a pair of split/glue utilities like this to help me manage Pascal source code on the PDP-11 many years ago. Except for fixing NAME links, this project is actually easier to code that to explain.

Implementation

REFRESH can be handled in a single pass. When you find an INFO tag you remember the information for use in later tag expansion. You discard the INFO tag itself. When you find an INCLUDE tag, you copy in data from the FROM file and recursively process it for embedded tags, including more INCLUDES. The only file you change is the one you are processing. You don’t refresh or expand the included boilerplate files themselves. You discard the INCLUDE tag itself. If you see a BEGIN REFRESH, you discard up to and including the corresponding END REFRESH. You may discard some nested BEGIN END pairs in the process! Then you treat like an INCLUDE. Don’t automatically generate new BEGIN END REFRESH pairs. If they are wanted, they will be inside the included text. If you see a THIS, ICON or MOTHER, replace it from the information in the most recent INFO tag. Discard the tag itself. This way refreshable boilerplate can be composed of refreshable boilerplate. When your refresh your documents, the latest and greatest will be recursively refreshed.

As a first cut, you might use Funduc Search and Replace as your scanning and replacing engine. The key is the *[] regular expression marker that will match anything, e.g. the old boilerplate sandwiched between two markers and the binary replace mode that lets you insert arbitrary multiline text.

Special Purpose Solutions

Outstanding Questions

Implementations

I have also done a some specific solutions to the splitter problem — to split up large glossaries, making each <dt> its own file.

standard footer

	This page is posted on the web at:	http://mindprod.com/project/htmlsplitter.html
	Optional Replicator mirror of mindprod.com on local hard disk J:	J:\mindprod\project\htmlsplitter.html
	Please read the feedback from other visitors, or send your own feedback about the site. Contact Roedy. Please feel free to link to this page without explicit permission.
	Canadian Mind Products IP:[65.110.21.43] Your face IP:[18.188.110.150]
Feedback	You are visitor number