Encoding Identification
by Roedy Green ©1996-2008 Canadian Mind Products
This essay is about a suggested
student project in
Java programming. This essay gives a rough overview of how it might work. It
does not describe an actual complete program. I have
no source, object,
specifications, file layouts or anything else useful to implementing this
project. Everything I have to say to help you with this project is written below.
I am
not prepared to help you implement it; I have too many other
projects of my own.
I do contract work for a living, which could include writing a program such as
this. However, I don’t do people’s homework
for them. That just robs them of an education.
You have my full permission to implement this project any way you please.
Unfortunately, the encoding scheme used is not usually embedded as a signature
in the document. See encoding
identification for a fuller description of the problem.
Your job is to look at the document and make an educated guess at the encoding
scheme used to encode it. You might provide a list of guesses in descending
order of probability for someone to make the final decision manually.
There are two parts to the project.
- The viewer, which displays either the entire document in a given encoding, or
just selected parts of the document that would render differently in different
likely encodings. This is a simple text viewer with no HTML rendering. It can
strip tags for HTML and XML.
- The guesser.
The guesser can use the following clues:
- Look for the optional content-type meta tag in HTML
and the optional encoding tag in XML.
- Note how various common words from various foreign languages encoded.
- The presence of BOMs (Byte Order
Marks).
- The frequencies of various letters/bytes compared with sample known documents.
This is a measure both of language and encoding.
- Look for symbols used in the context of a currency marker.
- Various ad hoc schemes you come up with to distinguish two similar encodings.
- Rules that help track the source of a document from its name, and knowing that a
given source usually emitted a given encoding or set of encodings over a given
date range.
The guesser can also guess the language(s) used simply by looking for common but
unique words in each language.
You also might want to tackle this as a neural
net problem. Teaching it with thousands of documents with known encoding.
If you have control over the source of the documents, you can sidestep the
problem by embedding the encoding as the first field followed by a line
terminator. Better still, settle on UTF-8 or UTF-16BE as your encoding and be
done with the problem.
File Format Identification
There is a related broader problem, identifying what format a file is. In
general, there is no way to tell what format a file is, or what program can
process it. You must simply remember or guess from the extension. Unfortunately,
many extensions like doc give little clue. This is a
royal mess and one of the side effects that males designed most of computing. I
can’t imagine such a thing would have happened if Martha Stewart had a hand in.
Files would have been automatically neatly labeled with the format, creating
program and encoding.
Files are sometimes labelled with what they are in the first few bytes in
a signature.
| File Signatures |
|---|
| type |
signature |
|---|
| class |
CAFEBABE in hex |
| *.gif |
GIF87a or GIF89a |
| *.jpg |
FFD8 in hex |
| *.png |
89504e470d0a1a0a in hex |
You can collect these signatures and use them to guess what you have.
Unfortunately, there is no central registry of signatures and most files formats
don’t have them. You will have to discover them yourself for the sorts of file
in your universe using a hex viewer.
Please email your feedback for publication, errors, omissions, broken/redirected link reports
