HTTP : Java Glossary

Message Headers From Browser To Server

Some of the acronyms you will encounter in deciphering HTTP headers include: HTTP, MSIE, CLR Fields in the headers let browsers and servers communicate. You set them up with .setHeaderField or more specialised methods. These codes are idiotic. In a sane universe, should just contain the browser name and version. For example:

Beware using  HttpURLConnection.setFollowRedirects ( false); This reportedly causes trouble in recent JDKs (Java Development Kits). When it is set true, it will not automatically follow responses with: <META HTTP-EQUIV=Refresh.

Message Headers From Server To Browser

In the real world, the conversations between browser/client and server are much more complicated as slipshod than you might suppose. Each query often results is a flurry of permanent and temporary redirects back and forth. Each element on an HTML page must be requested independently. Sometimes servers will send back a fail error code, then send the page anyway. Or they will send a 404 with an OK text response code. Sometimes servers refuse HEAD requests, but accept the equivalent GET. Sometimes servers send back https: in response to an http: request. Sometimes servers give you a totally different page from the one you requested and don’t tell you the one you wanted is on longer available. Sometimes servers redirect to localhost, or send back gibberish messages. Sometimes a server won’t send you a page if you have recently previously requested it. They expect you to have cached it. Browsers just do their best to muddle through. When you start emulating browsers with code, you get pretty flaky programs.

Language and Charset

GET vs POST

In a GET, the parameters are embedded in the URL send to the host, with various header fields following the URL. The first parameter starts with a ?. Subsequent ones start with &. = separates the keyword from the value. Here is what a typical message sent to the server looks like:

A POST, is like a GET, with optional embedded parameters in the URL sent to the host. In addition in has a message tacked on the end after a blank line like this:

Sample Code

Code to do a GET

Code to do a POST

Code to do a HEAD

Code to do a Fetch (generic GET)

Code to do a PROBE (check if page present)

Base class for GET, POST and PROBE

Code to READ

Read the response either as bytes (readBytesBlocking) or converted to a String

Under the Hood

1. URL.openConnection just sets up a place to build the HTTP header. It does no communicating with the outside world.
2. HTTPConnection.connect () requests sending the header to the server. This request triggers opening a TCP/IP (Transmission Control Protocol/Internet Protocol) socket connection to the server. This is done by sending a SYN connection request packet. The server sends back a SYN+ACK. Then the client sends an ACK, upon which may be piggybacked some data.
3. If you look at what URL. openConnection gives you with getClass to will see a variety of objects such as:
   • HttpURLConnection
   • HttpsURLConnectionImpl
   • FileURLConnection
   • JarURLConnection

   depending on which protocol you specified e.g. http:, https:, file:, jar:…

4. This triggers sending the GET header composed of all the header fields set up before the .connect call. The GET request header includes a list of the encodings and compression algorithms the browser would like in response. .connect does not return until the HTTP header is safely sent out the wire. .connect can take a long time to return since it waits for the other end to respond, or a timeout.
5. The browser calls  HTTPConnection.getResponseCode to see if request went ok. This blocks until the server responds with an HTTP response header.
6. Then the browser calls  HTTPConnection.getInputStream and reads the text of the message from the server containing the requested web page. Using the standard TCP/IP protocol flow-control features, the server sends data only as fast the browser can read it.
7. The browser then scans the web page for the URLs (Uniform Resource Locators) of embedded images and puts out GET requests for them.
8. Then various images usually come back from the server on the original socket. The browser could elect to request each image on it own socket so they can arrive simultaneously.

Speeding Up HTTP

• gzip (compress) the responses from your server to the client. You might pre-compress the pages if you have no dynamic content even trivial SSI (Server Side Includes). Otherwise, you are putting a CPU (Central Processing Unit) burden on your server. Modern servers have plenty of spare CPU cycles, so even compressing on the fly will not likely be too onerous a burden. For this to work, the browser must send an Accept-Encoding: gzip in the request header. You also need to provide the uncompressed version too for older browsers (about 15% of browsers extant) that don’t announce they accept gzip.

  IE 5 & 6 had a bug involving javascript and gzip encoding. Microsoft failed to change the User-Agent string after the bug fix and thus the server does not have enough information to determine whether or not the browser is defective.

• Compact the responses removing unnecessary white space from your HTML. See Compactor
• Strip out the HTML comments.
• Transfer-Encoding header: chunked. Use this when you don’t know in advance how long a long response will be, e.g. a database query result. You write the data in chunks, each preceded by a length with a 0-length chunk as end-of-message marker.
• Cookie scoping.

Rant

HTTP is an unnecessarily fluffy and asymmetrical protocol. If I were redesigning it:

• I would create official defaults for all the header fields, so much of the time they could be eliminated.
• There is no need for the User-Agent field to be longer than 5 characters.
• I would replace all the separator characters = ? & with a single one 0x00.
• With 0x00 as the separator there would be no need for URL-Encoding
• Headers would always be UTF-8 encoded and bodies would be UTF-8 encoded by default.
• The default for downloading large file would be to use two streams, one for the head and one for the tail.
• By default the bodies of all messages would be compressed.

Learning More

RFC 2049 MIME part 2, non ASCII (American Standard Code for Information Interchange).

RFC 1945 HTTP 1.0 specification.

RFC 2045 MIME Part One: Format of Internet Message Bodies, specifies the various headers used to describe the structure of MIME messages.

RFC 2046 MIME Part Two: Media Types, describes the general structure of the MIME media typing system and defines an initial set of media types.

RFC 2047 MIME Part Three: Message Header Extensions for non-ASCII text

RFC 6838 and RFC 4289 MIME Part Four: Registration Procedures

RFC 2049 MIME Part Five: Conformance Criteria and Examples, Provides some illustrative examples of MIME message formats

RFC 2183 MIME Part Five: Conformance Criteria and HTTP Content-Disposition

RFC 7230 updates the HTTP protocol. Details of what all the header fields do.

RFC 2617 (obsolete) replaced by RFC 7235: for details on how to send username and password in http headers to restrict access

RFC 2183 MIME Part Five: Conformance Criteria and HTTP Content-Disposition

HttpsURLConnectionImpl and FileURLConnection are undocumented.