The Lowly HTTP HEAD Request

Often overlooked, the lowly HTTP HEAD Request offers useful results, especially where speed and bandwidth conservation are at stake.

The little - used HEAD HTTP verb can become extremely useful when you need to do things like:

1) Check if a Webpage is "up".
2) Check if the content length has changed.
3) Check the last - modified datetime value.

For example, let's say you have an application that updates RSS feeds from various blogs and sites. You don't want to download the full feed just to see if something has changed.

By issuing a HEAD Request, we are asking the server to return only the headers. Here is an example of one of our forum feeds, showing the header collection returned via a HEAD request:



As can be seen above, we have the Content-Length, which can be compared with a database stored value for the url to find out if the "page" has changed. We also have the Last-Modified date, which can be compared with a previously stored value as well.

Based on this information, we can then make a bandwidth - conserving decision in our code whether we want to issue a subsequent GET Request to read the entire feed (or web page, whatever the case may be).

Here is some "just enough" sample code to illustrate how to issue a HEAD Request and capture the keys and values in the WebHeaderCollection that is returned:

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;

namespace HeadRequest
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                WebHeaderCollection headers = 
                    GetContent("http://www.eggheadcafe.com/forumrss.aspx?topicid=2");                
                foreach (string sKey in headers.Keys)
                {
                    Console.WriteLine(sKey + ": " + headers[sKey]);
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }
            Console.WriteLine("Any key to quit...");
            Console.ReadLine();
        }

        private static WebHeaderCollection GetContent(String url)
        {
            WebHeaderCollection headers = null;
            HttpWebResponse response = null;    
            try
            {
                // create the request
                HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
                // instruct the server to return headers only
                request.Method = "HEAD";
                // make the connection
                response = request.GetResponse() as HttpWebResponse;
                // get the headers
              headers = response.Headers;
            }
            catch  
            {
                throw;
            }
             
            finally
            {
                // make sure the response gets closed
                //  this avoids leaking connections
                if (response != null)
                {
                    response.Close();
                }
            }
            return headers;
        }
    }
}

I can see using this for limiting the data transfer before doing a full data transfer.

I have a question

Scenario:  A "regular page" with advertising

Question:  Theretically, will the returned content length value be different for a single page that has dynamically generated content (advertising) that points to variable length URL? 

I will be trying this out on an Ad Rotater Component Page.

Great article!