I mentioned about wanting to start a business in my previous post. Well it looks like I might have my chance. While it isn’t exactly what I was thinking of in my previous post, I foresee many opportunities to flex my programming muscle in this endeavor. Plus, I will be starting up with a good friend, so there is a good chance our motivation will actually produce some results.

A piece of software I am starting to research/design/create will be an internal application we will use to automatically extract some government provided public data. The current issue is time. With the amount of data and the requirement of using their web interface, it is not worth the time needed to do it by hand. The only other option would be to buy the data from the government, but that’s just an unnecessary cost seeing as there is a free option available.

So I will be writing a program in C# to automatically access a few websites and download the needed data in chunks. I have never delt with .NET’s web access objects, so I started looking at what they have to offer today…. which looks like a lot.

I modified an example from here:

http://msdn.microsoft.com/en-us/library/456dfw4f.aspx

To do a simple test to see how reading a web page works in .NET.

My revised code below:
Take note of the two windows form controls (webBrowser1) and (textbox1).

//due to the finally statement, these variables need to be created outside the try block
WebRequest request = null;
WebResponse response = null;
Stream dataStream = null;
StreamReader reader = null;
try
{
    // Create a request for the URL.
    request = WebRequest.Create("http://page_to_access");

    // If required by the server, set the credentials.
    request.Credentials = CredentialCache.DefaultCredentials;

    // Get the response.
    response = request.GetResponse();

    // Display the status.
    textBox1.Text = ((HttpWebResponse)response).StatusDescription;

    // Get the stream containing content returned by the server.
    dataStream = response.GetResponseStream();

    // Open the stream using a StreamReader for easy access.
    reader = new StreamReader(dataStream);

    // Read the content.
    string responseFromServer = reader.ReadToEnd();

    // Display the content.
    webBrowser1.DocumentText = responseFromServer;
}
catch (Exception error)
{
    Console.Write(error.ToString());
}
finally
{
    // Clean up the streams and the response.
    if (reader != null)
    {
        reader.Close();
    }
    if (response != null)
    {
        response.Close();
    }
}

All it does is just request a webpage (only with get data) and then stream the response to a string. After that it takes the string and inserts it into the standard WebBrowser control.

Next up I will setup/research:
- Searching the results from a web request.
- POSTing data to a page as apposed to using the get string
- Downloading files that are available from a webpage

Should be interesting!

I guess you could call this part three of the “ajax, json, html, css, easy to update non-server-scripted website series!” This time I will get into my design plans for the client application. It has not been created yet, so there will be some guess work here. Writing this will also help me figure out what exactly I am trying to accomplish here.

Now that the website is:
A simple xhtml page that uses javascript to fetch json text files. That means all of the content is now separate from the main structure of the site.

Next up is deciding on how to make the client application. I have decided to use Visual C# or VB.net 2005 Express Edition. It’s completely free, yeah I was surprised when I found that out… Anyways, A few weeks ago I created an application (which I will post about here in the future) with that development environment and enjoyed using it a lot.

The program may have features like:
- add/update/delete the main page links
- add/update/delete content pages
- modify existing content pages
- upload changes by way of FTP
- Possibly keeps track of what needs to be added/deleted and performs the necessary FTP commands when updating the server. For example, deleting a jpg image when none of the pages use it.
- have a easy to use interface

Underlying functionality:
be able to parse json files
be able to write json files
be able to add/update/delete files from a server

The great thing about newer programming languages like C# and VB.net is that there is probably a lot of that functionality already built into the language or easily accessible.

References:
JSON Website
JSON and Visual Studio 2005
Visual C# 2005 Express Edition