Grokking the Google API with VS.NET
(or - Is ScreenScraping Dead?)

Peter A. Bromberg, Ph.D.
Printer - Friendly Version

Peter Bromberg

Google has absolutely become the developer's favorite search engine. Its just that in a couple of years, they've put together a massive cadre of computers, a fast crawler/indexer, and a very intuitive search engine that offers newsgroup content going back some 20 years.

Now the Google people are gonna let us use their service (for non-commercial applications and limited to 1,000 searches per day) without having to screen-scrape their content. They've published a web service, WSDL - compliant API that includes all the stuff in their regular search engine (including Base64 encoded cached pages) as well as their spell checker and other features. You can get the developer "kit" and a free license key here.



The easiest way to get into the Google API is to either include the WSDL-generated proxy class from their Developer Kit into your project, or run WSDL.EXE on their WSDL file externally and include the generated class file in your project. In this manner you can avoid the potential problems involved in generating a proxy as their API does not directly expose the familiar "?WSDL" interface that VS.NET developers have become accustomed to.

Once that's done, it's easy to look at Class View or the Object Browser and see what's available and how to use it. Theirs is the typical cross platform webservice API and it is very object - oriented:

As can be seen from the Class View screen shot above, the API exposes a GoogleSearchService Class, a ResultElement, a DirectoryCategory and a GoogleSearchResult class. To use it, you simply create a GoogleSearchService instance and call the doGoogleSearch method for synchronous operation, passing in the license key, textual query, start page for results, maximum number of results (limited to 10 in the Beta), any filter, a string search restriction and some optional other junk which frankly I haven't even bothered to look at yet.

You get back a GoogleSearch result instance which you can then use to build a DataSet which can be DataBound to your favorite DataGrid or other ASP.NET control. The whole exercise, including a couple of helper methods I used to create the DataSet looks like the following:

string lic = "You key here";
GoogleSearchService s = new GoogleSearchService();

try
{
int start = ( Convert.ToInt32(TextBox2.Text)-1) *10;

GoogleSearchResult r= s.doGoogleSearch(lic, TextBox1.Text ,
start, 10, false, "", false, "", "", "");
// Extract the estimated number of results for the search and display it
int estResults = r.estimatedTotalResultsCount;
Label1.Text = Convert.ToString(estResults) + " Results found";
DataSet ds1 = new DataSet();
DataSet ds =FillGoogleDS(ds1,r);
DataGrid1.DataSource=ds;
DataGrid1.DataBind();
}
catch (Exception ex)
{
Label1.Text =ex.Message;
return;
}

return;

}

private DataSet CreateGoogleDS(DataSet ds )
{
ds.Tables.Add("GoogleSearch");
ds.Tables[0].Columns.Add("Title");
ds.Tables[0].Columns.Add("Snippet");
ds.Tables[0].Columns.Add("Summary");
ds.Tables[0].Columns.Add("URL");
return ds;
}

private DataSet FillGoogleDS(DataSet ds, GoogleSearchResult srchResult)
{
try
{
ds = CreateGoogleDS(ds);
int i =0;
DataRow dr;
string strURL=null;
for (i = 0;i<srchResult.resultElements.Length ;i++)
{
dr = ds.Tables["GoogleSearch"].NewRow();
dr["Title"] = srchResult.resultElements[i].title.ToString();
dr["Snippet"] = srchResult.resultElements[i].snippet.ToString();
dr["Summary"] =srchResult.resultElements[i].summary.ToString();
strURL=srchResult.resultElements[i].URL.ToString();
dr["URL"] = "<a href=" +strURL + ">"+strURL+"</a>";
ds.Tables["GoogleSearch"].Rows.Add(dr);
}

}
catch(Exception e)
{
Label1.Text=e.Message;
return null;
}
return ds;
}

If you wanted to, you could get pretty sophisticated with this. One idea is to create a usercontrol that can simply be plugged into a web page at the location of your choice. You can also use the tool to restrict the search to only a particular site (like their "put Google Search on your site" feature) so that it only searches on your site.

If you'd like to try my live sample of the above code, check this out. To get a full VS.NET Solution in C#, download the file here.

 


Peter Bromberg is a C# MVP, MCP, and .NET consultant who has worked in the banking and financial industry for 20 years. He has architected and developed web - based corporate distributed application solutions since 1995, and focuses exclusively on the .NET Platform. Pete's samples at GotDotNet.com have been downloaded over 41,000 times. You can read Peter's UnBlog Here.  --><--NOTE: Post QUESTIONS on FORUMS!

 
Do you have a question or comment about this article? Have a programming problem you need to solve? Post it at eggheadcafe.com forums and receive immediate email notification of responses.