Episode 4: Building a Dictionary App : Using the GAEJ URL Fetch Service

Welcome to Episode 4 of GAEJ Experiments. In this episode, we will learn how your GAEJ Application can invoke external URLs and get data back from them. In the process we will build a fully capable Dictionary Application. While it may not be a complete application, it will demonstrate just about enough how you can integrate external services into your application.

This episode is particularly important given the fact that there are hundreds of public APIs now available out there on the web, which your application can integrate. I strongly recommend visiting The Programmable Web that catalogs hundreds of Web APIs that are available today and which you can integrate into your application.

Dictionary Application in Action

In order to maintain a consistent style across the episodes so far, let us first watch the application in action to better understand what we will be building over here. Follow these easy steps:

  1. Navigate to http://gaejexperiments.appspot.com. This will result in a page as shown below:s1
  2. Click on Dictionary Service link. This will lead you to a page shown below:s2
  3. Enter any word that you wish to lookup in the dictionary and click on Lookup Dictionary. For e.g. engine. This will display the meaning of the word as shown below:s3

Behind the Scenes

Let us understand how the above application is working. The key points to note in the flow are given below:

  1. The dictionary page shown in step 2 above is a simple HTML form page. The only user input needed in this step is the word that you wish to look up in the dictionary.
  2. On click of the Lookup Dictionary button, a Servlet is invoked in our GAEJ Application. The Servlet will utilize the GAEJ URL Fetch Service to invoke an external Dictionary Service.
  3. The Dictionary Service is hosted at a site called http://services.aonaware.com/DictService/DictService.asmx . One of the service operations is Define and you can try it out here. It supports a REST style interface (GET/POST) for invoking it, so that makes our task simple. Give it a try here to better understand what we are integrating or simply punch in the following in your browser:
    http://services.aonaware.com/DictService/DictService.asmx/Define?word=trial
    
  4. This service takes in a word and returns us a XML Response which contains the meaning of the word in several dictionaries. We parse out the dictionary meaning and return that back to the client (HTML form).
  5. The HTML form that renders the meaning as shown in the screen above.

URL Fetch Service

The core piece that we will be utilizing in our application here is an ability to invoke external URLs. This is provided by the URL Fetch service of the Google App Engine. It allows us to invoke any external URL and receive the data from the URL. At a high level, usage of the URL Fetch service is pretty simple and a boiler plate code is shown below: [Most details have been omitted]

 

try {
URL url = new URL(PUT_EXTERNAL_URL_HERE);
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));

String line;
StringBuffer responseData = new StringBuffer();

//Read the entire response in this loop
while ((line = reader.readLine()) != null) {
responseData.append(line);
}
reader.close();

//Process the data from the responseData StringBuffer
//Might involve Parsing the XML or JSON Response format, etc.

}
catch (Exception e) {
//...
}

 

This is pretty much standard Java code and it involves opening an inputstream to the external URL and collecting the response in a local variable. You can then decide what to do with the response data received. For e.g. most external services provide a XML or JSON formatted response, which you may have to parse out and then proceed with your application logic.

In our Dictionary Application here, we will be receiving a XML response, which we will simply parse out and return back to the HTML client.

Developing our Application

The first thing to do is to create a New Google Web Application Project. Follow these steps:

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin.
2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine GAEJExperiments. I suggest you go with the same name so that things are consistent with the rest of the article, but I leave that to you. In case you are following the series, you could simply use the same project and skip all these steps altogether. You can go straight to the Servlet Development section.
3. Click on Finish

This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet.

The Front end HTML form [dictionary.html]

Create a new HTML file in the war directory of your application. In that directory, you will find an index.html that is generated for you. So you can either use index.html or generate a new HTML file as I have done. Name this file as dictionary.html and code it as shown below:

The first thing that we will do is to code out a simple HTML form that takes a single parameter as an input. This parameter is the word that we need to lookup in the dictionary. We will use a bit of AJAX here to dynamically call our servlet and display the response that we get.

 

<html>
<head>
<script type="text/javascript">
var xmlhttp;
function lookupDictionary(word)
{
xmlhttp=null;
if (window.XMLHttpRequest)
{// code for IE7, Firefox, Opera, etc.
xmlhttp=new XMLHttpRequest();
}
else if (window.ActiveXObject)
{// code for IE6, IE5
xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
}
if (xmlhttp!=null)
{
xmlhttp.onreadystatechange=state_Change;
var url = "/gaejdictionaryservice?word="+word;
xmlhttp.open("GET",url,true);
xmlhttp.send(null);
}
else
{
alert("Your browser does not support XMLHTTP.");
}
}

function state_Change()
{
if (xmlhttp.readyState==4)
{// 4 = "loaded"
if (xmlhttp.status==200)
{
// 200 = "OK"
document.getElementById('DictionaryServiceResponse').innerHTML=xmlhttp.responseText;
}
else
{
alert("Problem looking up Dictionary Service :" + xmlhttp.statusText);
}
}
}
</script>
</head>

<body>
<h2>Dictionary Lookup </h2>
<hr/>
<h3>(Powered by Aonaware <a href="http://services.aonaware.com/DictService/">Dictionary Service</a>)</h3>
<hr/>
<p>
<b>Lookup meaning of word:</b><input type="text" id="word"></input>
</p>
<p><b>Meaning :</b>
<br /><span id="DictionaryServiceResponse"></span>
</p>

<button onclick="lookupDictionary(word.value)">Lookup Dictionary</button>

</body>

 

Here are some key points from the code above. For most web programmers it is pretty much standard stuff:

  1. We have a single input field HTML element with an id named word. There is a button with a label Lookup Dictionary, on click on which, we invoke a Javascript function called lookupDictionary, passing in the word value.
  2. The lookupDictionary method builds up the standard XMLHttpRequest object that we shall use to send our request across. The request url is /gaejdictionaryservice with a single parameter named word, which contains the value of the word that we need to lookup in the dictionary.
  3. Note that /gaejdictionaryservice is our servlet endpoint that we will be seeing in the next section. This servlet will take it the word parameter, and use the URL Fetch Service to invoke the external Dictionary API and return us back the response.
  4. The response when returned is inserted into the span element named DictionaryServiceResponse in the above HTML form to display the response received.

Next we shall code the heart of our application a servlet named GAEJDictionaryService.

Coding the GAEJDictionaryService Servlet [GAEJDictionaryService.java]

Create a new Servlet in your Project as shown below. I have created the GAEJDictionaryService.java in the package com.gaejexperiments.networking. You can choose a package of your choice.  The code is straightforward and is listed below:

 

package com.gaejexperiments.networking;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.StringReader;
import java.net.URL;
import javax.servlet.ServletException;
import javax.servlet.http.*;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;

@SuppressWarnings("serial")
public class GAEJDictionaryService extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {

String strCallResult = "";
resp.setContentType("text/plain");
try {
//Extract out the word that needs to be looked up in the Dictionary Service
String strWord = req.getParameter("word");

//Do validations here. Only basic ones i.e. cannot be null/empty
if (strWord == null) throw new Exception("Word field cannot be empty.");

//Trim the stuff
strWord = strWord.trim();
if (strWord.length() == 0) throw new Exception("Word field cannot be empty.");

String strDictionaryServiceCall = "http://services.aonaware.com/DictService/DictService.asmx/Define?word=";
strDictionaryServiceCall += strWord;
URL url = new URL(strDictionaryServiceCall);
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuffer response = new StringBuffer();
String line;

while ((line = reader.readLine()) != null) {
response.append(line);
}
reader.close();

strCallResult = response.toString();

DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = builderFactory.newDocumentBuilder();
Document doc = builder.parse(new InputSource(new StringReader(strCallResult.toString())));

XPathFactory factory = XPathFactory.newInstance();
XPath xpath = factory.newXPath();
XPathExpression expr = xpath.compile("//Definition[Dictionary[Id='wn']]/WordDefinition/text()");

Object result = expr.evaluate(doc, XPathConstants.NODESET);
NodeList nodes = (NodeList) result;
for (int i = 0; i < nodes.getLength(); i++) {
strCallResult = nodes.item(i).getNodeValue();
}

resp.getWriter().println(strCallResult);

}
catch (Exception ex) {
strCallResult = "Fail: " + ex.getMessage();
resp.getWriter().println(strCallResult);
}
}

@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
doGet(req, resp);
}

}

 

Let us go over the key points in the code above:

  1. I have provided both GET and POST handlers in the servlet and the POST handler simply invokes the GET handler here.
  2. Then we parse the request parameters for the word parameter that we need to look up in the dictionary (word) and we do some basic validation to make sure that it is not empty.
  3. The Dictionary Service that we plan to use is available at the following URL:
    http://services.aonaware.com/DictService/DictService.asmx/Define?word=%5BYOUR_WORD_HERE%5D
    
  4. In the above URL, you need to provide the word. So what we do in the code is to simply append the word request parameter that was passed.
  5. Next, we use the URL Fetch Service as discussed earlier to collect the entire response.
  6. The response returned to use is in XML format and the service returns us the meaning of the word based on 6 dictionaries. We will be using just one of those dictionaries WordNet 2.0 which is the 3rd definition in the XML. I suggest that you punch in the following url to understand what we will be parsing out here. I have used the word ‘engine’ here.
    http://services.aonaware.com/DictService/DictService.asmx/Define?word=engine
  7. Finally we use XPath. I intentionally used this to demonstrate how easy it is to use XPath to extract out the element text that we are interested in. You are free to choose an alternative way of extracting out the text value. You could use standard SAX/DOM parsing too if you wish. Whatever you are comfortable with will suffice for the example.
  8. We first build the Document object by using the standard DocumentBuilderFactory and DocumentBuilder classes.
  9. Then on the Document object doc, we evaluate the XPath expression. The XPath expression is //Definition[Dictionary[Id=’wn’]]/WordDefinition/text().
  10. The XPath expression given above can be read as following. First consider the //Definition[DictionaryId =’wn’]] which means that find all definitions anywhere in the document which have a child element named DictionaryId whose value is ‘wn’. This is the Definition that we are interested in extracting.
  11. Once that is found, comes the rest of the XPath expression, which says that for that Definition element found, get a child element named WordDefinition and extract out its text() value. This is returned as a collection of Text nodes.
  12. Finally, we iterate through the Text Nodes and get the value which we then send back as a response.
  13. I suggest that if you are still having a problem following the code, try out the URL as mentioned in step 6, study the XML and then the XPath expression will become clear. The rest of the code is standard XML/XPath code from the Java SDK.

Servlet Configuration

To complete our Servlet development, we will also need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so.

<servlet>
<servlet-name>GAEJDictionaryService</servlet-name>
<servlet-class>com.gaejexperiments.networking.GAEJDictionaryService</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>GAEJDictionaryService</servlet-name>
<url-pattern>/gaejdictionaryservice</url-pattern>
</servlet-mapping>

Deploying and running your application

To deploy the application, you will need to first create your Application ID. The Application Identifier can be created by logging in at http://appengine.google.com with your Google Account. You will see a list of application identifiers already registered under your account (or none if you are just getting started). To create a new Application, click on the Create Application button and provide the Application Identifier as requested. Please note this name down since you will be using it for deployment.

For e.g. I have registered an application identifier named gaejexperiments.

To deploy the application, follow these steps (they should be familiar to you now):

  1. Click on the Deploy Icon in the Toolbar.
  2. In the Deploy dialog, provide your Email and Password. Do not click on Deploy button yet.
  3. Click on the App Engine Project settings link. This will lead you to a dialog, where you need to enter your Application ID [For e.g. my Application Identifier gaejexperiments]
  4. Click on OK. You will be lead back to the previous screen, where you can click on the Deploy button. This will start deploying your application to the GAEJ cloud. You should see several messages in the Console window as the application is being deployed.
  5. Finally, you should see the message “Deployment completed successfully”.

This means that you application is ready to serve. Depending on whether you used index.html or dictionary.html, you should be able to access your application at the following url:

http://<YourAppId>.appspot.com/dictionary.html

or

http://<YourAppId>.appspot.com/index.html

Type in a word and check if the meaning is being returned.

Moving forward

In this episode, we have seen how to utilize the URL Fetch Service provided by the Google App Engine. This service provides us with a convenient way to access external services that can be integrated into our application. Creating Mashup applications would be straightforward with the URL Fetch Service.

You can integrate other external services, RSS feeds in this fashion. Just give it a try.

Till the next episode, good bye and happy coding!

Read more Episodes on App Engine Services

 

Advertisements

12 thoughts on “Episode 4: Building a Dictionary App : Using the GAEJ URL Fetch Service

  1. hi…i tried this code.
    i am getting an error which is status unknown….
    can u tell me what can be done?

    1. Hi Kamaljeet,

      It is difficult to understand what could be going wrong. My guess based on the issues that you are having is probably a network issue of some sort. However I suggest that the best way is to debug it locally first. Alternately, use logger to track your code and then see in the AppEngine logs what could be the problem.

      Thanks
      Romin

  2. this is wat i get in console window….in both the codes even for memcache.
    connection timed out:retry
    connection timed out:retry
    could not fetch the url

    1. It looks like a network connection problem to me. If you punch the URL directly in the browser or some other curl kind utility — are you able to get the response?

      Thanks
      Romin

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s