I don’t want to complain about Android development too much. It has its quirks, but generally its satisfactory. One area however that was totally opaque to me, and frustrating, is Android resources. I never saw them explained very well in tutorials. The documentation, when read through thoroughly, is very good. It was probably just too much for me at first.

So for everyones enjoyment I present my Android resources crib notes. Writing these down helped me get to grips with that pesky res directory.

Android Resources

Android resources are used to separate out non programatic elements of your application such as images, user-facing strings, colours and application layouts. This helps when writing applications for multiple devices and languages by separating out the presentation of the application.

Resource types

The res directory of an Android project holds all the application resources. Android is very restrictive about where resources can be placed within this directory, even to the point of refusing to build if there are any resources placed in the root res directory.

Android groups certain types of resources together and forces you to place these types in specific directories. This directory layout is very important. A table of the directories and the general types of resources allowed is found at the Providing Resources page of the Android documentation.

A directory doesn’t necessarily hold one single type of resource. The drawable directory for example can contain bitmap files, jpeg files and XML files. However, the structure of the XML files must be such that they describe a type of object that can be drawn to the screen. All resources in the drawable directory must be able to instantiate a Drawable class.

The values directory

One directory that may confuse is the values directory. This is because it is a grab-bag of different types of resources including colours, scalar values, strings and ‘styles’. The only common thing about these types is that they are described via XML files. Documentation on the different types and how they are described in XML is contained at Style Resources, String Resources and More Resource Types.

Accessing Resources

Within the program, resources are accessed via the R class. From there, access follows the directory structure. R.drawable contains all the resources in the drawable directory, R.layout contains all resources in the layout directory. The name of each resource is generally the same as the filename without the file extension. The resource res/drawable/blue_circle.xml would be accessed as R.drawable.blue_circle.

The values directory does not follow the normal scheme. There is no R.values package, instead the many different types of resource that can be placed in the values directory each have their own package. Some examples include R.dimen for dimensions, R.string for string values and R.color for colour values. Also different from the rest of the resources is the naming of the value resources. Instead of being named after the filename these resources are named directly in the XML. This allows many resources, even of different types to be placed in one XML file, the name of which has no effect. There are however strong conventions about the file names. All strings are usually placed in res/values/strings.xml and all colours res/values/colors.xml.

Suppose you had two string tags in your strings.xml with attributes name="app_name" and name="app_description". These two resources would then be accessed as R.string.app_name and R.string.app_description

One thing to keep in mind is that when using R.string.app_name, or anything inside the R package is that they are only references to the real values. R.drawable.blue_circle doesn’t contain a Drawable object but a reference that can be used in different locations within the Android API. R.drawable.blue_circle for example could be used to instantiate a Drawable class using getDrawable.

Accessing Resources from XML

Very often you will need to reference a resource not within code but within XML files. These XML files are generally other resources, but also include the AndroidManifest.xml file. The naming scheme for the XML reference follows the R package. If a resources is accessed as R.type.name in code it is accessed as @type/name within XML.

Resource Namespaces

Resources are placed in Java packages. The canonical way to access a resource would be package.R.type.name. So assuming that the base package for your application is com.example.firstapp resources could be accessed as com.example.firstapp.R.string.app_name, or @com.example.firstapp:string/app_name in XML. The package name is in this case superfluous. One instance where you will often need to use the package name, is when you are accessing resources from the base Android platform. For this you would use android.R.type.name or @android:type/name. I have most commonly used this for accessing Android styles and themes such as @android:style/TextAppearance.Large.

Alternative Resources

Android provides a very restricted workflow for creating applications that work with different screen sizes, resolution densities and languages. This is based on resources, and is again dependant on the naming of directories. While the drawable, values and layout directories could be considered the base resources, you are allowed to create directories that contain resources specific to specific device configurations. The configuration is determined using ‘qualifiers’. These are specific strings, placed after the base directory name, and separated by dashes.

For example, to create drawables that will be used only on high dpi screens a directory called drawable-hdpi should be created and populated. For french translation strings a directory called values-fr should be used. For drawables that should only be used in landscape mode on low resolution displays, a directory called drawable-land-ldpi will be useful. Any number of qualifiers can be used after the base name, but there seem to be some complicated rules on how the resources are then determined. See Providing Resources for details.

Roundup

Android resources are a varied bunch. From simple bitmap files, to XML files with complicated schemas such as layouts. Generally the XML schemas for each type of resource are well documented, but there are lots of them. Just keep in mind that each resource directory generally holds a completely different type of object, and remember to look up the rules for the one that applies. The Providing Resources documentation is usually the best place to start.

The use of resources is as varied as the types. Methods taking resource references are to be found everywhere within the Android API, but a few of them, layout, strings and drawables are core. The use of these is included in every example android Application and tutorial.

I’ve been working on Android projects for the past few weeks. For one of them I wanted some drag-and-drop functionality that is sadly missing from versions of Android prior to 3.0. I had some time on my hands so decided to create a library that mimics the Honeycomb drag-and-drop API.

As I couldn’t retrofit the drag-and-drop functionality in to the core View class, I created a DragArea class to handle visualisation and event dispatch for the drag operation. All Views wishing to receive drag events, or start a drag must be children of a DragArea. The DragArea class itself is based on a FrameLayout so is very easy to add in to the widget hierarchy.

Starting a drag is very similar to the Honeycomb API. A View wishing to start a drag operation must have a reference to the DragArea and provide some clip data to transfer along with an object used to draw the drag visualisation.

Bundle data = new Bundle();
data.putCharSequence("cliptext", "Some clip data goes in this bundle");
dragArea.startDrag(data, new ViewDragShadowBuilder(this);

As methods for receiving drag events are not built in to the View API, views wishing to receive drag events must first register for them. The DragEvent class itself is very similar to Honeycomb.

dragArea.addDragListener(this, new OnDragListener {
    @Override
    public void onDrag(View view, DragEvent dragEvent) {
        switch dragEvent.getAction() {
            case DragEvent.ACTION_DRAG_STARTED:
                break;
            case DragEvent.ACTION_DRAG_ENTERED:
                break;
            case DragEvent.ACTION_DRAG_EXITED:
                break;
            case DragEvent.ACTION_DROP:
                Bundle data = dragEvent.getBundle();
                CharSequence dropText = data.getCharSequence("cliptext");
                reportView.setText(dropText);
            case DragEvent.ACTION_DRAG_ENDED:
            default:
                break;
        }
    }
});

The code, as an android library project, is available on github as well as the documentation. A full example application that makes use of the library is also available.

I know that this work becomes pointless when most android phones are updated to Ice Cream, but until then it might be useful to someone who needs drag-and-drop on android 2.0. The similarity to the official API might make it easier to port code in the future.

I recently attempted to use Google Appengine to host my very simple personal website. This was something of an experiment to learn about cloud hosting. These are my initial thoughts on Google Appengine. They should be taken with a pinch of salt as I haven’t yet hosted a high bandwidth, or high availability site or application.

Google Appengine is a software platform for hosting web based applications developed in either Java or Python.

The Good

1. The SDK is super simple. It installed cleanly and easily, and contains a development web server, database and sandboxed Python environment. Its genuinely a pleasure.

2. Setting up is easy. Signing up for Google Appengine and creating a first app is straightforward. One doesn’t have to do anything dull like set-up Apache, MySQL, memcached or Ubuntu.

3. The Python runtime provides flexibility. Google Appengine uses a fairly standard Python 2.5 runtime with most of the batteries included. This made me feel fairly at home and means that you can use a number of Python web development libraries.

4. The Appengine is moderately Django friendly. Version 0.96 is available by default but more recent versions can be used instead. The documentation is explicit about how to do this. There is an official Google project for making it easy to port Django applications and an alternative based on Django non-rel.

5. The Appengine specific API’s are well documented and thought out. There are a few proprietary API’s for task management, datastore access and mailing. The ones I tried worked very well.

6. The Admin console is awesome. It is easy to search and add data to the datastore. There are good views of site traffic, and error logs. Its just generally useful. All without any setup.

The Bad

1. Appengine isn’t Django. Its not even close. The important part is that Google Appengine doesn’t provide any SQL database. This makes Django models incompatible with Appengine. Any existing Django application will require a decent amount of ported code to work properly. Djangoappengine from allbuttonspressed is supposed to get around this by providing a Django Model implementation on-top of Google Bigtable. For me it wasn’t without its flaws and probably isn’t production ready.

2. Domain setup is ugly. To set up domain hosting for Appengine you need to use a Google Apps account. This is confusing enough but while the Appengine Admin site is a joy the Apps console is clunky confusing and poor. It sullies an otherwise great experience. Besides, these are two fairly separate products, one is called Google Appengine and the other Google Apps. What went wrong in the naming department?

3. Google Appengine has no support for naked urls. These are of the form ‘ foobar.com’. This is related to integration with Google Apps. Apps are deployed as an ‘App’ on your Google Apps domain. This might be a dealbreaker for some. For me it was just a great annoyance.

The Horrible

1. Google Appengine does not provide a SQL server and its Bigtable data store is a proprietary Google interface. This means that once your site is written and deployed you may be locked-in to Googles servers for a very long time.

Conclusion

I have mixed feelings about Google Appengine. On the one hand it has served to convince me that cloud hosting is probably the best bet for future application deployment. On the other hand I don’t believe I would choose Google to do so.

It probably isn’t suited for people who want to base their business around the success of a large web application. More flexible hosting such as that provided by Rackspace and Amazon is a better bet. On the other hand its lack of an SQL server makes it fairly useless for existing Django or Java application deployment.

Its closely linked with Google Apps, probably the Apps Marketplace, and has API’s for using Google Accounts for authentication. If you want to hitch your business wagon eternally to Google it might be a good idea.

Perhaps the best use of Google Appengine would be for development of new business centric, or intranet applications. There is some sort of by-user pricing available that might be great. The ease of setup and development would be a help and the proprietary nature of the thing might not be such a party killer.

This post is a homage to Java, which this week taught me an important lesson. Although minor in itself, Apple’s decision to deprecate Java on OS X marks the point at which the project is strategically dead. If only by serving as a reminder of how useless desktop Java has been almost since its inception. This isn’t a major problem for me. I haven’t spent much time with it, and I’ve never written a single commercial Java application.

Strangely though, it is only six months since I thought I would one day invest heavily in the Java ecosystem. Languages like Scala and Clojure were making the JVM seem pretty sexy. The platform is a mature one and has an extremely healthy and innovative open-source community, creating some much hyped projects like Hadoop. Perhaps most importantly, Android has given its backing to the Java language. A combination of the Oracle lawsuit, abandonment of the Harmony project, and the official death of desktop Java, has however made things look rapidly gloomy.

Realising that Java is now well-and-truly over the hill caused me to have a very common epiphany. If something as large as Java is headed for the scrapheap, no technology is safe. Anything we invest our time in learning, loving and patching will eventually be an old war story for a crusty programmer. I remember seeing Andrew Morton at FOSDEM in 2007. He told me, and a room full of others, that they were preparing for the Linux kernel to last another twenty years. This probably gets to the root of the problem. Twenty years is an eternity in the fast-paced software industry, but less than half the length of a normal career.

When commenting on one of the recent nails in Java’s coffin Michael Meeks said, ‘Guard your Heart!’. He was referring to falling in love with a technology controlled by a single company. I’d go further than this. Guard your heart against falling in love with any single bit of technology. Open source projects can’t as easily die from a corporate decision, but they can fall prey to obsolescence, neglect or community implosion.

I don’t know whether this comment by Aaron Griffin is famous, but it deserves to be. Its probably a bad idea to call yourself a ‘Java programmer’ right now. Heaven forbid, it might one day be taboo to call oneself a ‘Ruby Developer’.

This week Techcrunch caused a stir by reporting that Facebook are building a phone. This has been flatly denied by Facebook, so we’ll assume for the moment that the reporting was erroneous and no such phone is on the way. I’m desperately disappointed. To repeat a joke first made by Nat Friedman: I think I’m going to mail $1000 to Palo Alto with a note, and sit waiting for my FacePhone. Here is why Facebook should be building a phone.

A phone is social

Nokia has apparently used its “Connecting People” advertising slogan since 1992, early recognition that phones are essentially social tools. There has been a huge amount bolted to phones since that time. They have become ‘convergant’ devices, an amalgam of your laptop, camera, walkman, GPS and Gameboy. There are some very obvious synergies between these functional modes and the core social networking purpose of a phone, but I see them as essentially separate. The camera is used in a pinch, the internet browser for info on-the-run, and gaming for entertainment on-the-bog. Phones are all about connecting people and they always will be. Unfortunately this is the aspect I believe has seen the smallest real innovation so-far in the smartphone era.

Despite the rise and rise of Facebook the 160 char text message is still the most widely used social application with 4.1 billion text messages sent daily. I don’t wish to belittle Facebook’s achievements here, 1 billion daily messages is truly a phenomenon.

Integration is poor

The feature most-likely to impress people when shown my Nexus One is that their Facebook profile photo shows up in the contact list. In-fact the Facebook integration is probably one of the best features of Android. However integration doesn’t run very deep, and in many ways is clearly broken. Merging of contacts is somewhat hit and miss. I’ve yet to work out when a contact gets merged and when it is duplicated. Facebook chat doesn’t work, or at least not in the default settings. While I’m able to keep in touch with a few of my friends through Google chat, this isn’t really where my canonical contact list lives. Part of the contact list problem is that so few people leave their phone numbers with Facebook, forcing me to keep two lists. A Facebook phone would surely go some-way to alleviate this issue by encouraging users to tie their Facebook and Phone accounts together.

All of the above issues could I’m sure be solved through a better, more integrated Facebook app experience but they will constantly be fighting against the vested interests closer to the consumer. Google especially has good reason to keep the Facebook experience poor and push its GMail / Google Talk experience as default on Android. The carriers would prefer consumers to continue using text messages. Although Apple doesn’t have any serious competitive areas with Facebook, I doubt whether the real ownership of user experience required for deep integration is something they would countenance.

There is a golden user experience waiting to be built here. One where ‘Blocking’ someone on Facebook also causes their calls to be screened. Profile pictures of long-lost friends will be shown with every phone call. Making friends could be as simple as bumping phones together and give you access to numbers, email and Facebook feeds. None of the existing phone OS’s have gone far enough and the number of players involved makes me feel it could be a long slog.

Location means revenue

I have no idea how Facebook are doing in building their revenue stream but ownership of the mobile experience is surely one way to give it a massive boost. Facebook are the kings of social and their recent addition of ‘Places’ shows that they are serious about competing with Foursquare and Yelp when it comes to location. Currently the Foursquare experience is as integrated in-to smart phones as Places, probably more so. A better integrated experience means more check-ins.

Google might have created a huge advantage here. If they can find a way to leverage Android to get more information about user location they will have the double whammy over Facebook in terms of user data. Google already have more ‘purchasing intent’ when people search on their web-site. With knowledge of where the user is they will likely have an insurmountable lead in delivering targeted advertising to consumers. Facebook is desperate to get more knowledge about their users, surely location is the next big driver for that success?

Consumers will win

For the consumer a FacePhone is win-win, even a commercial failure would force other innovators such as Goole and Apple to move at a faster pace. For Facebook the risks are great, but so are the rewards. Please Facebook, step up, and show us what real innovation means in mobile + social.

A long time ago, while working on D-Bus accessibility, I became frustrated with the existing methods for specifying the interfaces. D-Bus interfaces are normally specified using an XML format. Although this format describes the D-Bus protocol perfectly well, it is lacking lots of information that would be useful for D-Bus bindings and for documentation. To deal with this the D-Bus wizards decided that the XML could be littered with ‘annotations’, to extend the format and number of standard annotations sprang up around specific D-Bus bindings: EggDBus, Telepathy and QtDBus.

These annotation formats are difficult to read and edit. XML is moderately acceptable, but some restrictions of D-Bus XML make keeping the document consistent vey hard. The fact that they are limited to specific D-Bus libraries is also not ideal. I started work on a language for describing D-Bus interfaces that would address these issues. My idea was to have a readable syntax, enough features to clearly document a D-Bus interface, and tools to generate XML or code for the different D-Bus libraries.

After a long hiatus dbuf is finally in a state where the language parser is complete, and generation of D-Bus XML is supported. The source can be found at http://github.com/doffm/dbuf. There is a decent tutorial located in the doc folder, but a taste of what the language is like follows. The code is part of a real example; the AT-SPI D-Bus interface translated into dbuf.

using Attributes = org.freestandards.atspi.Attributes;
using Reference = org.freestandards.atspi.Reference;

/*
  The base interface which is implemented by all
  accessible objects.
*/

interface org.freestandards.atspi.Accessible {
    enum  Role {
        ROLE_INVALID = 0,
        ROLE_ACCELERATOR_LABEL,
        ROLE_ALERT,
        ROLE_ANIMATION,
        ROLE_ARROW,
        ROLE_CALENDAR,
        ROLE_CANVAS,
        ROLE_CHECK_BOX,
        ROLE_CHECK_MENU_ITEM,
    }

    /*
      Represents a bit-field of currently held states.
      TODO Could just be a uint64?
    */
    typedef uint32[] State;

    /* A short string representing the object's name. */
    read property string Name;

    /*
      The accessible object which is this objects containing
      parent.
     */
    read property Reference Parent;

    /*
      Access this objects non-hierarchical relationships
      to other accessible objects.
    */
    method GetRelationSet reply {
            RelationSet relations;
    }

    /*
     Get the Role indicating the type of
     UI role played by this object.
    */
    method GetRole reply {
            Role role;
    }

    /* Access the states currently held by this object. */
    method GetState reply {
            State state;
    }

    /*
      Get a properties applied to this object as a whole, as an
      set name-value pairs. As such these attributes may be
      considered weakly-typed properties or annotations, as
      distinct from the strongly-typed interface instance data.
    */
    method GetAttributes reply {
            Attributes attributes;
    }
}

There is lots more work to do with dbuf, but even in its current state I think it is a useful tool for describing complex D-Bus interfaces.

In a previous post I showed a small example of how to create a grammar and a parser using Antlr and Python. I made the point that simply parsing a file, and checking it conforms to the language grammar, isn’t usually all we want to do. As well as creating parsers for a language Antlr also makes it easy to create more useful programs through actions.

To follow these examples you should take a look at my previous post where I used a very simple CSS like language grammar which is shown below.

grammar simplecss;

options {
    language = Python;
}

/* Parser rules */

cssdocument : cssrule+ ;

cssrule : IDENTIFIER '{' declarationlist '}' ;

declarationlist : declaration+ ;

declaration : IDENTIFIER ':' IDENTIFIER ';' ;

/* Lexer rules */

fragment LETTER : 'A'..'Z'
                | 'a'..'z'
                | '_'
                ;

fragment DIGIT : '0'..'9' ;

IDENTIFIER : LETTER (LETTER | DIGIT)* ;

COMMENT : '/*' .* '*/' {$channel = HIDDEN} ;

WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+
             {$channel = HIDDEN} ;

CSS is built up of selectors and a list of properties for selectors. It seems that a useful data-type in Python would be a dictionary of dictionaries. The outer set of keys being the selectors, and the inner set being the property names. The css banana {color:yellow; length:10cm; shape:curved;} would be translated to the Python data type: {'banana': {'color': 'yellow', 'length': '10cm', 'shape': 'curved'}}.

A simple way of doing this would be to add actions to the Antlr grammar. An action appears as a block of code in curly-braces. The position of the action roughly equates to where in the parsing algorithm it will occur. For example we could print out the name of each css selector as we parse the document.

cssrule : i=IDENTIFIER {print $i.text}
          '{' declarationlist '}' ;

Notice that the $i symbol refers to a variable outside the code block. This is substituted for the real IDENTIFIER variable when Antlr generates the parser. In this sense the code block can be thought of as a template for the real code generated by Antlr. Detailed information about what symbols are allowed in actions can be found in the Antlr documentation. Details about what attributes can be used on the parser and lexer rules is found separately in the Attributes section. The code block will be executed after parsing the identifier, but before the declarationlist.

At this point we have to give a little thought to the code generated by Antlr from our grammar description. Each ‘rule’ within the grammar file is translated in to a function on the parser class that takes the stream of symbols as the input. For example, when parsing a whole document, the top-level rule function is used: res = parser.cssdocument (). The cssdocument function corresponds to the grammar rule of the same name. These rule functions have return values. It is possible to add members to the return type within the grammar file.

cssdocument returns [selectordict] : cssrule+ ;

To access this rule within an action the attribute selectordict is used as follows.

cssdocument returns [selectordict] :
{$cssdocument.selectordict = {}} cssrule+ ;

The problem with this is that the selectordict variable is only available within the cssdocument rule. To make it available to sub-rules, where it is needed, it would have to be passed as an argument. This is cumbersome. The solution is Antlr scopes. A scope is a stack-type variable that is available to all sub-rules. The stack type is useful for language features such as namespaces and variable scopes, hence the name. Scopes are declared using the scope keyword.

cssdocument
scope {
    selectorscope;
} : cssrule+ ;

Access for scopes is different to that of rule attributes and uses a syntax like C++ namespaces: $cssdocument::selectorscope.

Using actions, return-values and scopes it is now possible to create a parser that generates a python dictionary from a source file.

grammar simplecssactions;

options {
    language = Python;
}

/* Parser rules */

cssdocument returns [ruledict]
scope {
    ruledictscope;
    rulename;
}
@init {
    $cssdocument::ruledictscope = {}
    $cssdocument::rulename = ''
} : cssrule+ ;

cssrule : IDENTIFIER
{
    $cssdocument::rulename = $IDENTIFIER.text
    $cssdocument::ruledictscope[$IDENTIFIER.text] = {}
}
'{' declarationlist '}' ;

declarationlist : declaration+ ;

declaration : p=IDENTIFIER ':' v=IDENTIFIER ';'
{$cssdocument::ruledictscope[$cssdocument::rulename][$p.text] =
[$v.text]};

/* Lexer rules */

fragment LETTER : 'A'..'Z'
                | 'a'..'z'
                | '_'
                ;

fragment DIGIT : '0'..'9' ;

IDENTIFIER : LETTER (LETTER | DIGIT)* ;

COMMENT : '/*' .* '*/' {$channel = HIDDEN} ;

WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+
            {$channel = HIDDEN} ;

The new syntax in this example is the @init block of the cssdocument rule. This is called before anything else in the rule and is generally used for initialising local variables.

Using instructions given in my first blog post, it is now possible to compile this grammar in-to a parser using Antlr. Once the parser has been generated a compiler can be written that outputs a Python dictionary.

import sys
import antlr3
from simplecssactionsLexer import simplecssactionsLexer
from simplecssactionsParser import simplecssactionsParser

def main (argv):
    filename = argv[1]
    input = antlr3.FileStream (filename)
    lexer = simplecssactionsLexer (input)
    tokens = antlr3.CommonTokenStream (lexer)
    parser = simplecssactionsParser (tokens)
    res = parser.cssdocument ()
    print res

    if __name__ == "__main__":
        sys.exit (main (sys.argv))

As well as actions Antlr also has the ability to generate ASTs (Abstract Syntax Trees) from a conforming document. This is a tree data type that somewhat matches the syntax of the language. It is very useful when making multiple passes over the document, as it saves parsing it twice. Antlr has special grammar file syntax for easily generating the desired AST from any language.

Retrospective

I have had fun with Antlr. Despite only having rudimentary knowledge of parsing and compiling I created a decent compiler for a language of my own design. Perhaps the most impressive part of Antlr’s parser generation is the error reporting. Error reporting is extremely important not only for eventual users of any compiler but also during development. Antlr allowed me to create a professional feel to the program when I would not otherwise have had the time.

There are some issues. Antlr is big, and tries to remain consistent between its different modes with only some success. The different data types of Token, Rule, Tree, Template often caused me confusion. They are all handled in similar ways in the grammar files, but very differently in the eventual code. The code generation is also sometimes difficult. There does not seem to be a well worked template system for this, possibly because the code used isn’t restricted to any one particular language. The best example is the common use of the % symbol in python, which is also used in Antlr and causes issues.

If you are thinking about playing about with your-own language grammar you should probably try Antlr first. If you are lucky enough to be working in a language with a well worked parser combinator library then this may be more suitable. I doubt many are as complete or actively-maintained as Antlr.

A while ago I started writing an IDL for D-Bus, which was envisioned as a fairly complex language. A large number of libraries and programs were available to help but I settled on Antlr. I had tried flex / bison before and never had a good experience. Besides ruling out the ‘C’ version, this also discouraged me from using language implementations such as ply.

Antlr is a parser generator, similar to flex / bison, and written in Java. It has the advantage of having many ‘target’ languages. This means it is possible to create parsers in many languages including Python, C, and Java. As a parser generator Antlr has its own language for describing languages. This is supposed to make the grammar of the language clear, and act as a specification for the language as well as generating an implementation of a parser for it.

In Antlr the lexer and parser are described in the same file. The difference between a lexer and parser is technical but Antlr makes this distinction, parsing a language in two separate stages. The lexer is simpler and serves to split a sequence of characters into a sequence of words or tokens. The parser can be more complicated and is used to ‘recognise’ a sequence of symbols that conform to the grammar of the language.

Descriptions of languages are made up of a sequence of ‘rules’ each of these rules what is permissible within the language. Lexer rules start with an upper-case character, and parser rules start with a lower-case character. Below is a grammar file that parses a very simple css-like language.

grammar simplecss;

options {
    language = Python;
}

/* Parser rules */

cssdocument : cssrule+ ;

cssrule : IDENTIFIER '{' declarationlist '}' ;

declarationlist : declaration+ ;

declaration : IDENTIFIER ':' IDENTIFIER ';' ;

/* Lexer rules */

fragment LETTER : 'A'..'Z'
                | 'a'..'z'
                | '_'
                ;

fragment DIGIT : '0'..'9' ;

IDENTIFIER : LETTER (LETTER | DIGIT)* ;

COMMENT : '/*' .* '*/' {$channel = HIDDEN} ;

WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+
             {$channel = HIDDEN} ;

Starting with the big-picture the first rule in this file is simplecss. This is the top-level rule and matches a sequence of one or more css rules. Antlr syntax is somewhat consistent with regexp and feels moderately natural. The symbol indicating one-or-more is +.

cssrule : IDENTIFIER '{' declarationlist '}' ;

The css rule itself is an IDENTIFIER token followed by a declarationlist inside braces. The '{' is a literal token within a parser rule.

In the parser rules it is acceptable to use sub-rules, such as when declaring a declarationlist as one or more of the rule declaration. This is not true for lexer rules. The lexer is used to process a stream of characters in to a stream of tokens. Each lexer rule becomes a token. To use a lexer rule within others you must declare that it as a fragment. This means that it will not become a token and is used just for composing other lexer rules.

fragment DIGIT : '0'..'9' ;

IDENTIFIER : LETTER (LETTER | DIGIT)* ;

From the above rule you it can be seen that an IDENTIFIER token is very simple, just a word composed of letters and digits. As the IDENTIFER token is used in place of the selector, this language only allows very basic selectors.

COMMENT : '/*' .* '*/' {$channel = HIDDEN} ;

WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+
             {$channel = HIDDEN} ;

The above rules are taken from a page describing Antlr grammars. They serve to remove comments and extraneous whitespace from the token stream. This is very useful as comments and whitespace can appear almost anywhere within the language. If these were processed in to tokens the parser rules would become excessively complicated.

The documentation for Antlr grammars is excellent, as well as more involved tutorials available on the Antlr website there is also a book available that does a decent job in this area. The list of operators used to compose parser and lexer rules is found on the Antlr cheat sheet.

Generating Python code

Once the grammar of the language has been declared, Antlr is used to generate programs that recognise this language. To indicate that a program should be created in the Python language language = Python; is placed in the grammar options. The default language is Java.

Antlr version 3 is available in the Ubuntu repositories in the antlr3 package. I did not use this as it is a substantially older version. The latest can be downloaded from the project website. Some Python dependencies will also be required: the Antlr runtime module and the Stringtemplate module. The Antlr runtime library needs to be exactly the same version as the Antlr tool. This may mean that it is not possible to use the very latest version of Antlr with Python. I used version 3.1.2 of the Antlr tool and the Python runtime library. I used version 3.2.1 of the Stringtemplate module.

Once you have downloaded the Antlr tool as a jar you can run it directly.

> java -classpath antlr-3.1.2.jar org.antlr.Tool simplecss.g

This will generate two python files simplecssLexer.py and simplecssParser.py. Make sure that the python dependencies are in the PYTHONPATH. It is now possible to write a script that checks if a file conforms to the language grammar.

import sys
import antlr3
from simplecssLexer import simplecssLexer
from simplecssParser import simplecssParser

def main (argv):
    filename = argv[1]
    input = antlr3.FileStream (filename)
    lexer = simplecssLexer (input)
    tokens = antlr3.CommonTokenStream (lexer)
    parser = simplecssParser (tokens)
    res = parser.cssdocument ()

if __name__ == "__main__":
    sys.exit (main (sys.argv))

This tool is not very useful. All it can do is find out if a document is correct, and report errors if it is not. Antlr also has support for attaching actions to parser rules and translating a sequence of tokens in to an abstract syntax tree. A data structure that is easy to manipulate within Python. In a future post I will show how to use actions and abstract syntax trees to create something a little more exciting.