Monday, 12 December 2011

Clojure REPL Tip: Loading Scripts

Ever wanted to write a script once and then run it using the Clojure REPL for Android? I was recently asked about this via e-mail, and I thought it would be good to make a note of it here.

First, you will need to write your script and get it onto your phone in a world-readable location. Usually, the SD card should work out well enoughi, which is generally mounted at /sdcard. Therefore, if you transfer your script coolscript.clj to the SD card, it should show up at /sdcard/coolscript.clj.

Now, all you need is the load-file command. To run the script above, you would just enter:

(load-file "/sdcard/coolscript.clj")

Limitation

Note there is one important limitation to this approach. It does not change the path for which scripts are looked up. If your script requires searching for other files that define namespaces, it generally will not work. Adding directories to the search path is proposed feature that would be nice to add.

Behind the scenes

I have not abandoned the REPL. Since the Conj, I have been working on the REPL to help get some more feedback to the Clojure/dev team to improve Clojure on Android. At the Conj, I announced Neko, the Clojure/Android Toolkit. I have since updated it to be compatible with the latest versions of both Clojure and the Android SDK. I also now have a version of Clojure for Android that is compatible with the latest 1.4 developments.

Please stay tuned for more Clojure/Android news and a Clojure/conj wrap-up.

Monday, 18 July 2011

Announcing Baitha: The Scala/Android toolkit

Over the past month or so, I have started developing an Android application for a client. Perhaps one of the most interesting aspects of developing this application is that it is being created in Scala and C++. In addition to being a great opportunity to do some larger-scale development for Android and with Scala, it has given me the opportunity to start developing a toolkit for Scala and Android developers.

Baitha is hosted on GitHub and is licensed using the Apache Software License 2.0. I hope it may be useful to other Scala/Android developers, and I would love feedback from developers in terms of comments, patches, etc. The library is currently designed to be used an Android library project and can easily be integrated using a snapshot version of the SBT Android plugin. For more information, please consult the readme.

Right now, the library supports the following features:

  • AlertDialogBuilder: A wrapper around Android’s AlertDialog.Builder class that provides some implicit conversions and a mini-DSL.
  • BuildMode: A trait that makes converting between debug/production code easier.
  • EnhancedViews: Provides some implicit conversions to make adding listeners as functions.
  • Logger: A replacement logging interface inspired by RoboGuice.
  • SdkVersions: Offers conditional execution depending on the runtime version of the Android platform.

Please check it out, use it, and help improve it.

Thursday, 19 May 2011

The Motorola Xoom and the Asus Eee Pad Transformer: a developer’s perspective

I have been getting more deeply involved in Android development of late, and these days that means developing for both phones and tablets. Unfortunately, the emulator for Honeycomb is incredibly slow. Therefore, I decided to purchase the Wi-Fi version of the Motorola Xoom shortly after it was released.

Just a month later, I learnt about the Asus Eee Pad Transformer and its well-integrated keyboard dock. Seeing the opportunity to have a device that can be used as a netbook, I decided to get one of these as well.

Now that I have had the chance to use both of these devices, I thought I would take the time to write about both of them primarily from the standpoint of a developer.

The Motorola Xoom

The Motorola Xoom was the first Honeycomb tablet, and Google used it as their development device. It was initially released as a 3G model for Verizon, and later as a less expensive Wi-Fi only model. Both versions have 32GB of built-in memory, and a non-functioning microSDHC card slot. Overall, it is a well-built and solid tablet.

Developing with the Xoom

Developing applications with the Xoom is a fairly standard experience. Simply connect the tablet to your computer via USB, just as you would with most any other Android device.

However, if you want to work at a lower level, the Xoom’s bootloader can easily be flashed just like a Nexus device. Unfortunately, it seems like the full source code for Honeycomb will not be released, at least not until it’s released as Ice Cream Sandwich.

On the cutting edge

One thing that makes the Xoom particularly interesting to developers is that it appears to be the primary tablet used for development at Google. As a result, it is the first to get operating system updates. Android 3.1 was announced at Google I/O, and within days both versions of the tablet started getting OTA updates to this latest Android release.

Not quite complete

Unfortunately, despite its expense, it is not quite complete. It has now been three months since the initial release of the Xoom, and the software to make the microSDHC card slot is still vapour-ware. This is particularly embarrassing since other tablets have now shipped with working slots.

Similarly, for the 3G version, the upgrade to LTE will require mailing the device to Motorola and waiting at least a week for it to be returned. For a user, this is very annoying; for a developer, this is potentially downtime.

The Asus Eee Pad Transformer

The Transformer was released in the US late last month. In many ways, it is a typical Android tablet: 10.1 inch 1280×800 pixel display, 1GB RAM, and a 1.0GHz dual core Tegra 2 processor. It is currently on Wi-Fi only, and available in both 16GB and 32GB configurations. It also has a working microSDHC card slot.

However, what really sets the Transformer apart is this keyboard dock. Slipping your tablet into the dock transforms your tablet into a nice Honeycomb netbook. In addition to the full keyboard, the dock also adds a full size SDHC/MMC card reader, two full-size USB host mode ports, and a battery that combines with the tablet’s battery to provide upwards of 14 hours’ worth of power.

Developing with the Transformer

Like the Xoom, you can just plug the Transformer into your computer’s USB slot, and you are ready to develop. However, the Transformer has a couple of differences that make it less developer-friendly.

The tablet (and dock) use a proprietary socket that handles charging the battery, connecting to the dock, and connecting to a computer. The most important implication of this is that you cannot charge your tablet at the same time you do development. To make matters worse, the cable included with the tablet is quite short.

In practice, the battery life of the tablet is fairly good. Using the dock should give enough battery life to handle any marathon coding session, but if you forget to recharge your batteries you may end up having to disconnect from the computer in order to charge the tablet. I have not investigated if the dock’s USB ports can be used for an adb session.

Additionally, the Transformer does not have an unlocked bootloader. This makes it much more difficult to root the device or to do any low-level development. It is too bad Asus has chosen not to make the device more hackable.

Keeping up to date

According to Asus, Android 3.1 will not be released for the Transformer until sometime in June. While this may not be as speedy as some may like, it is roughly in line with other non-Xoom tablets.

One way in which I believe Asus has been better than Motorola is in releasing updates to fix problems experienced by users. Some engineers from Asus apparently hang out in the XDA developer forums, and have worked with users to solve various software problems. The result has been a series of updates since the product’s launch.

Beyond the tablet experience

The one thing that struck me while using the Xoom is that a tablet makes a splendid consumption device. It is good for surfing the web, watching a video, or playing a game. However, as a production device, such as writing e-mail or taking notes, it could not replace my laptop. It is true that the large screen makes typing on the tablet much better than trying to do the same on a phone, but it still cannot match the productivity of having a full keyboard.

The Transformer promises to change all of that. The keyboard dock will allow me to replace my laptop for many lighter tasks, such as checking e-mail, writing, and even some system administration and programming. I am currently working on modifying [ConnectBot][cbot], the popular Android SSH client, to work with the Asus keyboard (and other full keyboards via USB and bluetooth). This will help make the tablet not just a platform for development, but a useful tool in its own right.

Choosing a tablet

The Xoom and the Transformer are not the only options for an Android tablet. However, at the moment, I think these two tablets are the most compelling for the Android developer. Which should you choose? That all depends on your needs.

Choose the Xoom, if you…
  • Want to be running the latest Android release
  • Need to easily root your device or modify the firmware
  • Do not mind paying a premium. The Wi-Fi version retails at $600, $100 more than the equivalent Transformer.
Choose the Transformer, if you…
  • Want the best value in an Android tablet. At $400 for the 16GB version, it is the least expensive Honeycomb tablet on the market.
  • Would like to get more out of your tablet. You can spend the same amount of money and get a better netbook. However, if you are going to get an Android tablet for development anyway, spending the extra $150 for the dock may change how you use your tablet.

Tuesday, 05 April 2011

Decaffeinated Robot: source, slides, and audio

This past weekend the second Texas Linux Fest was held in Austin, Texas. I had a great time attending the various sessions and the expo space. I am also glad to have had the opportunity to speak about alternative approaches to Android development. Mobile development is certainly a hot topic these days, and I have been told that the room overflowed into the hallway for my talk.

As promised, here are the slides, source code, and audio from my presentation:

I would like to thank the TXLF organisers for a great event. I am looking forward to TXLF 2012.

Wednesday, 23 March 2011

Decaffeinated robots at Texas Linux Fest

In just a week and a half, I will be in Austin, Texas, presenting at Texas Linux Fest, a ‘state-wide, community-run conference for Linux and open source software users and enthusiasts from around the Lone Star State’.

My presentation, The decaffeinated robot: Developing on Android without Java, will present an overview of the current landscape for Android development without using Java, including HTML-based applications, alternative JVM languages, use of the native development tools (NDK), the Scripting Layer for Android (SL4A), and other development tools.

I have created a simple game that should allow you to compare implementations in different languages. I have completed the following:

  • Python using SL4A
  • Ruby using Ruboto
  • Clojure
  • Scala using sbt-android-plugin
  • Java with C snippets using the NDK and JNI

Time permitting, I may create more implementations.

So, for this and many more great sessions, register now for Texas Linux Fest in Austin, Texas, on 2 April 2011. Registrations start at $15, and there will be free Friday training sessions. I hope to see you there!

Sunday, 27 February 2011

Clojure for Android source published

Over the next few weeks, I will be publishing the source code for the Clojure REPL for Android in a few different instalments:

  1. Clojure for Android, a modified version of Clojure adapted to run on the Dalvik virtual machine;
  2. Clojure Android Toolkit, a library of utilities for Clojure developers working on Android; and
  3. Clojure REPL, the source code of the application itself.

I have now published the modifications to my source code in a repository available on GitHub. My work is based on the 1.2.x branch of the Clojure source code and is available in the android-1.2.x branch.

This post will document my goals for Clojure on Android, give an overview of the changes I have made, describe the current implementation of dynamic compilation, and present areas for future work.

Goals

The three primary goals of the Clojure for Android release are as follows:

  1. Create a version of Clojure that works for both the Java and Dalvik virtual machines in the hope that the changes can eventually be included in Clojure itself,
  2. Create a development version of Clojure that supports dynamic compilation to enable more rapid development of applications, and
  3. Create a lean Clojure runtime that will deliver acceptable performance on Android devices.

Overview of changes

There really are not many changes in this initial release of Clojure for Android. They fall into three categories: the addition of a Dalvik-specific dynamic class loader, some minor runtime changes, and an update to the build configuration to support Android.

New DynamicClassLoader hierarchy

The most significant change is to DynamicClassLoader. In the original implementation, this class manages a constant pool, maintains a cache of class definitions, provides (deprecated) class path alteration capability, and is in charge of turning compiled class byte codes into classes available within the virtual machine.

In my implementation, it retains all of those abilities save for the last one. It is now an abstract class that delegates class realisation to its subclasses, of which there are two:

  1. JvmDynamicClassLoader relies on Java's standard ClassLoader.defineClass method.
  2. DalvikDynamicClassLoader uses a tortuous method, described later.
Runtime changes

There are a few relatively minor runtime changes:

  • The addition of a new var, clojure.core/*vm-type*, which will be set to either :dalvik-vm or :java-vm at runtime.
  • Choosing the correct DynamicClassLoader implementation depending on *vm-type*
  • A workaround for a bug fixed in ‘FroYo’ where the context class loader is set to a system class loader instead of the application’s class loader
  • The pre-emptive loading of clojure.set, clojure.xml, and clojure.zip is disabled on Dalvik.

That’s it.

Build system update

The build system has received somewhat more extensive changes. There are two basic scenarios:

  1. Building Clojure without Android support: This should work just fine. Just run ant as usual.
  2. Building Clojure with Android support: You will need to create a local.properties file with pointers to the Android SDK directory and SDK version you want to use. More documentation is available in readme.txt.

When building with Android support, the build file will create a stripped down version of the dx.jar file from the Android SDK. By default, this will do a simplistic removal of purely test classes. However, if you have ProGuard, it can do a more exhaustive shrinking. This is enabled by setting the proguard.jar property.

When Android is enabled, the build will create two additional JAR files:

  1. clojure-nosrc.jar, the opposite of clojure-slim.jar, a compiled-only version of Clojure. This JAR also contains the dx tool classes that are needed at runtime.
  2. clojure-dex.jar, a version of clojure-nosrc.jar where all of the classes have been compiled into a Dalvik executable. This file is suitable for loading by one of Android’s class loading mechanisms.

Dynamic compilation

To illustrate how I implemented dynamic compilation in Clojure, I will first present the traditional path from compiled Java class to instantiated Dalvik class. Next, I will show how the modified version of Clojure takes dynamically generated classes through the same process. Finally, I will present the trade-offs involved in the current implementation and what you should keep in mind when using the dynamic compilation.

Traditional work flow

The following is a brief description of the path of a compiled class file from build to execution:

  1. At build time:
    1. Java files are compiled into Java classes made up of JVM byte codes.
    2. All of the classes are prepared into a Dalvik Executable (DEX file) by the dx tool. This file is called classes.dex.
    3. The DEX file is placed into the Android package.
  2. At install time:
    1. The installer reads the DEX file from the package.
    2. The DEX file is verified to remove illegal instructions and performs some computations to aid in garbage collection.
    3. The verified DEX data is then optimized, creating a hardware- and platform-specific version of the code. Some optimizations include replacing virtual method call resolution with indices in a vtable, inlining method calls, pruning empty methods, etc.
    4. The resulting optimised DEX file (ODEX file), is written to a special cache directory.
  3. At run time:
    1. The ODEX file is checked to make sure it is still valid. If not, then the original DEX file is again verified and optimised.
    2. The application loads its classes from the ODEX file.
Dynamic Clojure work flow
  1. The Clojure evaluator compiles a form into a class using the embedded ASM bytecode engineering library.
  2. The DalvikDynamicClassLoader processes the compiled byte code as follows:
    1. It uses the embedded dx tool to translate the JVM class into an in-memory DEX file.
    2. It writes the DEX file into a temporary JAR file in *compile-path*.
    3. It uses Android’s dalvik.system.DexFile to load the JAR file. In doing so, Android will create an ODEX file in *compile-path*.
    4. Loads the class from the DexFile object and returns it.
Trade-offs

The main disadvantage to this form of dynamic compilation is that it is slow. It requires using the disk, as well as performing all sorts of computations at runtime. Anyone who has used the Clojure REPL for Android can attest to its sluggishness.

Unfortunately, to the best of my knowledge, there are no other accessible APIs available for doing this better. Most of the work is done in native code, making it difficult to bundle it into Clojure. There is some hope that this may change in the future. From the Dalvik documentation:

Some languages and frameworks rely on the ability to generate bytecode and execute it. The rather heavy dexopt verification and optimization model doesn't work well with that.

We intend to support this in a future release, but the exact method is to be determined. We may allow individual classes to be added or whole DEX files; may allow Java bytecode or Dalvik bytecode in instructions; may perform the usual set of optimizations, or use a separate interpreter that performs on-first-use optimizations directly on the bytecode

Until such an API is released, it is necessary to either take the slow but simple route, or to create a Clojure compiler for the Dalvik VM from scratch.

I think that dynamic compilation is of interest primarily to developers. Most applications will have no need for dynamic compilation as they can be AOT-compiled. As such, the slowness may well be acceptable. After all, waiting seconds for a function to recompile in your running application is much more tolerable than needing to go through a full compile-deploy cycle that may be measured in minutes.

Caveats

There are two things to be aware of when using dynamic compilation:

  1. You will need to be sure to point *compile-path* to some place where your application has write access.
  2. Some forms may blow the stack during compilation, such as (for [x (range 5) y (range 5)] [x y]). This is a limitation of the runtime.

Future work

There is still much to be done. As the source is now released, I look forward to seeing what sorts of feedback and improvements will come from others. Given the three goals I stated above, I think much of the work is as follows:

Integration with upstream

Get feedback on how to best integrate these changes into the language from other Clojure developers in general and, I hope, the Clojure/core team itself. Of course, it will most likely take some time before these changes make it into a Clojure release.

Clojure for Android development

While not perfect, I think the current solution largely satisfies this goal. Writing a new compilation back-end may make things better, but I am not sure that it will provide as good a return as working on the third goal.

I think that any new development should follow the master branch of Clojure going forward. The patches to the code itself should be simple enough to manage. Porting the build system changes to Maven will be more cumbersome.

Lean Clojure runtime

This is the place where the most work needs to be done. There have already been some good ideas presented on how to improve this, such as:

  • Eliminating metadata from compiled code to reduce memory footprint,
  • Finding ways to cut down on the immense amount of object churn during bootstrap, and
  • Generally finding ways to cut down on the amount of work done during the bootstrapping process.

My current idea is to find ways to modularise clojure/core.clj somewhat to be able to either completely eliminate some functionality (such as those for dynamic compilation) or at least delay loading it. Not every program makes use of every feature of the language. Some programs may never use one or more of: agents, futures, primitive vectors and arrays, etc. If there were some way to make some of these things load-on-demand, if only in an Android environment, that could significantly improve bootstrap times.

I look forward to the feedback from others and welcome any help in trying to get these things working. I think that it is quite possible to make Clojure a first-class development language for the Android platform.

Wednesday, 23 February 2011

Presenting Clojure REPL at February CHUG meeting

From the mailing list:

We'll be meeting this Thursday at the studio (1024 Studewood), at 7pm. Daniel will be talking about his Clojure REPL for Android. Very cool!

I'll order Pink's pizza again, if people liked that last time. Please RSVP for pizza so I know how much to order. And bring $5 to donate to the cause if you're having any. :)

The usual instructions:

LOCATION

1024 Studewood is a block-ish south of 11th on Studewood, aka Studemont, aka Montrose. If you're coming north on Studewood, it's on your right just past Stella Sola (which is on the left). It's a small green house, the only one that sticks out almost all the way to the road. It will make you think it's a photography studio, because it is.

DO NOT PARK IN THE LOT NEXT DOOR. YOU MAY GET TOWED.

There are driveways immediately on either side of the house, sufficient for two cars each (unless you drive something big). Otherwise, park on a side street (10 1/2 St.), they're easily accessible across Studewood, or curbside on Studewood. Just don't park in the the lot(s) right next to the studio, they are used as valet space for Stella Sola.

My plan is to give an informal presentation, primarily focusing on the changes I made to get Clojure with dynamic compilation working on the Dalvik VM. I will also demonstrate the VimClojure integration with a REPL running on a device. I plan to share my thoughts on the current state of and future expectations for Android development with Clojure.

It will be a fun evening filled with Clojure, Android, and pizza. I hope to see you there.

Monday, 21 February 2011

Creating Android applications with Clojure: Slimming things down with ProGuard

Last time, I described how to integrate Clojure into the standard Android build process. In this post, I will build on this by integrating ProGuard into the build process in an effort to make applications smaller and faster.

First, I will give a brief overview of ProGuard and describe how to enable the ProGuard capability of the Android build process. Next, I will show how to configure ProGuard so that it can work with your Clojure code. Finally, I will summarise my recommendations and let you know what to expect next in this series.

The sample application I will use is available on my Clojure-Android-Examples repository on GitHub under the slimmed directory.

About ProGuard

As described on its web site, ‘ProGuard is a free Java class file shrinker, optimizer, obfuscator, and preverifier.’ What does this mean?

Minimisation
ProGuard can rid your application of unused classes, methods, and fields, leading to smaller application sizes and load times.
Optimisation
ProGuard can analyse your code and perform a large number of optimisations, such as removing unnecessary field accesses and method calls, merging identical code blocks, inlining method invocations, etc. This should make your code smaller and run faster (depending on VM implementation).
Obfuscation
ProGuard can remove debug information normally included class files and rename classes, methods, etc. to make them meaningless. The idea is to make it harder to reverse engineer a compiled application into source code.
Preverification
This is something specific to Java 6 or Java ME VMs. As a result, it does not apply to working with Android.

Now that you have an idea of what ProGuard can do, how can you start using it?

Enabling ProGuard

It is very simple to enable ProGuard when you are creating a release build of your application using ant release. All you have to do is edit the build.properties file created by Android and add the following line:

proguard.confg = proguard.cfg

If you want to enable ProGuard for debug builds, you will also need to override the -debug-obfuscation-check target. I set up my target as follows:

<target name="-debug-obfuscation-check" if="proguard.enabled">
    <path id="out.dex.jar.input.ref"/>
<target/>

Line two above ensures that the libraries you are using, such as Clojure, will be processed in addition to your application’s code.

These two changes will give you the ability to do the following:

  • When running ant install or ant debug, your program will be built as usual.
  • If you want to create a ProGuard-processed debug build, simply use ant -Dproguard.enabled=true install or ant -Dproguard.enabled=true debug.

If you want to run ProGuard for every build, you can simply add proguard.enabled = true to your build.properties file.

However, if you try to run ProGuard now, you will find your build fails. First, it has to be configured.

Configuring ProGuard

The ProGuard configuration bundled with the latest Android SDK is configured with some sane defaults. However, if you are doing Clojure development, you will need to make some adjustments in order for ProGuard to work correctly. The following sections should help guide you to use ProGuard with your application.

All of these changes will be made to the proguard.cfg file in the root directory of your application.

Shrinking your application with ProGuard

ProGuard’s default configuration will not even allow the build process to complete. To get proper shrinking, you must silence some warnings and ensure that all of your application’s code is not shrunk away. If you are following along, you may want to add the following two lines to your proguard.cfg before continuing any further:

-dontoptimize
-dontobfuscate

Naturally, these disable ProGuard’s optimization and obfuscation functionality. By disabling them here, you can concentrate on getting one thing working at a time.

Silencing Clojure-related warnings and notes

Clojure refers to a number of classes that are not available on Android, particularly the Swing components used from the clojure.inspector and clojure.java.browse namespaces and some java.beans classes used by clojure.core/bean. ProGuard will complain about these missing referenced classes, but these warnings can be disabled by adding the following lines to proguard.cfg:

-dontwarn clojure.inspector**,clojure.java.browse**,clojure.core$bean*
-dontnote clojure.inspector**,clojure.java.browse**

With these two lines in place, the sample application shrinks from an install size of 4.04MB to only 460KB. Also, instead of taking 4.7 seconds to load, it crashes almost instantaneously. Why? It's because most of Clojure has been stripped out.

Keeping Clojure

The reason why Clojure is stripped out is because ProGuard does not understand how Clojure initialises a namespace. Briefly, when loading a namespace, Clojure tries to do one of two things:

  1. Read and evaluate the source file, if available.
  2. Check to see if the namespace was AOT-compiled by trying to load the namespace’s initialisation class and its corresponding load method.

For example, when trying to load the namespace org.deepbluelambda.example, Clojure will look for the file org/deepbluelambda/example.clj or the class org.deepbluelambda.example__init.

Therefore it is necessary to configure ProGuard to keep Clojure’s core initialization classes. This can be done by adding the following lines to proguard.cfg:

-keep class clojure.core__init { public static void load(); }
-keep class clojure.core_proxy__init { public static void load(); }
-keep class clojure.core_print__init { public static void load(); }
-keep class clojure.genclass__init { public static void load(); }
-keep class clojure.core_deftype__init { public static void load(); }
-keep class clojure.core.protocols__init { public static void load(); }
-keep class clojure.gvec__init { public static void load(); }
-keep class clojure.java.io__init { public static void load(); }

You must include these eight lines, otherwise your application will fail to run. If you use any other Clojure namespaces, you will need to add a corresponding line. For example, if you use clojure.set and clojure.contrib.string, you will need to add the following lines to your proguard.cfg:

-keep class clojure.set__init { public static void load(); }
-keep class clojure.contrib.string__init { public static void load(); }

Taking these steps will help ensure that Clojure itself loads up, but how about your application?

Keeping your Application

In addition to making sure all of Clojure’s namespaces load properly, you will need to ensure that all of your application’s namespaces load as well. The approach is the same; just be sure not to forget any gen-classed namespaces.

Furthermore, if you expose superclass methods with gen-class, as below, you will need to ensure that ProGuard does not delete those methods.

(ns com.sattvik.android.clojure.slimmed.ClojureBrowser
  (:gen-class :extends android.app.Activity
              :main false
              :exposes-methods {onCreate superOnCreate})
  …)

To ensure that the above activity works fine, the following lines need to be added to the ProGuard configuration:

-keep class com.sattvik.android.clojure.slimmed.ClojureBrowser__init {
    public static void load();
}
-keep class com.sattvik.android.clojure.slimmed.ClojureBrowser {
    public *** super*(...);
}

The first three lines should be familiar to you, the second three lines will preserve any public method, with any return and parameter types, so long as its name starts with ‘super’. Thus, it should be able to handle any number of :exposes-methods so long as the exposed method name starts with ‘super’.

Measuring the results

Now that you have successfully shrunk your application, what can you expect? The following table summarises the results from my sample application:

Metric Basic Slimmed Savings (%)
Package size (KB) 1007 644 36
Installed application size (MB) 4.43 2.77 37
Start time on device A (s) ~4.7 ~4.2 12
Start time on device B (s) ~3.0 ~2.3 23

As can be seen, shrinking definitely helps significantly reduce application package and installation sizes. The effect on performance tends to be much more device-dependent.

However, one caveat about the above data. Most of the runtime savings is realised by the absence of the clojure.set, clojure.xml, and clojure.zip namespaces. In Clojure version 1.2, these namespaces are automatically loaded, if available. If you actually use and include the namespaces, much of the runtime savings will disappear. In version 1.3, Clojure will no longer try to automatically load these namespaces.

Optimizing with ProGuard

The default proguard.cfg already disables certain optimisations known not to work on Android. Unfortunately, I find that ProGuard often fails to analyse Clojure code. During optimization, you may encounter an error similar to the following:

Unexpected error while performing partial evaluation:
  Class       = [clojure/java/io$buffer_size]
  Method      = [invoke(Ljava/lang/Object;)Ljava/lang/Object;]
  Exception   = [java.lang.IllegalArgumentException] (Stacks have different current sizes [2] and [1])

I have tried using the latest released version of ProGuard, but have found that it still has problems analysing Clojure code. As a result, I recommend that you disable optimisation by including the following in your proguard.cfg:

-dontoptimize
Obfuscating with ProGuard

As with optimization, the default configuration enables obfuscation. Your application will most likely build, but you will get an error similar to the following at runtime:

java.lang.IllegalStateException: b already refers to: class clojure.b.a.b in namespace: clojure.core

Unfortunately, I do not see any easy way to resolve this problem. As a result, for now, I recommend you disable obfuscation as follows:

-dontobfuscate

Final Thoughts

ProGuard can definitely help reduce the size of your application by eliminating any namespaces you are not using. Additionally, if you are not using clojure.set, clojure.xml, or clojure.zip, you can expect a runtime benefit of faster loading when using Clojure 1.2.

Whether or not to enable ProGuard during regular development depends on the nature of your application. If the number of classes that are removed is large enough, the time spent shrinking will be offset by the time not spent processing all of those classes into a Dalvik executable. In some cases, you may forced to shrink your application, as there is a limit of 65536 method references per dex file. You may run into this limit if you are using large libraries (such as the Scala or JRuby core libraries) or many different libraries.

Unfortunately, at this time it does not seem like optimization and obfuscation with ProGuard is compatible with Clojure code. There may be ways to work around some of these problems, such as fine-tuning optimizations or running your code through ProGuard in stages with different configurations.

Next time

In my next post, I demonstrate how improve application start-up responsiveness by using a Java bootstrap activity. Additionally, I plan to show how to exploit the bootstrap activity to use supplemental class loaders to circumvent the dex file format shortcomings and drastically improve build times.

Friday, 18 February 2011

Clojure REPL for Android now in beta

Just a quick post to announce that my Clojure REPL is now officially in ‘beta’. This new release includes some key usability enhancements:

  • It is now possible to click on historical input and have it be inserted into the entry field.
  • If an exception occurs while processing input, the entry field text is retained.
  • Focus automatically shifts back to the entry field after evaluation.

This does not make entering new input any easier, but should nonetheless help make things more pleasant.

Of course, this release also includes the minor enhancements from the silent ‘alpha2’ release from yesterday:

  • Shorten the time from the end of the load animation to the display of the REPL
  • Removal of some dead or unusable code, including the clojure.inspector, clojure.java.browse, and clojure.java.javadoc namespaces
  • Add Clojure sources so the source macro now works
Download

The REPL is available via the Android Market and from Deep Blue Lambda.

Coming soon

I will be presenting the Clojure REPL at the Clojure Houston User Group (CHUG) meeting this coming Thursday, the 24th of February. I will post more details about this later. Join the mailing list for the latest information.

This weekend, I plan to publish the next post in the ‘Creating Android applications with Clojure’ series, which will focus on using ProGuard.

I am also planning to release all of the Clojure REPL source code in the next week or so.

In the meantime, I appreciate your feedback via twitter, e-mail, or by commenting below.

Tuesday, 15 February 2011

Clojure REPL for Android now available

My Clojure REPL is now available on the Android Market. For now, there really is not all that much to it, but it does show that the full power of the Clojure language, including dynamic compilation, is possible on the Android platform.

The one extra feature the REPL includes is an embedded VimClojure server. I added this to the REPL in order to speed up development. In the end, instead of disabling that code, I simply made it an option that can be toggled by the user.

I do plan to develop the application further. Some of my ideas for new features include:

  • Make it easier to use, especially if a hardware keyboard is unavailable
  • Add the ability to run scripts from files

I will be publishing my modified Clojure source soon. At that time, I will describe in detail some of the changes that I have made, as well as some of the challenges that remain.

Of course, I am also interested in any feedback from users. So, please feel free to leave a comment, tweet, or e-mail me.

UPDATE: If you do not have access to an Android device and want to try it out on the SDK, the Clojure REPL is now availble from Deep Blue Lambda.

Friday, 11 February 2011

Preview of Clojure REPL for Android

I have taken some time over the past week on getting a Clojure REPL running on Android. At this point, it is pretty close to being ready for an initial release. Once ready, it should be available from the Android Market. In the meantime, here are a couple of screen shots:

Animated loading screen The REPL in action

Stay tuned for more.

Sunday, 06 February 2011

Creating Android applications with Clojure: Building with Ant

In part one of this series, I wrote about some of the current obstacles to creating a user-friendly Clojure application for Android, as well as some techniques which may be useful in creating a better user experience. In this post, I will begin concentrating on Android Clojure development from the developer's point of view by examining how to integrate Clojure into Android’s build process.

I will briefly describe the typical Android compile and deploy cycle, and then show how to modify the build to integrate Clojure. Afterwards, I will note some of the ways developing for Android with Clojure differs from other Clojure development. Finally, I will provide a link to some example code that implements the ideas presented here and provide a preview for my next post.

The Android build work flow

Broadly speaking, the standard Android build uses Ant to automate the following steps:

  1. Perform automated code generation based on the application’s resources and Android Interface Definition Language1 files.
  2. Compile both the generated code and the application source code to Java class files using the standard Java compiler javac.
  3. Generate Dalvim VM byte codes from the Java classes and any project libraries using the dx tool provided by the Android SDK. The result is a file called classes.dex.
  4. Finally, create and sign an Android package file that includes the classes.dex file and any application resources. This package can then be installed on an Android device.

The good thing about using Ant as a build tool, is that it is relatively easy to modify the build process. In fact, Android includes a number of extension points, such as -pre-compile or -post-compile. It is also possible to completely replace Android’s stock tasks with your own.

In the following section, I will show how to modify the second step above to include Clojure compilation.

Clojure and Android integration using Ant2

The following will demonstrate how to add Clojure compilation to a stock Android project. For the purposes of this exercise, I will assume that this is a fresh project created using the Android SDK and its android create project command. If you need more help with this step, check the Android Documentation.

It will be necessary to make a number of changes to the build.xml file created by Android. Be sure to make these changes in the file before the line that reads <setup />.

Adding new input directories

The following set of lines modifies where Android will look for source code such that instead of having one set of source files under src, it will look under src/java. You can then place your code under src/clojure.

<!-- Modified input directories -->
<property name="source.dir" value="src/java" />
<property name="clojure.source.dir" value="src/clojure" />
<property name="clojure.source.absolute.dir" location="${clojure.source.dir}" />

It is important to do this even if you have no Java code, as Android will copy any non-Java source files out of its input directory and add them to the Android package. As such, if you place your Clojure code where it expects to find its Java code, your source will be included in your compiled application, resulting in unneeded bloat and an unintentional release of your code.

Adding new output directories

The next bit of code allows you to direct the output of Clojure compilation to a different location than Android expects. For now, Clojure will compile to the same location as Java.

<!-- Clojure output directories -->
<property name="out.dir" value="bin"/>
<property name="clojure.out.dir" value="${out.dir}" />
<property name="clojure.out.absolute.dir"
          location="${clojure.out.dir}" />
<property name="clojure.out.classes.dir"
          value="${clojure.out.absolute.dir}/classes" />
<property name="clojure.out.classes.absolute.dir"
          location="${clojure.out.classes.dir}" />
Adding the Clojure targets

This Ant target will ensure that the output directories for Clojure exist before compilation:

<!-- Ensure that Clojure directories exist -->
<target name="-clojure-dirs">
    <mkdir dir="${clojure.out.classes.absolute.dir}" />
</target>

Finally, this next target is in charge of doing the actual Clojure compilation:

<!-- Compile Clojure namespaces -->
<target name="-clojure-compile" depends="-clojure-dirs">
    <java classname="clojure.lang.Compile"
          classpathref="android.target.classpath"
          fork="true"
          failonerror="true">
        <sysproperty key="clojure.compile.path"
                     value="${clojure.out.classes.absolute.dir}"/>
        <classpath>
            <pathelement path="${clojure.source.absolute.dir}"/>
            <path refid="jar.libs.ref"/>
            <fileset dir="${jar.libs.dir}" includes="*.jar" />
            <pathelement path="${out.classes.absolute.dir}"/>
        </classpath>
        <!-- Add your Clojure namespace here -->
        <arg value="com.sattvik.android.clojure.basic.ClojureBrowser"/>
    </java>
</target>

You will need to modify line 16 above and replace with your namespace. If you have multiple namespaces to compile, just add multiple lines, each with a different value.

Wiring Clojure to the Android build process

All that is left is to ensure that Ant will run the targets added above. In this implementation I replace Android’s compile target with a new one. This new target will execute both Android’s standard compilation and the Clojure compilation added above.

<!-- Hook Clojure compile into Android build work flow  -->
<target name="compile" depends="android_rules.compile,-clojure-compile">

Final steps

Before you compile and deploy your Clojure application to the emulator or your device, there are some things you will have to do:

  1. Remove the “Hello, world!” activity generated by Android under the src directory.
  2. Create the src/java and src/clojure directories.
  3. Install the Clojure JAR file to the libs directory.
  4. Create your new Clojure application under src/clojure.

Once you have done these steps, you can use the following commands:

  • ant compile compiles your application.
  • ant debug creates a development version of your program as an Android package.
  • ant install installs the Android package onto an Android device or a running emulator.

Observations

In my experience, developing with Clojure for Android is a bit different from developing in other environments. For one, the build speed is quite a bit slower, which is problematic because there is no REPL that will run on the platform itself. Also, as I noted my last post, application sizes are relatively large for a mobile. However, there are ways to address some of these issues.

Build speed

Once everything is set up, you will find that compiling your application will be fairly quick. However, when it comes time to package your application and deploy it to your device, it will be much slower. In my experience, compilation takes only a few seconds, but creating the full package takes about a half minute.

The reason for this is that once the application has been compiled, the JVM bytecodes need to be transformed to Dalvik bytecodes, a memory- and CPU-intensive task. Ideally, the Clojure library could be processed once and the result could be reused. Unfortunately, Android does not support this.

No REPL

One of the great things about developing in Clojure is being able to use a REPL within your application for the purpose of interactive development. Being able to dynamically update your code without waiting for a complete compile-deploy cycle is invigorating. Unfortunately, on Android, with the stock Clojure library, this is impossible at this time.

The reason for this is that Clojure generates JVM classes, and these are incompatible with the Dalvik VM. Off-hand, I can see two possible resolutions to this:

  1. Use the Android dx tool at runtime to convert Clojure’s JVM bytecode into Dalvik executables which can then be loaded.
  2. Create a whole new byte-compiling back end for Clojure that uses Dalvik VM bytecodes instead of JVM bytecodes.

The primary advantage of the first approach is that it is probably the easiest and least intrusive method. In fact, I am using it to develop a Clojure REPL application I plan to release soon. Unfortunately, it is also extremely inefficient.

The second approach is really the ideal solution, but it is going to be a much more difficult task and will quite possibly be too disruptive to include as a standard part of Clojure any time soon.

Application size

If you examine the contents of an Android package built using the above steps, you may find that it includes source files from Clojure, such as clojure/core.clj. This is because the standard Clojure JAR file includes them, and Android happily copies them into your package. However, these files are useless and add nothing but dead weight. For best results, remove these files from the Clojure JAR.

Things to come

I originally intended this to be a two-part series, but it turns out that I have much more material than I originally thought. Next time, I will show how to include ProGuard into the build cycle in order to slim your application and weed out unneeded classes. I will also show how this affects the runtime behaviour of the application.

I appreciate your feedback. If you have any requests or ideas, feel free to comment below.

Example code

I have created an example application written in Clojure that provides a very basic web browser. With each new instalment of this series, I will update the repository with a version of the code that uses the new techniques. The examples are available from GitHub.

Footnotes
  1. From the Android documentation: Android Inteface Definition Language is an IDL language used to generate code that enables two processes on an Android-powered device to talk using interprocess communication (IPC).
  2. If you prefer working with Maven, check out android-clojure-flashlight-example.

Friday, 04 February 2011

Creating Android applications with Clojure, Part 1

Over the past few weeks, I have started doing some application development on the Android platform. Overall, it has been a pleasant experience. The documentation, while not the best, is fairly good. Being an experienced Java programmer, I was comfortable enough to start writing real code quickly.

Despite all of this, writing in Java just is not as much fun any more. There is too much boilerplate code, and the language is missing some key features, such as closures. So, I decided to look into developing with some alternate JVM languages such as Scala and Clojure

In this two-part series of posts, I will discuss my experiences with coding in Clojure for Android in an effort to document what has worked for me. First, I will concentrate on those issues that affect the user’s experience. In the next part I will write about the development process.

Loading, please wait…

By far, the biggest problem with a Clojure application on Android is the load time. On a device, it take at least five to ten seconds for an activity written in Clojure to load up. This may be acceptable for some long-running applications, such as a game, but it is likely to be frustratingly slow for many other applications.

Investigating the problem

Profiling an application during this long wait time reveals that it is due to to the loading of the class clojure.lang.RT. One of the primary functions of this class is to bootstrap the Clojure runtime so that all of the basic language functionality is available for your program.

Unfortunately, for a constrained environment such as a mobile phone, this simply takes too long. In fact, it is loading the clojure.core name space that is root of the problem, and some of that work is unnecessary.

Slimming down Clojure

The most direct way to solve this problem is to have less of the language to load. In fact, there are some parts of Clojure that you simply cannot use on an Android device, such as dynamic code generation and compilation, e.g. gen-class or defrecord. You can AOT compile code that uses these forms, but you cannot generate new types within your application at runtime.

Perhaps the most direct way to help resolve the loading issue to create a modified version of Clojure that omits some of this functionality, a sort of Mini-Clojure. Perhaps a better solution may be to somehow make it so that at start-up only a minimal set of functionality is available and the rest loads up either in the background or on demand. Either way, slimming clojure.core is not a trivial endeavour.

Bootstrapping your application

Presuming that you are not going to hack Clojure for a quick start-up time, how do you create a better user experience during start-up? Since the problem is caused by loading the clojure.lang.RT class, the key is to choose when and how that happens.

If you code your main activity in Clojure, this means that loading Clojure will happen before anything is displayed to the user. Ideally, what would be better is to show a splash screen or progress dialog while Clojure loads in the background.

The easiest way to do this is ensure that your initial activity is written in Java. Using this approach, the onCreate method of your activity launches another thread that will load Clojure in the background. Once it is loaded, it can call back to your activity in the GUI thread so that your application can continue. At this point, you have two options:

  1. Integrate your Java-based activity with your Clojure code by directing all of the various call-back methods to your Clojure code.
  2. Launch a second activity that is pure Clojure.

I have had success with the first approach. Once you get passed the long initial wait, the rest of the application seems to run as well as a Java-based program. I have not tried the second approach, but I do not see why it should not work.

I believe a long start-up time is the biggest obstacle to a user-friendly Clojure program on Android. However, by loading Clojure in the background, it is possible to distract the user with a progress bar, splash screen, or animation.

Application size

The second major problem for users is the size of a Clojure-based application. Most users will be running your program on a mobile where disk space is limited, RAM is rare, and bandwidth is expensive.

A minimal Clojure application including the full Clojure libraries will be about one megabyte to download. This may not seem too bad as far as applications go, but it is enormous compared to the minimal Java application of about twenty kilobytes.

Once downloaded, the platform will uncompress the minimal Clojure program to about four megabytes. Again, not too bad, but still relatively large given that it does nothing. In my experiments, deleting some of the unused libraries such as clojure.test and clojure.java.javadoc helps save some space, but not significantly.

It is hard to calculate exactly how much memory use can be attributed to Clojure, but my informal experiments indicate that it may be about 1.5 megabytes. It has certainly been my experience that small Clojure programs use much more memory than comparable Java programs. On Android, a small memory footprint is quite important as you do not want to have your program killed to free up memory. If your application is killed, the user will have to endure the long start-up time once more.

Final thoughts

I am certainly happy to note that you can do Android development in Clojure, but the current standard Clojure library for the JVM does not make it easy. The good news is that good Android support is a goal for the Clojure development team. I believe that given some attention, it is quite feasible to develop a stream-lined Clojure library suitable for the Android platform1.

Nonetheless, it is possible to use the above bootstrapping technique to create a user experience better than a ‘black screen of death’. Given your particular application, it may even be perfectly acceptable.

Coming in part two

In part two, I will shift focus away from the user to describe how the development process works with Clojure. In particular, I will examine how the inability to dynamically generate code on the platform impacts development and some techniques for working around the problem.

Footnotes
  1. In fact, if someone is interested in sponsoring the work, I would be delighted to take on the task.

Friday, 28 January 2011

Slides on Java concurrency from CHUG

Yesterday, the Clojure Houston User Group (CHUG) met to discuss the concurrency models provided by different programming languages, namely Erlang, Go, Java, and Scala. As usual, the meeting lasted long into the evening with lively conversation. In particular, I was quite interested in learning more about Erlang and Scala—two languages I have been considering learning for some time.

I presented on the concurrency tools available in Java, briefly talking about the following key topics:

  • Task management using the Executor framework
  • State management using atomic variables and concurrency-aware collections
  • Task coordination using locks and synchronizers
  • A preview of things to come in Java 7

While my presentation was by no means exhaustive, I hope that it served to introduce people to the rich set of tools available in Java for writing concurrent applications. For more comprehensive resources, try:

The slides for my presentation are available here, and the source code to the demos is available via Github.