Monday, 21 February 2011

Creating Android applications with Clojure: Slimming things down with ProGuard

Last time, I described how to integrate Clojure into the standard Android build process. In this post, I will build on this by integrating ProGuard into the build process in an effort to make applications smaller and faster.

First, I will give a brief overview of ProGuard and describe how to enable the ProGuard capability of the Android build process. Next, I will show how to configure ProGuard so that it can work with your Clojure code. Finally, I will summarise my recommendations and let you know what to expect next in this series.

The sample application I will use is available on my Clojure-Android-Examples repository on GitHub under the slimmed directory.

About ProGuard

As described on its web site, ‘ProGuard is a free Java class file shrinker, optimizer, obfuscator, and preverifier.’ What does this mean?

ProGuard can rid your application of unused classes, methods, and fields, leading to smaller application sizes and load times.
ProGuard can analyse your code and perform a large number of optimisations, such as removing unnecessary field accesses and method calls, merging identical code blocks, inlining method invocations, etc. This should make your code smaller and run faster (depending on VM implementation).
ProGuard can remove debug information normally included class files and rename classes, methods, etc. to make them meaningless. The idea is to make it harder to reverse engineer a compiled application into source code.
This is something specific to Java 6 or Java ME VMs. As a result, it does not apply to working with Android.

Now that you have an idea of what ProGuard can do, how can you start using it?

Enabling ProGuard

It is very simple to enable ProGuard when you are creating a release build of your application using ant release. All you have to do is edit the file created by Android and add the following line:

proguard.confg = proguard.cfg

If you want to enable ProGuard for debug builds, you will also need to override the -debug-obfuscation-check target. I set up my target as follows:

<target name="-debug-obfuscation-check" if="proguard.enabled">
    <path id="out.dex.jar.input.ref"/>

Line two above ensures that the libraries you are using, such as Clojure, will be processed in addition to your application’s code.

These two changes will give you the ability to do the following:

  • When running ant install or ant debug, your program will be built as usual.
  • If you want to create a ProGuard-processed debug build, simply use ant -Dproguard.enabled=true install or ant -Dproguard.enabled=true debug.

If you want to run ProGuard for every build, you can simply add proguard.enabled = true to your file.

However, if you try to run ProGuard now, you will find your build fails. First, it has to be configured.

Configuring ProGuard

The ProGuard configuration bundled with the latest Android SDK is configured with some sane defaults. However, if you are doing Clojure development, you will need to make some adjustments in order for ProGuard to work correctly. The following sections should help guide you to use ProGuard with your application.

All of these changes will be made to the proguard.cfg file in the root directory of your application.

Shrinking your application with ProGuard

ProGuard’s default configuration will not even allow the build process to complete. To get proper shrinking, you must silence some warnings and ensure that all of your application’s code is not shrunk away. If you are following along, you may want to add the following two lines to your proguard.cfg before continuing any further:


Naturally, these disable ProGuard’s optimization and obfuscation functionality. By disabling them here, you can concentrate on getting one thing working at a time.

Silencing Clojure-related warnings and notes

Clojure refers to a number of classes that are not available on Android, particularly the Swing components used from the clojure.inspector and namespaces and some java.beans classes used by clojure.core/bean. ProGuard will complain about these missing referenced classes, but these warnings can be disabled by adding the following lines to proguard.cfg:

-dontwarn clojure.inspector**,**,clojure.core$bean*
-dontnote clojure.inspector**,**

With these two lines in place, the sample application shrinks from an install size of 4.04MB to only 460KB. Also, instead of taking 4.7 seconds to load, it crashes almost instantaneously. Why? It's because most of Clojure has been stripped out.

Keeping Clojure

The reason why Clojure is stripped out is because ProGuard does not understand how Clojure initialises a namespace. Briefly, when loading a namespace, Clojure tries to do one of two things:

  1. Read and evaluate the source file, if available.
  2. Check to see if the namespace was AOT-compiled by trying to load the namespace’s initialisation class and its corresponding load method.

For example, when trying to load the namespace org.deepbluelambda.example, Clojure will look for the file org/deepbluelambda/example.clj or the class org.deepbluelambda.example__init.

Therefore it is necessary to configure ProGuard to keep Clojure’s core initialization classes. This can be done by adding the following lines to proguard.cfg:

-keep class clojure.core__init { public static void load(); }
-keep class clojure.core_proxy__init { public static void load(); }
-keep class clojure.core_print__init { public static void load(); }
-keep class clojure.genclass__init { public static void load(); }
-keep class clojure.core_deftype__init { public static void load(); }
-keep class clojure.core.protocols__init { public static void load(); }
-keep class clojure.gvec__init { public static void load(); }
-keep class { public static void load(); }

You must include these eight lines, otherwise your application will fail to run. If you use any other Clojure namespaces, you will need to add a corresponding line. For example, if you use clojure.set and clojure.contrib.string, you will need to add the following lines to your proguard.cfg:

-keep class clojure.set__init { public static void load(); }
-keep class clojure.contrib.string__init { public static void load(); }

Taking these steps will help ensure that Clojure itself loads up, but how about your application?

Keeping your Application

In addition to making sure all of Clojure’s namespaces load properly, you will need to ensure that all of your application’s namespaces load as well. The approach is the same; just be sure not to forget any gen-classed namespaces.

Furthermore, if you expose superclass methods with gen-class, as below, you will need to ensure that ProGuard does not delete those methods.

  (:gen-class :extends
              :main false
              :exposes-methods {onCreate superOnCreate})

To ensure that the above activity works fine, the following lines need to be added to the ProGuard configuration:

-keep class {
    public static void load();
-keep class {
    public *** super*(...);

The first three lines should be familiar to you, the second three lines will preserve any public method, with any return and parameter types, so long as its name starts with ‘super’. Thus, it should be able to handle any number of :exposes-methods so long as the exposed method name starts with ‘super’.

Measuring the results

Now that you have successfully shrunk your application, what can you expect? The following table summarises the results from my sample application:

Metric Basic Slimmed Savings (%)
Package size (KB) 1007 644 36
Installed application size (MB) 4.43 2.77 37
Start time on device A (s) ~4.7 ~4.2 12
Start time on device B (s) ~3.0 ~2.3 23

As can be seen, shrinking definitely helps significantly reduce application package and installation sizes. The effect on performance tends to be much more device-dependent.

However, one caveat about the above data. Most of the runtime savings is realised by the absence of the clojure.set, clojure.xml, and namespaces. In Clojure version 1.2, these namespaces are automatically loaded, if available. If you actually use and include the namespaces, much of the runtime savings will disappear. In version 1.3, Clojure will no longer try to automatically load these namespaces.

Optimizing with ProGuard

The default proguard.cfg already disables certain optimisations known not to work on Android. Unfortunately, I find that ProGuard often fails to analyse Clojure code. During optimization, you may encounter an error similar to the following:

Unexpected error while performing partial evaluation:
  Class       = [clojure/java/io$buffer_size]
  Method      = [invoke(Ljava/lang/Object;)Ljava/lang/Object;]
  Exception   = [java.lang.IllegalArgumentException] (Stacks have different current sizes [2] and [1])

I have tried using the latest released version of ProGuard, but have found that it still has problems analysing Clojure code. As a result, I recommend that you disable optimisation by including the following in your proguard.cfg:

Obfuscating with ProGuard

As with optimization, the default configuration enables obfuscation. Your application will most likely build, but you will get an error similar to the following at runtime:

java.lang.IllegalStateException: b already refers to: class clojure.b.a.b in namespace: clojure.core

Unfortunately, I do not see any easy way to resolve this problem. As a result, for now, I recommend you disable obfuscation as follows:


Final Thoughts

ProGuard can definitely help reduce the size of your application by eliminating any namespaces you are not using. Additionally, if you are not using clojure.set, clojure.xml, or, you can expect a runtime benefit of faster loading when using Clojure 1.2.

Whether or not to enable ProGuard during regular development depends on the nature of your application. If the number of classes that are removed is large enough, the time spent shrinking will be offset by the time not spent processing all of those classes into a Dalvik executable. In some cases, you may forced to shrink your application, as there is a limit of 65536 method references per dex file. You may run into this limit if you are using large libraries (such as the Scala or JRuby core libraries) or many different libraries.

Unfortunately, at this time it does not seem like optimization and obfuscation with ProGuard is compatible with Clojure code. There may be ways to work around some of these problems, such as fine-tuning optimizations or running your code through ProGuard in stages with different configurations.

Next time

In my next post, I demonstrate how improve application start-up responsiveness by using a Java bootstrap activity. Additionally, I plan to show how to exploit the bootstrap activity to use supplemental class loaders to circumvent the dex file format shortcomings and drastically improve build times.