Want to chat? Donate to one of these charities!
Home Notes Popular RSS

August 5, 2021

Practical? Common Lisp on the JVM: A quick intro to ABCL for modern web apps

common lisplisparmed bear common lispjavajvm

In a ridiculous attempt to prove an internet wrong about the practicality of Lisp (Common Lisp specifically), I tried to get a simple (but realistic) web app running. After four days and a patch to ABCL I got something working.

The code I had in mind would look something like this:

(let* ((port 8080)
       (server (make-server port)))
  (route server "GET" "/" (lambda (ctx) "My index!"))
  (route server "GET" "/search"
    (lambda (ctx)
      (template "search.tmpl" '(("version" "0.1.0")
                                ("results" ("cat" "dog" "mouse")))))))

And search.tmpl would be some Jinja-like text file:

<html>
  <title>Version {{ version }}</title>
  {% for item in results %}
    <h2>{{ item }}</h2>
  {% endfor %}
</html>

The source code for this post can be found on Github.

Picking a language, libraries

Armed Bear Common Lisp (ABCL) is the only Common Lisp implementation I'm aware of that can hook into a major ecosystem of libraries like the JVM or CLR has. In theory, this makes it a safe suggestion for folks who want the stability and resources of the ecosystem even if they aren't using its flagship language.

I wanted to use some micro web framework like Spark or Micronaut.

The problem with libraries like Micronaut (and Jersey) is that they do a lot of dynamic inspection to figure out how to register controllers and whatnot. This is certainly convenient for developers using the library in Java. But it becomes an ordeal when you're trying to use the library through a foreign function interface (FFI) in another language. An example of this is if a framework scans all files in a directory for a @GET annotation.

On the other hand, Spark had a seeming hard-requirement about bringing in a Websocket library which caused some issues during configuration. So I ended up going with Jooby and Netty (as the underlying server).

Finally, I looked into a few Jinja-like template libraries and settled on Pebble since Jinjava wouldn't load for me.

3rd-party jars and foreign function calls

So you've got your maven dependencies and ran mvn install. Your pom.xml looks like this:

<?xml version="1.0" encoding="utf-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.github.eatonphil</groupId>
  <artifactId>abcl-rest-api-hello-world</artifactId>
  <version>1</version>

  <dependencies>
    <dependency>
      <groupId>io.jooby</groupId>
      <artifactId>jooby</artifactId>
      <version>2.10.0</version>
    </dependency>

    <dependency>
      <groupId>io.jooby</groupId>
      <artifactId>jooby-netty</artifactId>
      <version>2.10.0</version>
    </dependency>

    <dependency>
      <groupId>io.pebbletemplates</groupId>
      <artifactId>pebble</artifactId>
      <version>3.1.5</version>
    </dependency>
  </dependencies>
</project>

ABCL has a package called abcl-asdf that helps you resolve dependencies through Maven and your filesystem. We'll import it and a package it depends on (abcl-contrib):

(require :abcl-contrib)
(require :abcl-asdf)

All our code will go into a single main.lisp file.

To import a specific package from Maven you call abcl-asdf:resolve with a colon-separated string containing the Maven package group id and artifact id. Then you pass that result to abcl-asdf:as-classpath and pass that result to java:add-to-classpath.

It will look like this:

(setf imports '("io.jooby:jooby"
                "io.jooby:jooby-netty"
                "io.pebbletemplates:pebble"))
(loop for import in imports
      do (java:add-to-classpath
          (abcl-asdf:as-classpath (abcl-asdf:resolve import))))

Now you can call functions within these packages. If you want to call a Java method using only builtins it looks like (jcall "method" "com.organization.package.Class" object arg1 arg2 ... argN). If you want to call a static Java method you use (jstatic ...) instead of (jcall ...).

It seems that ABCL will automatically convert simple types from their Lisp representation to Java but it will not turn a list into an array. If a Java function requires an array you'll have to do that explicitly with a function like (java:jnew-array-from-list "java.lang.String" my-string-list).

When using the builtin Java FFI you always need to use the fully qualified name for classes like java.lang.Object for Object or java.util.Array for Array.

Alternatively you can (require :jss) to get access to a simpler syntax for making Java calls. A method call looks like (#"method" object arg1 arg2 ... argN). Creating a new instance of an object is calling (jss:jnew 'className). When you use JSS you don't need to fully qualify a class name unless there are more than one class with the same name. For example to create a new Jooby application instance we can call (jss:jnew 'Jooby). As long as the class can be found in the class path JSS will resolve it.

Some real code

The real code will look similar to the pseudo-code at the top of this article. We'll stub out the library-specific wrappers for rendering a template and for registering a route.

Fumbling around the Jooby source code we see this snippet of Java:

 * Server server = new Netty(); // or Jetty or Utow
 *
 * App app = new App();
 *
 * server.start(app);
 *
 * ...
 *
 * server.stop();

Netty comes from the jooby-netty artifact in the io.jooby group on Maven. And App is some object that extends io.jooby.Jooby. Since we're not using an OOP language though we're going to try avoiding classes as much as possible. So we'll just create a new instance of io.jooby.Jooby and add routes directly to it.

(defun template (filename context)
  "")

(defun route (app method path handler)
  nil)

(defun register-endpoints (app)
  (route app "GET" "/"
         (lambda (ctx) "An index!"))
  (route app "GET" "/search"
         (lambda (ctx)
             (template "search.tmpl" `(("version" "1.0.0")
                                       ("results" ,(java:jarray-from-list '("cat" "dog" "mouse")))))))
  (route app "GET" "/hello-world"
         (lambda (ctx) "Hello world!")))

(let* ((port 8080)
       (server (jss:new 'Netty))
       (app (jss:new 'Jooby)))
  (register-endpoints app)
  (#"setOptions" server (#"setPort" (jss:new 'ServerOptions) port))
  (#"start" server app)
  (#"join" server))

Easy enough. Now we just need to implement route and template.

Implementing Java classes in ABCL

We are again not going the happy path with fancy Java syntax (which is fine if you're using Java) like the Jooby documentation suggests. Scouring the Jooby source code again it looks like we can call route on the Jooby class with a method string, a path string, and an instance of an object implementing the io.jooby.Route.Handler interface.

Since this handler argument is an interface, we cannot cheat again by creating an instance of it we'll have to actually create a new class in Lisp that extends it. Thankfully there's only one method we need to implement to satisfy this interface, apply. It accepts a io.jooby.Context object and returns a java.lang.Object. The framework then does introspection to figure out what exactly the object is and if it needs to transform it into a string to be returned as an HTTP response body.

To create a new class in ABCL we call (java:jnew-runtime-class "classname" :interfaces '("an interface name") :methods '(("method name 1" "return type" ("first parameter type" ...) (lambda (this arg1 ...) body)))):

(defun route (app method path handler)
  (#"route"
   app
   method
   path
   (jss:new (java:jnew-runtime-class
             (substitute #\$ #\/ (substitute #\$ #\- path))
             :interfaces '("io.jooby.Route$Handler")
             :methods `(
                       ("apply" "java.lang.Object" ("io.jooby.Context")
                        (lambda (this ctx)
                          (funcall ,handler ctx))))))))

One thing to note is that when referring to a subclass within a file we need to address it with the io.jooby.Route$Handler syntax rather than as you might refer to it in Java as io.jooby.Route.Handler. In the latter case ABCL thinks Route is a package when in fact it's just a class.

If you run this now with abcl --load main.lisp. It will work until you hit an endpoint. The problem is how Jooby tries to figure out the real type of the returned object.

The app will crash somewhere around here calling analyzer.returnType(route.getHandle()).

In this case it tries to open and parse the (Java) source code of our application to try to find the return type for this apply function.

That's a problem since our code isn't Java. Through trial and error I realized we can trick Jooby/Java/somebody into figuring out the correct return type by adding another implementation of apply that returns a String to our class.

The full route code now looks like this:

(defun route (app method path handler)
  (#"route"
   app
   method
   path
   (jss:new (java:jnew-runtime-class
             (substitute #\$ #\/ (substitute #\$ #\- path))
             :interfaces '("io.jooby.Route$Handler")
             :methods `(
                       ;; Need to define this one to make Jooby figure out the return type
                       ;; Otherwise it tries to read "this file" which isn't a Java file so cannot be parsed
                       ("apply" "java.lang.String" ("io.jooby.Context")
                        (lambda (this ctx) nil))
                       ;; This one actually gets called
                       ("apply" "java.lang.Object" ("io.jooby.Context")
                        (lambda (this ctx)
                          (funcall ,handler ctx))))))))

You may wonder, why keep the original method around? Well it's because during reflection, ABCL says no such method that returns String exists in the Handler interface. That's fair I guess.

Implementing the template

The Java example on the Pebble homepage is perfect:

PebbleEngine engine = new PebbleEngine.Builder().build();
PebbleTemplate compiledTemplate = engine.getTemplate("home.html");

Map<String, Object> context = new HashMap<>();
context.put("name", "Mitchell");

Writer writer = new StringWriter();
compiledTemplate.evaluate(writer, context);

String output = writer.toString();

We can easily translate this into Lisp:

(defun hashmap (alist)
  (let ((map (jss:new 'HashMap)))
    (loop for el in alist
         do (#"put" map (car el) (cadr el)))
    map))

(defun template (filename context-alist)
  (let* ((ctx (hashmap context-alist))
         (path (java:jstatic "of" "java.nio.file.Path" filename))
         (file (#"readString" 'java.nio.file.Files path))
         (engine (#"build" (jss:new 'PebbleEngine$Builder)))
         (compiledTmpl (#"getTemplate" engine filename))
         (writer (jss:new 'java.io.StringWriter)))
    (#"evaluate" compiledTmpl writer ctx)
    (#"toString" writer)))

But if you run this abcl --load main.lisp and hit this /search endpoint, it will blow up saying "no such method" exists at the call to Path.of(filename).

After digging around I saw it was because Path.of is a variadic function.

And while there are examples of using variadic functions when the function only has a single parameter like java.util.Arrays.asList(T ...), employing that same technique here continued to result in "no such method":

         (path (java:jstatic "of" "java.nio.file.Path" filename (jnew-array "java.lang.String" 0)))

Eventually I found an example of someone doing reflect/invoke on this kind of a function call and tried this logic on a local copy of the ABCL source code.

It worked. So I opened a pull request.

So the full working code for template is:

(defun template (filename context-alist)
  (let* ((ctx (hashmap context-alist))
         (path (java:jstatic "of" "java.nio.file.Path" filename (java:jnew-array "java.lang.String" 0)))
         (file (#"readString" 'java.nio.file.Files path))
         (engine (#"build" (jss:new 'PebbleEngine$Builder)))
         (compiledTmpl (#"getTemplate" engine filename))
         (writer (jss:new 'java.io.StringWriter)))
    (#"evaluate" compiledTmpl writer ctx)
    (#"toString" writer)))

And to get this diff running locally:

$ mkdir ~/vendor
$ cd ~/vendor
$ git clone https://github.com/eatonphil/abcl
$ cd abcl
$ git checkout pe/more-variadic
$ sudo {dnf/brew/apt} install ant maven
$ ant -f build.xml

And to run main.lisp using this diff:

$ ~/vendor/abcl/abcl --load main.lisp

And to hit the API:

$ curl localhost:8080/search
<html>
<title>Version 1.0.0</title>
  <h2>cat</h2>
  <h2>dog</h2>
  <h2>mouse</h2>
</html>
$ curl localhost:8080/hello-world
Hello world!%

Phew! Easy peasy.

Next up

I'm porting this example to Kawa to see how it fares. Blog post to come.