In a ridiculous attempt to prove an internet wrong about the practicality of Lisp (Common Lisp specifically), I tried to get a simple (but realistic) web app running. After four days and a patch to ABCL I got something working.
The code I had in mind would look something like this:
(let* ((port 8080)
(server (make-server port)))
(route server "GET" "/" (lambda (ctx) "My index!"))
(route server "GET" "/search"
(lambda (ctx)
(template "search.tmpl" '(("version" "0.1.0")
("results" ("cat" "dog" "mouse")))))))
And search.tmpl would be some Jinja-like text file:
<html>
<title>Version {{ version }}</title>
{% for item in results %}
<h2>{{ item }}</h2>
{% endfor %}
</html>
The source code for this post can be found on Github.
Picking a language, libraries
Armed Bear Common Lisp (ABCL) is the only Common Lisp implementation I'm aware of that can hook into a major ecosystem of libraries like the JVM or CLR has. In theory, this makes it a safe suggestion for folks who want the stability and resources of the ecosystem even if they aren't using its flagship language.
I wanted to use some micro web framework like Spark or Micronaut.
The problem with libraries like Micronaut (and
Jersey) is that they do a
lot of dynamic inspection to figure out how to register controllers
and whatnot. This is certainly convenient for developers using the
library in Java. But it becomes an ordeal when you're trying to use
the library through a foreign function interface (FFI) in another
language. An example of this is if a framework scans all files in a
directory for a @GET annotation.
On the other hand, Spark had a seeming hard-requirement about bringing in a Websocket library which caused some issues during configuration. So I ended up going with Jooby and Netty (as the underlying server).
Finally, I looked into a few Jinja-like template libraries and settled on Pebble since Jinjava wouldn't load for me.
3rd-party jars and foreign function calls
So you've got your maven dependencies and ran mvn
install. Your pom.xml looks like this:
<?xml version="1.0" encoding="utf-8"?>
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>com.github.eatonphil</groupId>
<artifactId>abcl-rest-api-hello-world</artifactId>
<version>1</version>
<dependencies>
<dependency>
<groupId>io.jooby</groupId>
<artifactId>jooby</artifactId>
<version>2.10.0</version>
</dependency>
<dependency>
<groupId>io.jooby</groupId>
<artifactId>jooby-netty</artifactId>
<version>2.10.0</version>
</dependency>
<dependency>
<groupId>io.pebbletemplates</groupId>
<artifactId>pebble</artifactId>
<version>3.1.5</version>
</dependency>
</dependencies>
</project>
ABCL has a package called abcl-asdf that helps you resolve dependencies through Maven and your filesystem. We'll import it and a package it depends on (abcl-contrib):
(require :abcl-contrib)
(require :abcl-asdf)
All our code will go into a single main.lisp file.
To import a specific package from Maven you
call abcl-asdf:resolve with a colon-separated string
containing the Maven package group id and artifact id. Then you pass
that result to abcl-asdf:as-classpath and pass that
result to java:add-to-classpath.
It will look like this:
(setf imports '("io.jooby:jooby"
"io.jooby:jooby-netty"
"io.pebbletemplates:pebble"))
(loop for import in imports
do (java:add-to-classpath
(abcl-asdf:as-classpath (abcl-asdf:resolve import))))
Now you can call functions within these packages. If you want to call
a Java method using only builtins it looks like (jcall "method"
"com.organization.package.Class" object arg1 arg2 ... argN). If
you want to call a static Java method you use (jstatic
...) instead of (jcall ...).
It seems that ABCL will automatically convert simple types from their
Lisp representation to Java but it will not turn a list into an
array. If a Java function requires an array you'll have to do that
explicitly with a function like (java:jnew-array-from-list
"java.lang.String" my-string-list).
When using the builtin Java FFI you always need to use the fully
qualified name for classes like java.lang.Object
for Object or java.util.Array
for Array.
Alternatively you can (require :jss) to get access to a
simpler syntax for making Java calls. A method call looks
like (#"method" object arg1 arg2 ... argN). Creating a
new instance of an object is calling (jss:jnew
'className). When you use JSS you don't need to fully qualify a
class name unless there are more than one class with the same
name. For example to create a new Jooby application instance we can
call (jss:jnew 'Jooby). As long as the class can be found
in the class path JSS will resolve it.
Some real code
The real code will look similar to the pseudo-code at the top of this article. We'll stub out the library-specific wrappers for rendering a template and for registering a route.
Fumbling around the Jooby source code we see this snippet of Java:
* Server server = new Netty(); // or Jetty or Utow
*
* App app = new App();
*
* server.start(app);
*
* ...
*
* server.stop();
Netty comes from the jooby-netty artifact in
the io.jooby group on Maven. And App is some
object that extends io.jooby.Jooby. Since we're not using
an OOP language though we're going to try avoiding classes as much as
possible. So we'll just create a new instance
of io.jooby.Jooby and add routes directly to it.
(defun template (filename context)
"")
(defun route (app method path handler)
nil)
(defun register-endpoints (app)
(route app "GET" "/"
(lambda (ctx) "An index!"))
(route app "GET" "/search"
(lambda (ctx)
(template "search.tmpl" `(("version" "1.0.0")
("results" ,(java:jarray-from-list '("cat" "dog" "mouse")))))))
(route app "GET" "/hello-world"
(lambda (ctx) "Hello world!")))
(let* ((port 8080)
(server (jss:new 'Netty))
(app (jss:new 'Jooby)))
(register-endpoints app)
(#"setOptions" server (#"setPort" (jss:new 'ServerOptions) port))
(#"start" server app)
(#"join" server))
Easy enough. Now we just need to implement route
and template.
Implementing Java classes in ABCL
We are again not going the happy path with fancy Java syntax (which is
fine if you're using Java) like the Jooby documentation
suggests. Scouring the Jooby source code
again
it looks like we can call route on the Jooby
class with a method string, a path string, and an instance of an
object implementing the io.jooby.Route.Handler interface.
Since this handler argument is an interface, we cannot cheat again by
creating an instance of it we'll have to actually create a new class
in Lisp that extends it. Thankfully there's only one method we need to
implement to satisfy this interface,
apply. It
accepts a io.jooby.Context object and returns
a java.lang.Object. The framework then does introspection
to figure out what exactly the object is and if it needs to transform
it into a string to be returned as an HTTP response body.
To create a new class in ABCL we call (java:jnew-runtime-class
"classname" :interfaces '("an interface name") :methods '(("method
name 1" "return type" ("first parameter type" ...) (lambda (this arg1
...) body)))):
(defun route (app method path handler)
(#"route"
app
method
path
(jss:new (java:jnew-runtime-class
(substitute #\$ #\/ (substitute #\$ #\- path))
:interfaces '("io.jooby.Route$Handler")
:methods `(
("apply" "java.lang.Object" ("io.jooby.Context")
(lambda (this ctx)
(funcall ,handler ctx))))))))
One thing to note is that when referring to a subclass within a file
we need to address it with the io.jooby.Route$Handler
syntax rather than as you might refer to it in Java
as io.jooby.Route.Handler. In the latter case ABCL
thinks Route is a package when in fact it's just a class.
If you run this now with abcl --load main.lisp. It will
work until you hit an endpoint. The problem is how Jooby tries to
figure out the real type of the returned object.
The app will crash somewhere around
here
calling analyzer.returnType(route.getHandle()).
In this case it tries to open and parse the (Java) source
code
of our application to try to find the return type for
this apply function.
That's a problem since our code isn't Java. Through trial and error I
realized we can trick Jooby/Java/somebody into figuring out the
correct return type by adding another implementation
of apply that returns a String to our class.
The full route code now looks like this:
(defun route (app method path handler)
(#"route"
app
method
path
(jss:new (java:jnew-runtime-class
(substitute #\$ #\/ (substitute #\$ #\- path))
:interfaces '("io.jooby.Route$Handler")
:methods `(
;; Need to define this one to make Jooby figure out the return type
;; Otherwise it tries to read "this file" which isn't a Java file so cannot be parsed
("apply" "java.lang.String" ("io.jooby.Context")
(lambda (this ctx) nil))
;; This one actually gets called
("apply" "java.lang.Object" ("io.jooby.Context")
(lambda (this ctx)
(funcall ,handler ctx))))))))
You may wonder, why keep the original method around? Well it's because
during reflection, ABCL says no such method that
returns String exists in the Handler
interface. That's fair I guess.
Implementing the template
The Java example on the Pebble homepage is perfect:
PebbleEngine engine = new PebbleEngine.Builder().build();
PebbleTemplate compiledTemplate = engine.getTemplate("home.html");
Map<String, Object> context = new HashMap<>();
context.put("name", "Mitchell");
Writer writer = new StringWriter();
compiledTemplate.evaluate(writer, context);
String output = writer.toString();
We can easily translate this into Lisp:
(defun hashmap (alist)
(let ((map (jss:new 'HashMap)))
(loop for el in alist
do (#"put" map (car el) (cadr el)))
map))
(defun template (filename context-alist)
(let* ((ctx (hashmap context-alist))
(path (java:jstatic "of" "java.nio.file.Path" filename))
(file (#"readString" 'java.nio.file.Files path))
(engine (#"build" (jss:new 'PebbleEngine$Builder)))
(compiledTmpl (#"getTemplate" engine filename))
(writer (jss:new 'java.io.StringWriter)))
(#"evaluate" compiledTmpl writer ctx)
(#"toString" writer)))
But if you run this abcl --load main.lisp and hit this
/search endpoint, it will blow up saying "no such method"
exists at the call to Path.of(filename).
After digging around I saw it was because Path.of is a variadic function.
And while there are examples
of using variadic functions
when the function only has a single parameter like
java.util.Arrays.asList(T ...), employing that same
technique here continued to result in "no such method":
(path (java:jstatic "of" "java.nio.file.Path" filename (jnew-array "java.lang.String" 0)))
Eventually I found an example of someone doing reflect/invoke on this kind of a function call and tried this logic on a local copy of the ABCL source code.
It worked. So I opened a pull request.
So the full working code for template is:
(defun template (filename context-alist)
(let* ((ctx (hashmap context-alist))
(path (java:jstatic "of" "java.nio.file.Path" filename (java:jnew-array "java.lang.String" 0)))
(file (#"readString" 'java.nio.file.Files path))
(engine (#"build" (jss:new 'PebbleEngine$Builder)))
(compiledTmpl (#"getTemplate" engine filename))
(writer (jss:new 'java.io.StringWriter)))
(#"evaluate" compiledTmpl writer ctx)
(#"toString" writer)))
And to get this diff running locally:
$ mkdir ~/vendor
$ cd ~/vendor
$ git clone https://github.com/eatonphil/abcl
$ cd abcl
$ git checkout pe/more-variadic
$ sudo {dnf/brew/apt} install ant maven
$ ant -f build.xml
And to run main.lisp using this diff:
$ ~/vendor/abcl/abcl --load main.lisp
And to hit the API:
$ curl localhost:8080/search
<html>
<title>Version 1.0.0</title>
<h2>cat</h2>
<h2>dog</h2>
<h2>mouse</h2>
</html>
$ curl localhost:8080/hello-world
Hello world!%
Phew! Easy peasy.
Next up
I'm porting this example to Kawa to see how it fares. Blog post to come.
In a ridiculous attempt to prove an internet wrong about the practicality of Lisp (Common Lisp specifically), I tried to get a simple (but realistic) web app running. After four days and a patch to ABCL I got something working.https://t.co/5UUWNR8Wnn pic.twitter.com/cZsx32IlKD
— Phil Eaton (@phil_eaton) August 5, 2021