If you've only vaguely heard of Lisp before or studied Scheme in school, Common Lisp is nothing like what you'd expect. While functional programming is all the rage in Scheme, Common Lisp was "expressly designed to be a real-world engineering language rather than a theoretically 'pure' language" (Practical Common Lisp). Furthermore, SBCL -- a popular implementation -- is a highly optimized compiler that is competitive with Java.
Building blocks
Common Lisp symbols, imagine "first-class" variables/labels, are encapsulated in namespaces called packages. However packages don't account for organization across directories, among other things. So while packages are a part of the core Common Lisp language, the "cross-directory" organizational structure is managed by the (all-but-standard) ASDF "systems". You can think of packages as roughly similar to modules in Python whereas systems in ASDF are more like packages in Python.
ASDF does not manage non-local dependencies. For that we use Quicklisp, the defacto package manager. ASDF should come bundled with your Common Lisp installation, which I'll assume is SBCL (not that it matters). Quicklisp does not come bundled.
Getting Quicklisp
You can follow the notes on the Quicklisp site for installation, but the basic gist is:
$ curl -O https://beta.quicklisp.org/quicklisp.lisp
$ sbcl --load quicklisp.lisp
...
* (quicklisp-quickstart:install)
...
* ^D
$ sbcl --load "~/quicklisp/setup.lisp"
...
* (ql:add-to-init-file)
A minimal package
Now we're ready to get started. Create a directory using the name of the library you'd like to package. For instance, I'll create a "cl-docker" directory for my Docker wrapper library. Then create a file using the same name in the directory with the ".asd" suffix:
$ cd ~/projects
$ mkdir cl-docker
$ touch cl-docker/cl-docker.asd
It is important for the ".asd" file to share the same name as the directory because ASDF will look for it in that location (by default).
Before we get too far into packaging, let's write a function we'd like to export from this library. Edit "cl-docker/docker.lisp" (this name does not matter) and add the following:
(defun ps ()
(let ((output (uiop:run-program '("docker" "ps") :output :string)))
(loop for line in (rest (cl-ppcre:split "(\\n+)" output))
collect (cl-ppcre:split "(\\s\\s+)" line))))
This uses a portable library, "uiop", that ASDF exposes by default (we won't need to explicitly import this anywhere because the package is managed by ASDF). It will run the command "docker ps" in a subprocess and return the output as a string. Then we use the regex split function from the "cl-ppcre" library to split the output first into lines, take all but the first line, and split the lines up based one two or more whitespace characters.
Next let's define the package (think module in Python) by editing "cl-docker/package.lisp" (this name also does not matter):
(defpackage cl-docker
(:use cl)
(:import-from :cl-ppcre :split)
(:export :ps))
Here we state the package's name, say that we want to import all Common Lisp base symbols into the package, say we want to import the "split" symbol from the "cl-ppcre" package, and say we only want to export our "ps" function.
At this point we must also declare within the "cl-docker/docker.lisp" file that it is a part of this package:
(in-package :cl-docker)
(defun ps ()
(let ((output (uiop:run-program '("docker" "ps") :output :string)))
(loop for line in (rest (cl-ppcre:split "(\\n+)" output))
collect (cl-ppcre:split "(\\s\\s+)" line))))
Next let's define the system (ASDF-level, similar to a package in Python) in "cl-docker/cl-docker.asd":
(defsystem :cl-docker
:depends-on (:cl-ppcre)
:serial t
:components ((:file "package")
(:file "docker")))
This defines all the pieces of the system for ASDF: the system name, the package definition and the component of the package ("cl-docker/docker.lisp"), and tells ASDF to make the "cl-ppcre" system on disk available to us. We also tell ASDF to process the components in the order we specified (otherwise it will pick an order that may not be what we want).
In preparation for times when we don't have the "cl-ppcre" system (or any other dependencies) on disk, we always load the system indirectly through Quicklisp (rather than directly via ASDF) so Quicklisp can fetch any missing dependencies from its repository of systems.
But before then -- unless you put this directory in "~/common-lisp" -- you'll need to register the directory containing the directory of your system definitions so ASDF (and Quicklisp) know where to look if you ask to load this system.
To do this, add a ".conf" file to "~/.config/common-lisp/source-registry.conf.d/" and add the following:
(:tree "~/path/to/dir/containing/system/dir")
So if you had a repo called "cl-docker" in your "~/projects" directory that contained the "cl-docker" directory we previously created (that, in turn, contains the "cl-docker.asd", "package.lisp", and "docker.lisp" files) then you might create "~/.config/common-lisp/source-registry.conf.d/1-cl-docker.conf" and add:
(:tree "~/projects/cl-docker")
Using the system
Now you can use the library from anywhere on your computer. Enter a Common Lisp REPL and tell Quicklisp to load the system (and download any non-local dependencies):
$ sbcl
...
* (ql:quickload "cl-docker")
To load "cl-docker":
Load 1 ASDF system:
cl-docker
; Loading "cl-docker"
..................................................
[package cl-docker]
("cl-docker")
* (cl-docker:ps)
And that's it!
For the complete source of this example package, check out this Gist.
In conclusion
Common Lisp is easy to work with, the packages are many and mature. Configuring an ASDF package is even simpler than configuring a Python "setup.py". I didn't demonstrate pinning versions of dependencies in ASDF, but of course you can do that too. If any of this -- as simple as it is -- seems tedious, you can also use Zach Beane's (creator of Quicklisp) quickproject tool to build out the structure for you.
Resources for Common Lisp
You must read Practical Common Lisp. It is freely available online. It is one of the best resources I keep referring to in dealing with simple issues (as a new Lisper, I stumble on a lot of simple issues).
Paul Graham's On Lisp is also a must-read when you want to get a better understanding of macros in Lisp. It will help you out with macros in Scheme too. This book is freely available online, but out of print physically. I sent Lulu the PDF and I received my physical copy for under $20 (including shipping).
I'm currently making my way through Common Lisp the Language, 2nd Edition which I believe is also freely available online. However I don't really recommend this unless you are interested in implementing Common Lisp or are dying to learn the standard library (not a bad idea).
Finally, Peter Norvig's Paradigms of Artificial Intelligence Programming just recently became freely available online. I haven't yet read it but I'm queuing it up. Don't let the title scare you, apparantly it is primarily considered a practical guide to Common Lisp around old-school/classical AI that isn't supposed to encumber.
It was pointed out on Twitter that Paul Graham's ANSI Common Lisp and the CLHS are probably better resources for the Common Lisp that exists today than Common Lisp the Language 2. CLtL2 is pre-standard.
Additionally, the Common Lisp Cookbook is a great resource for Common Lisp recipes. It's been around since 2004 (on Sourceforge) but has been pretty active recently and has been revived on Github pages.
On Scheme
I've done one or two unremarkable web prototypes in Chicken Scheme, an R5RS/R7RS Scheme implementation. I don't think Chicken Scheme is the best bet for the web (I'm mostly biased to this topic) because it has no native-thread support and there are lighter interpreters out there that are easier to embed (e.g. in nginx). Chicken Scheme's "niche" is being a generally high-quality implementation with a great collection of 3rd-party libraries, but it is also not the fastest Scheme you could choose.
I've worked on a larger web prototype -- a Github issue reporting app -- in Racket, a derivative of Scheme R6RS. And I've blogged favorably about Racket. It is a high-performance interpreter with a JIT compiler, has thread support, and is also well known for its collection of 3rd-party libaries. However the Racket ecosystem suffers from the same issues Haskell's does: libraries and bindings are primarily proof-of-concept only; missing documentation, tests and use. Trying to render "templatized" HTML (like Jinja allows for in Flask) without using S-exp-based syntax was a nightmare. (Read: there's space for someone to write a good string templating library.)
Sorry, Racket
Last point on Racket (because it really is worth looking into), debugging in that Github issue project was not fun. The backtraces were mostly useless. Naively I assume this may have to do with the way Racket optimizes and rewrites functions. I was often left with zero context to find and correct my errors. But it could very well be I was making poor use of Racket.
On the other hand
Common Lisp (its implementations and ecosystem) seems more robust and developed. SBCL, with it's great performance and native-thread support, is a promising candidate for backend web development.
I wrote a post on putting together a Common Lisp project. It's easy! I also included some of my favorite CL books and a digression on Scheme. https://t.co/2LEDoFnAjk #commonlisp #lisp #scheme
— Phil Eaton (@phil_eaton) March 7, 2018