A good line counting program has two features: it only counts non-empty lines to get a fair estimate of the size of a project, and it groups line counts by file type to help see immediately which languages are used.

A long time ago I got frustrated with two well known line counters. Sloccount spits out multiple strange Perl warnings about locales, and most of the output is a copyright notice and some absurd cost estimations. Cloc has fourteen Perl packages as dependencies. Writing a simple line counter is an interesting exercise; at the time I was discovering Common Lisp, so I wrote my own version.

I made a few changes years after years, but most of the code stayed the same. I thought it would be interesting to revisit this program and present it part by part as a demonstration of how you can use Common Lisp to solve a simple problem.

We are going to write the program bottom-up, starting with the smallest building blocks and progressively building upon them.

The program

The program is written in Common Lisp. The most convenient way of storing and executing it is a single executable file stored in a directory being part of the PATH environment variable. In my case, the script will be called locc, for “line of code counter”, and will be stored in the ~/bin directory.

We start the file with a shebang indicating how to execute the file. We use the SBCL implementation because it is stable and actively developed. It also makes it easy to execute a simple file:

#!/usr/bin/sbcl --script

Finding files

Our line counter will operate on directories, so it has to be able to list files in them. Path handling functions are very disconcerting at first. Common Lisp was designed a long time ago, and operating systems were different at the time. Let us dig in!

First let us write a simple function to check if a pathname object is a directory pathname:

(defun directory-path-p (path)
  "Return T if PATH is a directory or NIL else."
  (declare (type (or pathname string) path))
  (and (not (pathname-name path))
       (not (pathname-type path))))

Then we write a function to identify hidden files and directories since we do not want to include them:

(defun hidden-path-p (path)
  "Return T if PATH is a hidden file or directory or NIL else."
  (declare (type pathname path))
  (let ((name (if (directory-path-p path)
                  (car (last (pathname-directory path)))
                  (file-namestring path))))
    (and (plusp (length name))
         (eq (char name 0) #\.))))

As you can see we use DIRECTORY-PATH-P to extract the basename of the path, then check if it starts with a full stop (only if it is not empty of course).

Finally we can write the function to actually list files in a directory recursively:

(defun directory-path (path)
  "If PATH is a directory pathname, return it as it is. If it is a file
pathname or a string, transform it into a directory pathname."
  (declare (type (or pathname string) path))
  (if (directory-path-p path)
      path
      (make-pathname :directory (append (or (pathname-directory path)
                                            (list :relative))
                                        (list (file-namestring path)))
                     :name nil :type nil :defaults path)))

(defun find-files (path)
  "Return a list of all files contained in the directory at PATH or any of its
subdirectories."
  (declare (type pathname path))
  (flet ((list-directory (path)
           (directory
            (make-pathname :defaults (directory-path path)
                           :type :wild :name :wild))))
    (let ((paths nil)
          (children (list-directory (directory-path path))))
      (dolist (child children paths)
        (unless (hidden-path-p child)
          (if (directory-path-p child)
              (setf paths (append paths (find-files child)))
              (push child paths)))))))

We use the DIRECTORY standard function with a path containing a wildcard component to list the files in a directory, and do so recursively.

Counting lines

Now that we have files, we can start counting lines. Let us first write a function to count the number of non-empty lines in a file.

(defun count-file-lines (path)
  "Count the number of non-empty lines in the file at PATH. A line is empty if
it only contains space or tabulation characters."
  (declare (type pathname path))
  (with-open-file (stream path :element-type '(unsigned-byte 8))
    (do ((nb-lines 0)
         (blank-line t))
        (nil)
      (let ((octet (read-byte stream nil)))
        (cond
          ((or (null octet) (eq octet #.(char-code #\Newline)))
           (unless blank-line
             (incf nb-lines))
           (when (null octet)
             (return-from count-file-lines nb-lines))
           (setf blank-line t))
          ((and (/= octet #.(char-code #\Space))
                (/= octet #.(char-code #\Tab)))
           (setf blank-line nil)))))))

We open the file to obtain a steam of octets, and read it octet by octet, keeping track of whether the current line is blank or not. Note how we make sure to count the last line even if it does not end with a newline character.

Reading a file one octet could be a disaster for performances. Fortunately SBCL file streams are buffered, something we can easily check by running our program with strace -e trace=openat,read. We would not rely on this property if we wanted our program to work on multiple Common Lisp implementations, but this is a non issue here.

Identifying the file type

Counting lines is one thing, but we need to identify their content. The simplest way is to do so based on the file extension.

Obviously we will want to ignore various files which are known not to contain text content, so we start by building a hash table containing these extensions:

(defparameter *ignored-extensions*
  (let ((extensions '("a" "bin" "bmp" "cab" "db" "elc" "exe" "gif" "gz"
                      "jar" "jpeg" "jpg" "o" "pcap" "pdf" "png" "ps" "rar"
                      "svg" "tar" "tgz" "tiff" "zip"))
        (table (make-hash-table :test 'equal)))
    (dolist (extension extensions table)
      (setf (gethash extension table) t)))
  "A hash table containing all file extensions to ignore.")

We then create another hash table to associate a type symbol to each known file extension:

(defparameter *extension-types*
  (let ((pairs '(("asm" . assembly) ("s" . assembly)
                 ("adoc" . asciidoc)
                 ("awk" . awk)
                 ("h" . c) ("c" . c)
                 ("hpp" . cpp) ("cpp" . cpp) ("cc" . cpp)
                 ("css" . css)
                 ("el" . elisp)
                 ("erl" . erlang)
                 ("go" . go)
                 ("html" . html) ("htm" . html)
                 ("ini" . ini)
                 ("hs" . haskell)
                 ("java" . java)
                 ("js" . javascript)
                 ("json" . json)
                 ("tex" . latex)
                 ("lex" . lex)
                 ("lisp" . lisp)
                 ("mkd" . markdown) ("md" . markdown)
                 ("rb" . ruby)
                 ("pl" . perl) ("pm" . perl)
                 ("php" . php)
                 ("py" . python)
                 ("sed" . sed)
                 ("sh" . shell) ("bash" . shell) ("csh" . shell)
                 ("zsh" . shell) ("ksh" . shell)
                 ("scm" . scheme)
                 ("sgml" . sgml)
                 ("sql" . sql)
                 ("texi" . texinfo)
                 ("texinfo" . texinfo)
                 ("vim" . vim)
                 ("xml" . xml) ("dtd" . xml) ("xsd" . xml)
                 ("yaml" . yaml) ("yml" . yaml)
                 ("y" . yacc)))
        (table (make-hash-table :test 'equal)))
    (dolist (pair pairs table)
      (setf (gethash (car pair) table) (cdr pair))))
  "A hash table containing a symbol identifying the type of a file for
  each known file extension.")

With these hash tables, the function identifying the type of a file is trivial:

(defun identify-file-type (path)
  "Return a symbol identifying the type of the file at PATH, or UNKNOWN if the
file extension is not known."
  (declare (type pathname path))
  (let ((extension (pathname-type path)))
    (unless (gethash extension *ignored-extensions*)
      (gethash extension *extension-types* 'unknown))))

Collecting file information

Up to this point, we wrote several functions without connecting them. But we now have all the building blocks we need. Let us use them to accumulate information about the files in a list of directories.

(defun collect-line-counts (directory-paths)
  "Collect the line count of all files in the directories located at one of the
paths in DIRECTORY-PATHS and return them grouped by file type as an
association list."
  (declare (type list directory-paths))
  (let ((line-counts (make-hash-table)))
    (dolist (directory-path directory-paths)
      (dolist (path (find-files directory-path))
        (handler-case
            (let ((type (identify-file-type path)))
              (when (and type (not (eq type 'unknown)))
                (let ((nb-lines (count-file-lines path)))
                  (incf (gethash type line-counts 0) nb-lines))))
          (error (condition)
            (format *error-output* "~&error while reading ~A: ~A~%"
                    path condition)))))
    (let ((line-count-list nil))
      (maphash (lambda (type nb-lines)
                 (push (cons type nb-lines) line-count-list))
               line-counts)
      line-count-list)))

We get to use all previous functions. We iterate through directory paths to find non-hidden files using FIND-FILES, then we use IDENTIFY-FILE-TYPE to obtain a type symbol and COUNT-FILE-LINES to count the number of non-empty lines in each file. Results are accumulated by file type in the LINE-COUNTS hash table. During this process, we handle errors that may occur while reading files with a message on the error output. Finally we transform the hash table into an association list and return it.

Presenting results

Executing all these functions in the Lisp REPL is quite practical during developement, but the default pretty printer is not really what you expect for the final command line tool:

So let us write a function to format this association list:

(defun format-line-counts (line-counts &key (stream *standard-output*))
  "Format the line counts in the LINE-COUNTS association list to STREAM."
  (declare (type list line-counts)
           (type stream stream))
  (dolist (entry (sort line-counts '> :key 'cdr))
    (let ((type (car entry))
          (nb-lines (cdr entry)))
      (format stream "~12A  ~8@A~%"
              (string-downcase (symbol-name type)) nb-lines))))

We print the list sorted in descending order, meaning that the file type with the most number of lines comes first. Of course the output is padded to make sure numbers are all aligned.

Finalizing the program

The only thing left to do is the entry point of the script. This is the only place where we need to call a non-standard function in order to access command line arguments. If no command line argument were passed to the script, we look for files in the current directory.

(let ((paths (or (cdr sb-ext:*posix-argv*) '("."))))
  (format-line-counts
   (collect-line-counts paths)))

Much better!

There does not seem to be any performance issue: on the Linux kernel source tree, this program is almost 11 times faster than sloccount. It would be interesting to profile the program to make sure IO is the bottleneck and improve inefficient parts, but this code is fast enough for my needs.

As you can see, it is not that hard to use Common Lisp to solve problems.

Hello,

It has been a long time since I posted something here.  I have been busy with my day job and bogged down in a major rewrite of something (more on this hopefully very soon now (tm)) that is full of rabbit's holes.

I was able to get out of one of these rabbit's holes with this little hack I cooked up that allows you to, possibly, write more concise code.

This little hack introduces two handy Common Lisp macros, LETV and LETV* that allow you to mix regular LET and MULTIPLE-VALUE-BIND forms in a less verbose way. The amount of indentation needed is also reduced.

The syntax of LETV (or LETV*) is very "loopy", with a nod to SML/OCaml/F#/Haskell/Julia. The current syntax is the following:

    letv     ::= 'LETV' [vars] [IN [body]]
    letvstar ::= 'LETV*' [vars] [IN [body]]

    vars     ::= var | var vars

    var      ::= ids '=' <form> [decls]

    decls    ::= decl | decl decls
    decl     ::= OF-TYPE idtypes

    ids      ::= <symbol> | '(' <symbol> + ')'
    idtypes  ::= <type designator> | '(' <type designator> + ')'

    body     ::= [<declarations>] <form> * 

(I know: the grammar is not completely kosher, but I trust you will understand it.)

The two macros expand in forms that mimic the semantics of let, LET* and MULTIPLE-VALUE-BIND. All type declarations, if present, are properly handled via locally forms. LETV expands into a LET, with variable initialized by PSETQs and multiple-VALUE-SETQs. LETV* expands into a form that is an interleaving of LET* and MULTIPLE-VALUE-BIND.

The library exports only the two symbols LETV and LETV*. The other "symbols" mentioned (=, IN, OF-TYPE) are checked as in LOOP; therefore you can use different styles to write your code, as you would when writing LOOPs.

The library is available here.

Examples

1. Simple case:

     (letv x = 42 in (format t "The answer is ~D.~%" x))
     

2. Same with declaraion:

     (letv x = 42 of-type fixnum
   	in (format t "The answer is ~D." x))
     

3. Simple case with MULTIPLE-VALUE-BIND:

     (letv (v found) = (gethash 'the-key *my-hash-table*)
           in (if found
                  (format t "Found THE-KEY, doing stuff.~%")
                  (error "THE-KEY not found.")))
     

4. Mixing things up:

     (letv (v found) = (gethash 'the-key *my-hash-table*)
                     of-type (fixnum boolean)
            x = 42
            in (if found
                    (format t "Found THE-KEY, adding to the answer ~D.~%"
                            (+ v x))
                    (error "THE-KEY not found.")))
     

5. With LETV*:

     (letv* (v found) = (gethash 'the-key *my-hash-table*)
                      of-type (fixnum boolean)
            x = (when found (+ v 42))
            in (if found
                   (format t "Found THE-KEY, adding to the answer ~D.~%"
   			 	        x)
                   (error "THE-KEY not found.")))
     

6. The other way around.

     (letv x = 42 of-type integer
           (v found) = (gethash 'the-key *my-hash-table*)
                     of-type (fixnum boolean)
           in (if found
                  (format t "Found THE-KEY, adding to the answer ~D.~%"
                          (+ v x))
                  (error "THE-KEY not found.")))
     

7. A more compelling example.

     (letv* (p found) = (gethash 'the-key *points-table*) of-type (point)
            (x y) = (if found
                        (unpack-point p)
                        (values :missing :missing))
            in
            (declare (type point p)
                     (type real x y))  ; Adding declarations here also works.
   
            (do-stuff-with x y)
            (do-things-with p))
     

All the examples are meant to illustrate the use of LETV and LETV*.

Notes

LETV and LETV* are obviously not the first macros of this kind floating around: others are available and all have their niceties. But I never claimed not to suffer from NIH syndrome.

LETV and LETV* do not do "destructuring" or "pattern matching". That is a different can of worms; but you can check cl-unification for a library (always by yours truly) that provides facilities in that sense.

'(Cheers)

The first major update for Kandria is now live on all platforms! It includes the level editor, a modding system, some new sample levels, and bugfixes!

Level Editor

The level editor received a big overhaul and is now a lot more accessible. There's also official documentation now to help get you started and explain all the tools and shortcuts available. If you want to dig in, just update your game and navigate to Mod Manager > Create Mod, which should place you directly into the editor in a new world of your own!

Sharing Mods

You can play levels made by others and download them directly from within the game. To browse existing levels, simply go to Mod Manager > Discover. From there you can select and install mods. Once installed, you can play their world under the Worlds tab.

We've provided a number of sample levels for you to play and edit. Open them up and see how they work, maybe you'll get inspired to make your own!

Join in on our Discord community event! We'll raffle out 20 Steam keys for Kandria to anyone that uploads a new level. You can freely give the key out to a friend so they can play the level you made!

Changelog

Here's the detailed changelog for this version:

• Fix issues with multiple worlds corrupting save files
  Reported by Dieting Hippo

• Fix issue with animation editor crashing the game
  Reported by Jeremiah Fieldhaven

• Fix issue with history dialog crashing the game
  Reported by Dieting Hippo

• Fix issue with placing lava in an empty world crashing the game
  Reported by Hempuli

• Fix issue with floating doors crashing the game on navmesh construction
  Reported by Hempuli

• Fix issue with world field edits leading to a crash on editor exit
  Reported by Hempuli

• Fix issue with text alignment on mixed fonts like Nicokaku
  Reported by Pir

• Add double-click tile selection
  Reported by Hempuli

• Make Ctrl+Z work, not just Z
  Reported by Dieting Hippo

• Select entities on mouse down rather than up
  Reported by Hempuli

• Fix rectangle tool being off by one at edges
  Reported by Hempuli

• Fix roaming NPCs hard-setting their velocity, preventing bounce and other velocity based mechanics
  Reported by Hempuli

• Hide the bsize field from edits as its duplicity with size is confusing
  Reported by Hempuli

• Don't overwrite the keymap.lisp if the shipped one is never, merge them instead.
  Reported by Hempuli

• Fix multi-tile picking being screwed when not selecting tiles from bottom left to top right
  Reported by Hempuli, Dieting Hippo

• Fix moving platforms moving faster when climbing them

• Fix default mod preview image size
  Reported by Jeremiah Fieldhaven

• Fix various minor usability issues
  Reported by Jeremiah Fieldhaven

• Fix issue related to spawning of child entities in new worlds
  Reported by Tim White

• Fix input field for spawner types
  Reported by Dieting Hippo

• Fix mod dependency management
  Reported by Jeremiah Fieldhaven

• Fix issues with the history dialog
  Reported by Dieting Hippo

• Fix issues with the animation editor
  Reported by Jeremiah Fieldhaven

• Fix issue related to empty GI
  Reported by Goldberg

There's a bunch more changes that were done internally. If you would like to see that kind of detail, you can check the source code repository.

Let's get the important news out of the way: the next major update for Kandria will launch on Wednesday, March 8th, at 15:00 CET. The update will include a number of improvements, new features, more content, and a community event!

Level Editing

The biggest part of the update is, no doubt, the polished level editor. The editor has always been a part of Kandria since release, but now it's nicely polished and much easier to use!

I've also included documentation to make it much easier for you to get a handle on how the editor works and how to get started with using it. I'm very excited to see what people can come up with!

And yes, the entire Kandria world was created in this editor, so it is fully capable of creating some very complex levels!

Partial Modding Support

But, the editor itself is not all by far. Not only can you create levels, you can also browse levels others have made easily directly from within the game, and upload your own as well! We've integrated with the great mod.io service to make levels and mods available to everyone, regardless of how they purchased Kandria.

Worlds are distributed as just another kind of mod, so the UI for browsing and managing mods has also been developed already. And, if you're very adventurous you can even get started with code mods for Kandria, too.

More Content

In order to give you some idea of what you can do with the editor, and to give more casual players something new to nibble on, too, I've also included a bunch small, new worlds as well.

You can simply download those through the mod manager's discovery tab when the update is out!

Community Event: Dare to Share

Finally, we will be holding a community content event: we will raffle out 20 Steam keys for Kandria to anyone that uploads a new level. You can freely give the key out to a friend so they can play the level you made!

If you're interested in participating, please join our Discord. We'll be sharing more details on how exactly the raffle will work closer to the release of the modding support.

Speedruns

Kandria will be the featured game for Speed Bump this month! A couple of speedrunners will be taking a closer look at the game and competing for time. Some runs have already started to appear on our leaderboard, as well! As a speedrunning fan and amateur runner myself, I'm really excited to see what kinds of tech they can figure out and how far they can optimise the run. The time has already come down quite considerably from my initial estimates, and I can definitely see things becoming quite competitive in the future.

Closing Thoughts

I'll have more news about the future development of Kandria for you next month. Suffice to say, we're not done yet! For now, just remember that the update will launch on Wednesday, March 8th, at 15:00 CET along with a 15% discount on Steam. Please continue to share the page with your friends and communities!

Everyone who has written Lisp has written tiny Lisp evaluators in Lisp: here are two more.

Following two recent articles I wrote on scope and extent in Common Lisp, I thought I would finish with two very tiny evaluators for dynamically and lexically bound variants on a tiny Lisp.

The language

The tiny Lisp these evaluators interpret is not minimal: it has constructs other than lambda, and even has assignment. But it is pretty small. Other than the binding rules the languages are identical.

• λ & lambda are synonyms and construct procedures, which can take any number of arguments;
• quote quotes its argument;
• if is conditional expression (the else part is optional);
• set! is assignment and mutates a binding.

That is all that exists.

Both evaluators understand primitives, which are usually just functions in the underlying Lisp: since the languages are Lisp–1s, you could also expose other sorts of things of course (for instance true and false values). You can provide a list of initial bindings to them to define useful primitives.

Requirements

Both evaluators rely on my iterate and spam hacks: they could easily be rewritten not to do so.

The dynamic evaluator

A procedure is represented by a structure which has a list of formals and a body of one or more forms.

(defstruct (procedure
            (:print-function
             (lambda (p s d)
               (declare (ignore d))
               (print-unreadable-object (p s)
                 (format s "λ ~S" (procedure-formals p))))))
  (formals '())
  (body '()))

The evaluator simply dispatches on the type of thing and then on the operator for compound forms.

(defun evaluate (thing bindings)
  (typecase thing
    (symbol
     (let ((found (assoc thing bindings)))
       (unless found
         (error "~S unbound" thing))
       (cdr found)))
    (list
     (destructuring-bind (op . arguments) thing
       (case op
         ((lambda λ)
          (matching arguments
            ((head-matches (list-of #'symbolp))
             (make-procedure :formals (first arguments)
                             :body (rest arguments)))
            (otherwise
             (error "bad lambda form ~S" thing))))
         ((quote)
          (matching arguments
            ((list-matches (any))
             (first arguments))
            (otherwise
             (error "bad quote form ~S" thing))))
         ((if)
          (matching arguments
            ((list-matches (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)))
            ((list-matches (any) (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)
               (evaluate (third arguments) bindings)))
            (otherwise
             (error "bad if form ~S" thing))))
         ((set!)
          (matching arguments
            ((list-matches #'symbolp (any))
             (let ((found (assoc (first arguments) bindings)))
               (unless found
                 (error "~S unbound" (first arguments)))
               (setf (cdr found) (evaluate (second arguments) bindings))))
            (otherwise
             (error "bad set! form ~S" thing))))
         (t
          (applicate (evaluate (first thing) bindings)
                     (mapcar (lambda (form)
                               (evaluate form bindings))
                             (rest thing))
                     bindings)))))
    (t thing)))

The interesting thing here is that applicate needs to know the current set of bindings so it can extend them dynamically.

Here is applicate which has a case for primitives and procedures

(defun applicate (thing arguments bindings)
  (etypecase thing
    (function
     ;; a primitive
     (apply thing arguments))
    (procedure
     (iterate bind ((vtail (procedure-formals thing))
                    (atail arguments)
                    (extended-bindings bindings))
       (cond
        ((and (null vtail) (null atail))
         (iterate eval-body ((btail (procedure-body thing)))
           (if (null (rest btail))
               (evaluate (first btail) extended-bindings)
             (progn
               (evaluate (first btail) extended-bindings)
               (eval-body (rest btail))))))
        ((null vtail)
         (error "too many arguments"))
        ((null atail)
         (error "not enough arguments"))
        (t
         (bind (rest vtail)
               (rest atail)
               (acons (first vtail) (first atail)
                      extended-bindings))))))))

The thing that makes this evaluator dynamic is that the bindings that applicate extends are those it was given: procedures do not remember bindings.

The lexical evaluator

A procedure is represented by a structure as before, but this time it has a set of bindings associated with it: the bindings in place when it was created.

(defstruct (procedure
            (:print-function
             (lambda (p s d)
               (declare (ignore d))
               (print-unreadable-object (p s)
                 (format s "λ ~S" (procedure-formals p))))))
  (formals '())
  (body '())
  (bindings '()))

The evaluator is almost identical:

(defun evaluate (thing bindings)
  (typecase thing
    (symbol
     (let ((found (assoc thing bindings)))
       (unless found
         (error "~S unbound" thing))
       (cdr found)))
    (list
     (destructuring-bind (op . arguments) thing
       (case op
         ((lambda λ)
          (matching arguments
            ((head-matches (list-of #'symbolp))
             (make-procedure :formals (first arguments)
                             :body (rest arguments)
                             :bindings bindings))
            (otherwise
             (error "bad lambda form ~S" thing))))
         ((quote)
          (matching arguments
            ((list-matches (any))
             (first arguments))
            (otherwise
             (error "bad quote form ~S" thing))))
         ((if)
          (matching arguments
            ((list-matches (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)))
            ((list-matches (any) (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)
               (evaluate (third arguments) bindings)))
            (otherwise
             (error "bad if form ~S" thing))))
         ((set!)
          (matching arguments
            ((list-matches #'symbolp (any))
             (let ((found (assoc (first arguments) bindings)))
               (unless found
                 (error "~S unbound" (first arguments)))
               (setf (cdr found) (evaluate (second arguments) bindings))))
            (otherwise
             (error "bad set! form ~S" thing))))
         (t
          (applicate (evaluate (first thing) bindings)
                     (mapcar (lambda (form)
                               (evaluate form bindings))
                             (rest thing)))))))
    (t thing)))

The differences are that when constructing a procedure the current bindings are recorded in the procedure, and it is no longer necessary to pass bindings to applicate.

applicate is also almost identical:

(defun applicate (thing arguments)
  (etypecase thing
    (function
     ;; a primitive
     (apply thing arguments))
    (procedure
     (iterate bind ((vtail (procedure-formals thing))
                    (atail arguments)
                    (extended-bindings (procedure-bindings thing)))
       (cond
        ((and (null vtail) (null atail))
         (iterate eval-body ((btail (procedure-body thing)))
           (if (null (rest btail))
               (evaluate (first btail) extended-bindings)
             (progn
               (evaluate (first btail) extended-bindings)
               (eval-body (rest btail))))))
        ((null vtail)
         (error "too many arguments"))
        ((null atail)
         (error "not enough arguments"))
        (t
         (bind (rest vtail)
               (rest atail)
               (acons (first vtail) (first atail)
                      extended-bindings))))))))

The difference is that the bindings it extends when binding arguments are the bindings which the procedure remembered, not the dynamically-current bindings, which it does not even know.

The difference between them

Here is the example that shows how these two evaluators differ.

With the dynamic evaluator:

? ((λ (f)
     ((λ (x)
        ;; bind x to 1 around the call to f
        (f))
      1))
   ((λ (x)
      ;; bind x to 2 when the function that will be f is created
      (λ () x))
    2))
1

The binding in effect is the dynamically current one, not the one that was in effect when the procedure was created.

With the lexical evaluator:

? ((λ (f)
     ((λ (x)
        ;; bind x to 1 around the call to f
        (f))
      1))
   ((λ (x)
      ;; bind x to 2 when the function that will be f is created
      (λ () x))
    2))
2

Now the binding in effect is the one that existed when the procedure was created.

Something more interesting is how you create recursive procedures in the lexical evaluator. With suitable bindings for primitives, it’s easy to see that this can’t work:

((λ (length)
   (length '(1 2 3)))
 (λ (l)
   (if (null? l)
       0
       (+ (length (cdr l)) 1))))

It can’t work because length is not in scope in the body of length. it will work in the dynamic evaluator.

The first fix, which is similar to what Scheme does with letrec, is to use assignment to mutate the binding so it is correct:

((λ (length)
   (set! length (λ (l)
                  (if (null? l)
                      0
                      (+ (length (cdr l)) 1))))
   (length '(1 2 3)))
 0)

Note the initial value of length is never used.

The second fix is to use something like the U combinator (you could use Y of course: I think U is simpler to understand):

((λ (length)
   (length '(1 2 3)))
 (λ (l)
   ((λ (c)
      (c c l 0))
    (λ (c t s)
      (if (null? t)
          s
          (c c (cdr t) (+ s 1)))))))

Source code

These two evaluators, together with a rudimentary REPL which can use either of them, can be found here.

In Common Lisp, dynamic bindings and lexical bindings live in the same namespace. They don’t have to.

Common Lisp has two sorts of bindings for variables: lexical binding and dynamic binding. Lexical binding has lexical scope — the binding is available where it is visible in source code — and indefinite extent — the binding is available as long as any code might reference it. Dynamic binding has indefinite scope — the binding is available to any code which runs between when the binding is established and when control leaves the form which established it — and dynamic extent — the binding ceases to exist when control leaves the binding form.

These are really two very different things. However CL places both of these kinds of bindings into the same namespace, relying on special declarations and proclamations to tell the system which sort of binding to create and reference for a given name.

That doesn’t have to be the case: it’s possible in CL to completely isolate these two namespaces from each other. This means you could write code where all variable references were to lexical bindings and where dynamic bindings were created and referenced by a completely different set of operators. Here is an example of that. Following practice in some old Lisps I will call this ‘fluid’ binding. I will also use / to delimit the names of fluid variables simply to distinguish them from normal variables.

(defun inner (varname value)
  (setf (fluid-value varname) value))

(defun outer (varname value)
  (call/fluid-bindings
   (lambda ()
     (values
      (fluid-value varname)
      (progn
        (inner varname (1+ value))
        (fluid-value varname))))
   (list varname)
   (list value)))

And now

> (outer '/v/ 1)
1
2

Here are a set of operators for dealing with these fluid variables:

fluid-value accesses the value of a fluid variable.

fluid-boundp tells you if a name is bound as a fluid variable.

call/fluid-bindings calls a function with one or more fluid variables bound.

define-fluid (not used above) defines a global value for a fluid variable.

Well, of course you can do something like this using an explicit binding stack and a single special variable to hang it from. But that’s not how this works: these ‘fluid variables’ are just CL’s dynamic variables:

(defun call/print-base (f base)
  (call/fluid-bindings  f '(*print-base*) (list base)))

> (call/print-base
   (lambda ()
     *print-base*)
   2)
2

So how does this work? Well fluid-value and fluid-boundp are obvious:

(defun fluid-value (s)
  (symbol-value s))

(defun (setf fluid-value) (n s)
  (setf (symbol-value s) n))

(defun fluid-boundp (s)
  (boundp s))

And the trick now is that CL gives you enough mechanism to bind named dynamic variables yourself, that mechanism being progv, which

[…] allows binding one or more dynamic variables whose names may be determined at run time […]

So now call/fluid-bindings just uses progv:

(defun call/fluid-bindings (f fluids values)
  (progv fluids values (funcall f)))

And finally define-fluid looks like this:

(defmacro define-fluid (var &optional (value nil)
                            (doc nil docp))
  `(progn
     (setf (fluid-value ',var) ,value)
     ,@(if docp
           `((setf (documentation ',var 'variable) ',doc))
         '())
     ',var))

The interesting thing here is that there are no special declarations or proclamations: you can create and bind new fluid variables without any recourse to special at all, in a way which is completely compatible with the existing dynamic variables, because fluid variables are dynamic variables.

So one way of thinking about special is that it is a declaration that says ‘for this variable name, access the namespace of dynamic bindings rather than lexical bindings’. This is not really what special was of course in Lisps before CL — it was historically closer to an instruction to use the interpreter’s variable binding mechanism in compiled code — but you can think of it this way in CL, where the interpreter and compiler do not have separate binding rules.

And, of course, using something like the above, you could write code in CL where all variable bindings were lexical and dynamic variables lived entirely in their own namespace. For instance this works fine:

(defun f ()
  (let ((x 2))
    (call/fluid-bindings
     (lambda ()
       (values x (fluid-value 'x)))
     '(x) '(3))))

> (f)
2
3

The reference to x as a variable refers to its lexical binding, while (fluid-value 'x) refers to its dynamic binding.

Whether writing code like that would be useful I am not sure: I think that the *-convention for dynamic variables is perfectly fine in fact. But it is perhaps interesting to see that you can think of dynamic bindings in CL this way.

Font Lock is the builtin Emacs minor mode used to highlight textual elements in buffers. Major modes usually configure it to detect various syntaxic constructions and attach faces to them.

The reason I ended up deep into Font Lock is because I was not satisfied with the way it is configured for lisp-mode, the major mode used for both Common Lisp and Emacs Lisp code. This forced me to get acquainted with various aspects of Font Lock in order to change its configuration. If you want to change highlighting for your favourite major mode, you will find this article useful.

Common Lisp highlighting done wrong

The core issue of Common Lisp highlighting in Emacs is that a lot of it is arbitrary and inconsistent:

• The mode highlights what it calls “definers” and “keywords”, but it does not really make sense in Common Lisp. Why would WITH-OUTPUT-TO-STRING be listed as a keyword, but not CLASS-OF?
• SIGNAL uses font-lock-warning-face. Why would it be a warning? Even stranger, why would you use this warning face for CHECK-TYPE?
• Keywords and uninterned symbols are all highlighted with font-lock-builtin-face. But they are not functions or variables. They are not even special in any way, and their syntax already indicates clearly their nature. Having so many yellow symbols everywhere is really distracting.
• All symbols starting with & are highlighted using font-lock-type-face. But lambda list arguments are not types, and symbols starting with & are not always lambda list arguments.
• All symbols preceded by ( whose name starts with DO- or WITH- are highlighted as keywords. There is even a comment by RMS stating that it is too general. He is right.

Beyond these issues, the mode sadly uses default Font Lock faces instead of defining semantically appropriate faces and mapping them to existing ones as default values.

The chances of successfully driving this kind of large and disruptive change directly into Emacs are incredibly low. Even if it was to be accepted, the result would not be available until the next release, which could mean months. Fortunately, Emacs is incredibly flexible and we can change all of this ourselves.

Note that you may not agree with the list of issues above, and this is fine. The point of this article is to show you how you can change the way Emacs highlights content in order to match your preferences. And you can do that for all major modes!

Font Lock configuration

Font Lock always felt a bit magic and it took me some time to find the motivation to read the documentation. As is turned out, it can be used for very complex highlighting schemes, but basic features are not that hard to use.

The main configuration of Font Lock is stored in the font-lock-defaults buffer-local variable. It is a simple list containing the following entries:

• A list of symbols containing the value to use for font-lock-keywords at each level, the first symbol being the default value.
• The value used for font-lock-keywords-only. If it is nil, it enables syntaxic highlighting (strings and comments) in addition of search-based (keywords) highlighting.
• The value used for font-lock-keywords-case-fold-search. If true, highlighting is case insensitive.
• The value used for font-lock-syntax-table, the association list controlling syntaxic highlighting. If it is nil, Font Lock uses the syntax table configured with set-syntax-table. In lisp-mode this would mean lisp-mode-syntax-table.
• All remaining values are bindings using the form (VARIABLE-NAME . VALUE) used to set buffer-local values for other Font Lock variables.

The part we are interested about is search-based highlighting which uses regular expressions to find specific text fragments and attach faces to them.

Values used for font-lock-keywords are also lists. Each element is a construct used to specify one or more keywords to highlight. While these constructs can have multiple forms for more complex use cases, we will only use the two simplest ones:

• (REGEXP . FACE) tells Font Lock to use FACE for text fragments which match REGEXP. For example, you could use ("\\_<-?[0-9]+\\_>" . font-lock-constant-face) to highlight integers as constants (note the use of \_< and \_> to match the start and end of a symbol; see the regexp documentation for more information).
• (REGEXP (GROUP FACE)...) is a bit more advanced. When REGEXP matches a subset of the buffer, Font Lock assigns faces to the capture group identified by their number. You could use this construction to detect a complex syntaxic element and highlight some of its parts with different faces.

Simplified Common Lisp highlighting

We are going to configure keyword highlighting for the following types of values:

• Character literals, e.g. #\Space.
• Function names in the context of a function call for standard Common lisp functions.
• Standard Common Lisp values such as *STANDARD-OUTPUT* or PI.

Additionally, we want to keep the default syntaxic highlighting configuration which recognizes character strings, documentation strings and comments.

Faces

Let us start by defining new faces for the different values we are going to match:

(defface g-cl-character-face
  '((default :inherit font-lock-constant-face))
  "The face used to highlight Common Lisp character literals.")

(defface g-cl-standard-function-face
  '((default :inherit font-lock-keyword-face))
  "The face used to highlight standard Common Lisp function symbols.")

(defface g-cl-standard-value-face
  '((default :inherit font-lock-variable-name-face))
  "The face used to highlight standard Common Lisp value symbols.")

Nothing complicated here, we simply inherit from default Font Lock faces. You can then configure these faces in your color theme without affecting other modes using Font Lock.

Keywords

To detect standard Common Lisp functions and values, we are going to need a regular expression. The first step is to build a list of strings for both functions and values. Easy to do with a bit of Common Lisp code!

(defun standard-symbol-names (predicate)
  (let ((symbols nil))
    (do-external-symbols (symbol :common-lisp)
      (when (funcall predicate symbol)
        (push (string-downcase (symbol-name symbol)) symbols)))
    (sort symbols #'string<)))
    
(standard-symbol-names #'fboundp)
(standard-symbol-names #'boundp)

The STANDARD-SYMBOL-NAMES build a list of symbols exported from the :COMMON-LISP package which satisfy a predicate. The first call gives us the name of all symbols bound to a function, and the second all which are bound to a value.

The astute reader will immediately wonder about symbols which are bound both a function and a value. They are easy to find by calling INTERSECTION on both sets of names: +, /, *, -. It is not really a problem: we can highlight function calls by matching function names preceded by (, making sure that these symbols will be correctly identified as either function symbols or value symbols depending on the context.

We store these lists of strings in the g-cl-function-names and g-cl-value-names (the associated code is not reproduced here: these lists are quite long; but I posted them as a Gist).

With this lists, we can use the regexp-opt Emacs Lisp function to build optimized regular expressions matching them:

(defvar g-cl-font-lock-keywords
  (let* ((character-re (concat "#\\\\" lisp-mode-symbol-regexp "\\_>"))
         (function-re (concat "(" (regexp-opt g-cl-function-names t) "\\_>"))
         (value-re (regexp-opt g-cl-value-names 'symbols)))
    `((,character-re . 'g-cl-character-face)
      (,function-re
       (1 'g-cl-standard-function-face))
      (,value-re . 'g-cl-standard-value-face))))

Characters literals are reasonably easy to match.

Functions are a bit more complicated since we want to match the function name when it is preceded by an opening parenthesis. We use a capture capture (see the last argument of regexp-opt) for the function name and highlight it separately.

Values are always matched as full symbols: we do not want to highlight parts of a symbol, for example MAP in a symbol named MAPPING.

Final configuration

Finally we can define the variable which will be used for font-lock-defaults in the initialization hook; we copy the original value from lisp-mode, and change the keyword list for what is going to be our own configuration:

(defvar g-cl-font-lock-defaults
  '((g-cl-font-lock-keywords)
    nil                                 ; enable syntaxic highlighting
    t                                   ; case insensitive highlighting
    nil                                 ; use the lisp-mode syntax table
    (font-lock-mark-block-function . mark-defun)
    (font-lock-extra-managed-props help-echo)
	(font-lock-syntactic-face-function
	 . lisp-font-lock-syntactic-face-function)))

To configure font-lock-defaults, we simply set it in the initialization hook of lisp-mode:

(defun g-init-lisp-font-lock ()
  (setq font-lock-defaults g-cl-font-lock-defaults))
  
(add-hook 'lisp-mode-hook 'g-init-lisp-font-lock)

Comparison

Let us compare highlighting for a fragment of code before and after our changes:

The differences are subtle but important:

• All standard functions are highlighted, helping to distinguish them from user-defined functions.
• Standard values such as *ERROR-OUTPUT* are highlighted.
• Character literals are highlighted the same way as character strings.
• Keywords are not highlighted anymore, avoiding the confusion with function names.

Conclusion

That was not easy; but as always, the effort of going through the documentation and experimenting with different Emacs components was very rewarding. Font Lock does not feel like a black box anymore, opening the road for the customization of other major modes.

In the future, I will work on a custom color scheme to use more subtle colors, with the hope of reducing the rainbow effect of so many major modes, including lisp-mode.

If one has been hesitating about using the latest ABCL because one "never uses a dot oh release", we have now sloughed off abcl-1.9.1 for  your appraisal from the depths of a Bear's long winter nap.  Now one can use the somewhat less buggy version of the Tenth Edition of Armed Bear Common Lisp, available, as usual, at <https://abcl.org/releases/1.9.1/> or (shortly) via Maven <https://search.maven.org/artifact/org.abcl/abcl/1.9.1/jar>.

Much has been written on Common Lisp; there is rarely one year without someone proclaming the death of the language and how nobody uses it anymore. And yet it is still here, so something must have been done right.

Common Lisp is not a software, it is a language described by the ANSI INCITS 226-1994 standard; there are multiple implementations available, something often used as argument for how alive and thriving the language is.

Let us see what the 2023 situation is.

General information

Note that all projects may have small parts with different licenses. This is particularily important for CLASP which contains multiple components imported from other projects.

I was quite surprised to see so many projects with recent releases. Clearly a good sign. Let us look at each implementation.

Implementations

SBCL

Steel Bank Common Lisp was forked from CMUCL in December 1999 and has since massively grown in popularity; it is currently the most used implementation by far. Unsurprisingly given its popularity, SBCL is supported by pretty much all Common Lisp libraries and tools out there. It is well known for generating fast native code compared to other implementations.

The most important aspect of SBCL is that it is actively maintained: its developers release new versions on a monthly basis, bringing each time a small list of improvements and bug fixes. Activity has actually increased these last years, something uncommon in the Common Lisp world.

CCL

Clozure Common Lisp has a long and complex history and has been around for decades. It is a mature implementation; it has two interesting aspects compared to SBCL:

• The compiler is much faster.
• Error messages tend to be clearer.

This is why I currently use it to test my code along SBCL. And according to what I have heard, this is a common choice among developers.

The main issue with CCL is that the project is almost completely abandonned. Git activity has slowed down to a crawl in the last two years, and none of the original maintainers from Clozure seem to be actively working on it. It remains nonetheless a major implementation.

ECL

Embeddable Common Lisp is a small implementation which can be used both as a library or as a standalone program. It contains a bytecode interpreter, but can also translate Lisp code to C to be compiled to native code.

While development is slow, improvements and bug fixes are still added on a regular basis. Clearly an interesting project: I could see myself using ECL to write plugins into an application able to call a C library.

ABCL

Armed Bear Common Lisp is quite different from other implementations: it produces Java bytecode and targets the Java Virtual Machine, making it a useful tool in Java ecosystems.

While it has not found the same success as Clojure, ABCL is still a fully featured Common Lisp implementation which passes almost the entire ANSI Common Lisp test suite.

Developement is slow nowadays but there are still new releases with lots of bug fixes. Also note that two of the developers are able to provide paid support.

CLASP

CLASP is a newcomer in the Common Lisp world (new meaning it is less than a decade old). Developed by Christian Schafmeister for his research work, this implementation has been used as an exemple of how alive and kicking Common Lisp, mainly due to two excellent presentations.

While very promising, CLASP suffers from its young age: trying to run the last release on my code resulted in a brutal error with now details and no backtrace. However I have no doubt that CLASP will get a lot better: it is actively maintained and used in production, two of the necessary ingredients for a software to stay relevant.

GCL

GNU Common Lisp is described as the official Common Lisp implementation for the GNU project. While it clearly does not have the popularity of other implementations, it is still a maintained project.

Trying to use it, I quickly realized it is not fully compliant with the standard. For example it will fail when evaluating a call to COMPILE-FILE with the :VERBOSE key argument.

Hopefully development will continue.

CLISP

CLISP is almost as old as I am; it was the first implementation I used a long time ago, and it still works. While it has all the usual features (multithreading, FFI, MOP, etc.), there is no real reason to use it compared to other implementations.

Even if it was to have any specific feature, CLISP is almost completely abandonned. While there are has been a semblant of activity a few years ago, active development pretty much stopped around 2012; the last release was more than 12 years ago.

Lispworks

Moving to proprietary implementations; Lispworks has been around for more than 30 years and the company producing it still release new versions on a regular basis.

While Lispworks supports most features you would expect from a commercial product (native compiler, multithreading, FFI, GUI library, various graphical tools, a Prolog implementation...), it is hampered by its licensing system.

The free “Personal Edition” limits the program size and the amount of time it can run, making it pretty much useless for anything but evaluation. The professional and enterprise licenses do not really make sense for anyone: you will have to buy separate licenses for every single platform at more than a thousand euros per license (with the enterprise version being 2-3 times more expensive). Of course you will have to buy a maintenance contract on a yearly basis... but it does not include technical support. It will have to be bought with “incident packs” costing thousands of euros; because yes, paying for a product and a maintenance contract does not mean they will fix bugs, and you will have to pay for each of them.

I do not have anything personal against commercial software, and I strongly support developers being paid for their work. But this kind of licensing makes Lispworks irrelevant to everyone but those already using their proprietary libraries.

Allegro

Allegro Common Lisp is the other well known proprietary implementation. Developped by Franz Inc., it is apparently used by multiple organizations including the U.S. Department of Defense.

Releases are uncommon, the last one being almost 6 years ago. But Allegro is a mature implementation packed with features not easily replicated such as AllegroCache, AllegroServe, libraries for multiple protocols and data formats, analysis tools, a concurrent garbage collector and even an OpenGL interface.

Allegro suffers the same issue as Lispworks: the enterprise-style pricing system is incredibly frustrating. The website advertises a hefty $599 starting price (which at least includes technical support), but there is no mention of what it contains. Interested developpers will have to contact Franz Inc. to get other prices. A quick Google search will reveal rumours of enterprise versions priced above 8000 dollars. No comment.

Conclusion

Researching Common Lisp implementations has been interesting. While it is clear that the language is far from dead, its situation is very fragile. Proprietary implementations are completely out of touch with the needs of most developers, leaving us with a single open source, actively maintained, high performance implementation: SBCL. Unless of course they are willing to deal with the JVM to use ABCL.

It might me interesting to investigate a possible solution to keep CCL somehow alive, with patches being merged and releases being produced. I sent a patch very recently, let us see what can be done!

The first rule of understanding closures is that you do not talk about closures. The second rule of understanding closures in Common Lisp is that you do not talk about closures. These are all the rules.

There is a lot of elaborate bowing and scraping about closures in the Lisp community. But despite that a closure isn’t actually a thing: the thing people call a closure is just a function which obeys the language’s rules about the scope and extent of bindings. Implementors need to care about closures: users just need to understand the rules for bindings. So rather than obsessing about this magic invisible thing which doesn’t actually exist in the language, I suggest that it is far better simply to think about the rules which cover bindings.

Angels and pinheads

It’s easy to see why this has happened: the CL standard has a lot of discussion of lexical closures, lexical and dynamic environments and so on. So it’s tempting to think that this way of thinking about things is ‘the one true way’ because it has been blessed by those who went before us. And indeed CL does have objects representing part of the lexical environment which are given to macro functions. Occasionally these are even useful. But there are no objects which represent closures as distinct from functions, and no predicates which tell you if a function is a closure or not in the standard language: closures simply do not exist as objects distinct from functions at all. They were useful, perhaps, as part of the text which defined the language, but they are nowhere to be found in the language itself.

So, with the exception of the environment objects passed to macros, none of these objects exist in the language. They may exist in implementations, and might even be exposed by some implementations, but from the point of the view of the language they simply do not exist: if I give you a function object you cannot know if it is a closure or not.

So it is strange that people spend so much time worrying about these objects which, if they even exist in the implementation, can’t be detected by anyone using the standard language. This is worrying about angels and pinheads: wouldn’t it be simpler to simply understand what the rules of the language actually say should observably happen? I think it would.

I am not arguing that the terminology used by the standard is wrong! All I am arguing is that, if you think you want to understand closures, you might instead be better off understanding the rules that give rise to them. And when you have done that you may suddenly find that closures have simply vanished into the mist: all you need is the rules.

History

Common Lisp is steeped in history: it is full of traces of the Lisps which went before it. This is intentional: one goal of CL was to enable programs written in those earlier Lisps — which were all Lisps at that time of course — to run without extensive modification.

But one place where CL didn’t steep itself in history is in exactly the areas that you need to understand to understand closures. Before Common Lisp (really, before Scheme), people spent a lot of time writing papers about the funarg problem and describing and implementing more-or-less complicated ways of resolving it. Then Scheme came along and decided that this was all nonsense and that it could just be made to go away by implementing the language properly. And the Common Lisp designers, who knew about Scheme, said that, well, if Scheme can do this, then we can do this as well, and so they also made it the problem vanish, although not in quite such an extreme way as Scheme did.

And this is now ancient history: these predecessor Lisps to CL are all at least 40 years old now. I am, just, old enough to have used some of them when they were current, but for most CL programmers these questions were resolved before they were born. The history is very interesting, but you do not need to steep yourself in it to understand closures.

Bindings

So the notion of a closure is part of the history behind CL: a hangover from the time when people worried about the funarg problem; a time before they understood that the whole problem could simply be made to go away. So, again, if you think you want to understand closures, the best approach is to understand something else: to understand bindings. Just as with closures, bindings do not exist as objects in the language, although you can make some enquiries about some kinds of bindings in CL. They are also a concept which exists in many programming languages, not just CL.

A binding is an association between a name — a symbol — and something. The most common binding is a variable binding, which is an association between a name and a value. There are other kinds of bindings however: the most obvious kind in CL is a function binding: an association between a name and a function object. And for example within a (possibly implicit) block there is a binding between the name of the block and a point to which you can jump. And there are other kinds of bindings in CL as well, and the set is extensible. The CL standard only calls variable bindings ‘bindings’, but I am going to use the term more generally.

Bindings are established by some binding construct and are usually not first-class objects in CL: they are just as vaporous as closures and environments. Nevertheless they are a powerful and useful idea.

What can be bound?

By far the most common kind of binding is a variable binding: an association between a name and a value. However there are other kinds of bindings: associations between names and other things. I’ll mention those briefly at the end, but in everything else that follows it’s safe to assume that ‘binding’ means ‘variable binding’ unless I say otherwise.

Scope and extent

For both variable bindings and other kinds of bindings there are two interesting questions you can ask:

• where is the binding available?
• when is the binding visible?

The first question is about the scope of the binding. The second is about the extent of the binding.

Each of these questions has (at least) two possible answers giving (at least) four possibilities. CL has bindings which use three of these possibilities and the fourth in a restricted case: two and a restricted version of a third for variable bindings, the other one for some other kinds of bindings.

Scope. The two options are:

• the binding may be available only in code where the binding construct is visible;
• or the binding may be available during all code which runs between where the binding is established and where it ends, regardless of whether the binding construct is visible.

What does ‘visible’ mean? Well, given some binding form, it means that the bindings it establishes are visible to all the code that is inside that form in the source. So, in a form like (let ((x 1)) ...) the binding of x is visible to the code that replaces the ellipsis, including any code introduced by macroexpansion, and only to that code.

Extent. The two options are:

• the binding may exist only during the time that the binding construct is active, and goes away when control leaves it;
• or the binding may exist as long as there is any possibility of reference.

Unfortunately the CL standard is, I think, slightly inconsistent in its naming for these options. However I’m going to use the standard’s terms with one exception. Here they are.

Scope:

• when a binding is available only when visible this called lexical scope;
• when a binding available to all code within the binding construct this is called indefinite scope¹;

Extent:

• when a binding ends at the end of the binding form this is called dynamic extent²;
• when a binding available indefinitely this called indefinite extent.

The term from the standard I am not going to use is dynamic scope, which it defines to mean the combination of indefinite scope and dynamic extent. I am not going to use this term because I think it is confusing: although it has ‘scope’ in its name it concerns both scope and extent. Instead I will introduce better, commonly used, terms below for the interesting combinations of scope and extent.

The four possibilities for bindings are then:

• lexical scope and dynamic extent;
• lexical scope and indefinite extent;
• indefinite scope and dynamic extent;
• indefinite scope and indefinite extent.

The simplest kind of binding

So then let’s ask: what is the simplest kind of binding to understand? If you are reading some code and you see a reference to a binding then what choice from the above options will make it easiest for you to understand whether that reference is valid or not?

Well, the first thing is that you’d like to be able to know by looking at the code whether a reference is valid or not. That means that the binding construct should be visible to you, or that the binding should have lexical scope. Compare the following two fragments of code:

(defun simple (x)
  ...
  (+ x 1)
  ...)

and

(defun confusing ()
  ...
  (+ *x* 1)
  ...)

Well, in the first one you can tell, just by looking at the code, that the reference to x is valid: the function, when called, establishes a binding of x and you can see that when reading the code. In the second one you just have to assume that the reference to *x* is valid: you can’t tell by reading the code whether it is or not.

Lexical scope makes it easiest for people reading the code to understand it, and in particular it is easier to understand than indefinite scope. It is the simplest kind of scoping to understand for people reading the code.

So that leaves extent. Well, in the two examples above definite or indefinite extent makes no difference to how simple the code is to understand: once the functions return there’s no possibility of reference to the bindings anyway. To expose the difference we need somehow to construct some object which can refer to a binding after the function has returned. We need something like this:

(defun maker (x)
  ...
  <construct object which refers to binding of x>)

(let ((o (maker 1)))
  <use o somehow to cause it to reference the binding of x>)

Well, what it this object going to be? What sort of things reference bindings? Code references bindings, and the objects which contain code are functions³. What we need to do is construct and return a function:

(defun maker (x)
  (lambda (y)
    (+ x y)))

and then cause this function to reference the binding by calling it:

(let ((f (maker 1)))
  (funcall f 2))

So now we can, finally, ask: what is the choice for the extent of the binding of x which makes this code simplest to understand? Well, the answer is that unless the binding of x remains visible to the function that is created in maker, this code can’t work at all. It would have to be the case that it was simply not legal to return functions like this from other functions. Functions, in other words, would not be first-class objects.

Well, OK, that’s a possibility, and it makes the above code simple to understand: it’s not legal and it’s easy to see that it is not. Except consider this small variant on the above:

(defun maybe-maker (x return-identity-p)
  (if return-identity-p
      #'identity
    (lambda (y)
      (+ x y))))

There is no way to know from reading this code whether maybe-maker will return the nasty anonymous function or the innocuous identity function. If it is not allowed to return anonymous functions in this way then there is no way of knowing whether

(funcall (maybe-maker 1 (zerop (random 2)))
         2)

is correct or not. This is certainly not simple: in fact it is a horrible nightmare. Another way of saying this is that you’d be in a situation where

(let ((a 1))
  (funcall (lambda ()
             a)))

would work, but

(funcall (let ((a 1))
           (lambda ()
             a)))

would not. There are languages which work that way: those languages suck.

So what would be simple? What would be simple is to say that if a binding is visible, it is visible, and that’s the end of the story. In a function like maker above the binding of x established by maker is visible to the function that it returns. Therefore it’s visible to the function that maker returns: without any complicated rules or weird special cases. That means the binding must have indefinite extent.

Indefinite extent makes it easiest for people reading the code to understand it when that code may construct and return functions, and in particular it is easier to understand than dynamic extent, which makes it essentially impossible to tell in many cases whether such code is correct or not.

And that’s it: lexical scope and indefinite extent, which I will call lexical binding, is the simplest binding scheme to understand for a language which has first-class functions⁴.

And really that’s it: that’s all you need to understand. Lexical scope and indefinite extent make reading code simple, and entirely explain the things people call ‘closures’ which are, in fact, simply functions which obey these simple rules.

Examples of the simple binding rules

One thing I have not mentioned before is that, in CL, bindings are mutable, which is another way of saying that CL supports assignment: assignment to variables is mutation of variable bindings. So, as a trivial example:

(defun maximum (list)
  (let ((max (first list)))
    (dolist (e (rest list) max)
      (when (> e max)
        (setf max e)))))

This is very easy to understand and does not depend on the binding rules in detail.

But, well, bindings are mutable, so the rules which say they exist as long as they can be referred to also imply they can be mutated as long as they can be referred to: anything else would certainly not be simple. So here’s a classic example of this:

(defun make-incrementor (&optional (value 0))
  (lambda (&optional (increment 1))
    (prog1 value
      (incf value increment))))

And now:

> (let ((i (make-incrementor)))
    (print (funcall i))
    (print (funcall i))
    (print (funcall i -2))
    (print (funcall i))
    (print (funcall i))
    (values))

0
1
2
0
1

As you can see, the function returned by make-incrementor is mutating the binding that it can still see.

What happens when two functions can see the same binding?

(defun make-inc-dec (&optional (value 0))
  (values
   (lambda ()
     (prog1 value
       (incf value)))
   (lambda ()
     (prog1 value
       (decf value)))))

And now

> (multiple-value-bind (inc dec) (make-inc-dec)
    (print (funcall inc))
    (print (funcall inc))
    (print (funcall dec))
    (print (funcall dec))
    (print (funcall inc))
    (values))

0
1
2
1
0

Again, what happens is the simplest thing: you can see simply from reading the code that both functions can see the same binding of value and they are therefore both mutating this common binding.

Here is an example which demonstrates all these features: an implementation of a simple queue as a pair of functions which can see two shared bindings:

(defun make-queue ()
  (let ((head '())
        (tail nil))
    (values
     (lambda (thing)
       ;; Push thing onto the queue
       (if (null head)
           ;; It's empty currently so set it up
           (setf head (list thing)
                 tail head)
         ;; not empty: just adjust the tail
         (setf (cdr tail) (list thing)
               tail (cdr tail)))
       thing)
     (lambda ()
       (cond
        ((null head)
         ;; empty
         (values nil nil))
        ((null (cdr head))
         ;; will be empty: don't actually need this case but it is
         ;; cleaner
         (values (prog1 (car head)
                   (setf head '()
                         tail nil))
                 t))
        (t
         ;; will still have content
         (values (pop head) t)))))))

make-queue will return two functions:

• the first takes one argument which it appends to the queue;
• the second takes no argument and either the next element of the queue and t or nil and nil if the queue is empty.

So, with this little function to drain the queue

(defun drain-and-print (popper)
  (multiple-value-bind (value fullp) (funcall popper)
    (when fullp
      (print value)
      (drain-and-print popper))
    (values)))

we can see this in action

> (multiple-value-bind (pusher popper) (make-queue)
    (funcall pusher 1)
    (funcall pusher 2)
    (funcall pusher 3)
    (drain-and-print popper))

1
2
3

A less-simple kind of binding which is sometimes very useful

Requiring bindings to be simple usually makes programs easy to read and understand. But it also makes it hard to do some things. One of those things is to control the ‘ambient state’ of a program. A simple example would be the base for printing numbers. It’s quite natural to say that ‘in this region of the program I want numbers printed in hex’.

If all we had was lexical binding then this becomes a nightmare: every function you call in the region you want to cause printing to happen in hex needs to take some extra argument which says ‘print in hex’. And if you then decide that, well, you’d also like some other ambient parameter, you need to provide more arguments to every function⁵. This is just horrible.

You might think you can do this with global variables which you temporarily set: that is both fiddly (better make sure you set it back) and problematic in the presence of multiple threads⁶.

A better approach is to allow dynamic bindings: bindings with indefinite scope & dynamic extent. CL has these, and at this point history becomes unavoidable: rather than have some separate construct for dynamic bindings, CL simply says that some variable bindings, and some references to variable bindings, are to be treated as having indefinite scope and dynamic extent, and you tell the system which bindings this applies to withspecial declarations / proclamations. CL does this because that’s very close to how various predecessor Lisps worked, and so makes porting programs from them to CL much easier. To make this less painful there is a convention that dynamically-bound variable names have *stars* around them, of course.

Dynamic bindings are so useful that if you don’t have them you really need to invent them: I have on at least two occasions implemented a dynamic binding system in Python, for instance.

However this is not an article on dynamic bindings so I will not write more about them here: perhaps I will write another article later.

What else can be bound?

Variable bindings are by far the most common kind. But not the only kind. Other things can be bound. Here is a partial list⁷:

• local functions have lexical scope and indefinite extent;
• block names have lexical scope and definite extent (see below);
• tag names have lexical scope and definite extent (see below);
• catch tags have indefinite scope and definite extent;
• condition handlers have indefinite scope and definite extent;
• restarts have indefinite scope and definite extent.

The two interesting cases here are block names and tag names. Both of these have lexical scope but only definite extent. As I argued above this makes it hard to know whether references to them are valid or not. Look at this, for example:

(defun outer (x)
  (inner (lambda (r)
           (return-from outer r))
         x))

(defun inner (r rp)
  (if rp
      r
    (funcall r #'identity)))

So then (funcall (outer nil) 1) will: call inner with a function which wants to return from outer and nil, which will cause inner to call that function, returning the identity function, which is then called by funcall with argument 1: the result is 1.

But (funcall (outer t) 1) will instead return the function which wants to return from outer, which is then called by funcall which is an error since it is outside the dynamic extent of the call to outer.

And there is no way that either a human reading the code or the compiler can detect that this is going to happen: a very smart compiler might perhaps be able to deduce that the internal function might be returned from outer, but probably only because this is a rather simple case: for instance in

(defun nasty (f)
  (funcall f (lambda ()
               (return-from nasty t))))

the situation is just hopeless. So this is a case where the binding rules are not as simple as you might like.

What is simple?

For variable bindings I think it’s easy to see that the simplest rule for a person reading the code is lexical binding. The other question is whether that is simpler for the implementation. And the answer is that probably it is not: probably lexical scope and definite extent is the simplest implementationally. That certainly approximates what many old Lisps did⁸. It’s fairly easy to write a bad implementation of lexical binding, simply by having all functions retain all the bindings, regardless of whether they might refer to them. A good implementation requires more work. But CL’s approach here is that doing the right thing for people is more important than making the implementor’s job easier. And I think that approach has worked well.

On the other hand CL hasn’t done the right thing for blocks and tags: There are at least three reasons for this.

Implementational complexity. If the bindings had lexical scope and indefinite extent then you would need to be able to return from a block which had already been returned from, and go to a tag from outside the extent of the form that established it. That opens an enormous can of worms both in making such an implementation work at all but also handling things like dynamic bindings, open files and so on. That’s not something the CL designers were willing to impose on implementors.

Complexity in the specification. If CL had lexical bindings for blocks and tags then the specification of the language would need to describe what happens in all the many edge cases that arise, including cases where it is genuinely unclear what the correct thing to do is at all such as dealing with open files and so on. Nobody wanted to deal with that, I’m sure: the language specification was already seen as far too big and the effort involved would have made it bigger, later and more expensive.

Conceptual difficulty. It might seem that making block bindings work like lexical variable bindings would make things simpler to understand. Well, that’s exactly what Scheme did with call/cc and call/cc can give rise to some of the most opaque code I have ever seen. It is often very pretty code, but it’s not easy to understand.

I think the bargain that CL has struck here is at least reasonable: to make the common case of variable bindings simple for people, and to avoid the cases where doing the right thing results in a language which is harder to understand in many cases and far harder to implement and specify.

Finally, once again I think that the best way to understand how closures in CL is not to understand them: instead understand the binding rules for variables, why they are simple and what they imply.

While Common Lisp has functions to open, read and write files, none of them takes care of reading and returning the entire content. This is something that I do very regularly, so it made sense to add such a function to Tungsten. It turned out to be a bit more complicated than expected.

A simple but incorrect implementation

The simplest implementation relies on the FILE-LENGTH function which returns the length of a stream (which of course only makes sense for a file stream). The Hyperspec clearly states that “for a binary file, the length is measured in units of the element type of the stream”. Since we are only reading binary data, everything is fine.

Let us write the function:

(defun read-file (path)
  (declare (type (or pathname string) path))
  (with-open-file (file path :element-type 'core:octet)
    (let ((data (make-array (file-length file) :element-type 'core:octet)))
      (read-sequence data file)
      data)))

Note that CORE:OCTET is a Tungsten type for (UNSIGNED-BYTE 8).

The function works as expected, returning the content of the file as an octet vector. But it is not entirely correct.

This implementation only works for regular files. Various files on UNIX will report a length of zero but can still be read. Now you might protest that it would not make sense to call READ-FILE on a device such as /dev/urandom, and you would be right. But a valid example would be pseudo files such as those part of procfs. If you want to obtain memory stats about your process on Linux, you can simply read /proc/self/statm. But this is not a regular file and READ-FILE will return an empty octet vector.

Doing it right and slow

The right way to read a file is to read its content block by block until the read operation fails because it reached the end of the file.

Let us re-write READ-FILE:

(defun read-file (path)
  (declare (type (or pathname string) path))
  (let ((data (make-array 0 :element-type 'core:octet :adjustable t))
        (block-size 4096)
        (offset 0))
    (with-open-file (file path :element-type 'core:octet)
      (loop
        (let* ((capacity (array-total-size data))
               (nb-left (- capacity offset)))
          (when (< nb-left block-size)
            (let ((new-length (+ capacity (- block-size nb-left))))
              (setf data (adjust-array data new-length)))))
        (let ((end (read-sequence data file :start offset)))
          (when (= end offset)
            (return-from read-file (adjust-array data end)))
          (setf offset end))))))

This time we rely on an adjustable array; we iterate, making sure we have enough space in the array to read an entire block each time. When the array is too short, we use ADJUST-ARRAY to extend it, relying on its ability to reuse the underlying storage instead of systematically copying its content.

Finally, once READ-SEQUENCE stops returning data, we truncate the array to the right size and return it.

This function worked correctly and I started using it regularly. Recently I started working with a file larger than usual and realized that READ-FILE was way too slow. With a NVMe drive, I would expect to be able to read a 10+MB file almost instantaneously, but it took several seconds.

After inspecting the code to find what could be so slow, I started to wonder about ADJUST-ARRAY; while I thought SBCL would internally extend the underlying memory in large blocks to minimize allocations, behaving similarly to realloc() in C, it turned out not to be the case. While reading the code behind ADJUST-ARRAY, I learned that it precisely allocates the required size. As a result, this implementation of READ-FILE performs one memory allocation for each 4kB block. Not a problem for small files, slow for larger ones.

A final version, correct and fast

Since I understood what the problem was, fixing it was trivial. When there is not enough space to read a block, we extend the array by at least 50% of its current size. Of course this is a balancing act: for example doubling the size at each allocation would reduce even more the number of allocations, but would increase the total amount of memory allocated. The choice is up to you.

(defun read-file (path)
  (declare (type (or pathname string) path))
  (let ((data (make-array 0 :element-type 'core:octet :adjustable t))
        (block-size 4096)
        (offset 0))
    (with-open-file (file path :element-type 'core:octet)
      (loop
        (let* ((capacity (array-total-size data))
               (nb-left (- capacity offset)))
          (when (< nb-left block-size)
            (let ((new-length (max (+ capacity (- block-size nb-left))
                                   (floor (* capacity 3) 2))))
              (setf data (adjust-array data new-length)))))
        (let ((end (read-sequence data file :start offset)))
          (when (= end offset)
            (return-from read-file (adjust-array data end)))
          (setf offset end))))))

This last version reads a 250MB file in a quarter of a second, while the original version took almost two minutes. Much better!

 New projects: 

• asdf-dependency-graph — A minimal wrapper around `dot` available at least on Linux systems to generate dependency-graphs. — MIT
• aws-sdk-lisp — AWS-SDK for Common Lisp — BSD 2-Clause
• chlorophyll — ANSI escape code library for Common Lisp — Expat
• ciao — OAuth 2.0 Client for Common Lisp — LGPLv3
• cl-cmark — Common Lisp bindings to libcmark, the CommonMark reference implementation — BSD-2-Clause
• cl-jingle — jingle -- ningle with bells and whistles — BSD 2-Clause
• cl-modio — A client library for the mod.io API. — zlib
• csv-validator — Validates tabular CSV data using predefined validations, similar to its Python counterpart 'Great Expectations'. — BSD-3
• history-tree — Store the history of a browser's visited paths. — BSD 3-Clause
• jzon — A correct and safe(er) JSON RFC 8259 parser with sane defaults. — MIT
• lisp-pay — Wrappers over multiple Payment Processor APIs — MIT
• mito-attachment — Mito mixin class for file management — LLGPL
• nclasses — Simplify class like definitions with define-class and friends. — Public Domain
• njson — NJSON is a JSON handling framework with the focus on convenience and brevity. — BSD-3 Clause
• nsymbols — A set of convenience functions to list class, variable, function, and other symbols. — BSD-3 Clause
• py4cl2-cffi — CFFI based alternative to PY4CL2, primarily developed for performance reasons. — MIT
• symath — A simple symbolic math library for Common Lisp — MIT

Updated projects: 3b-bmfont, 40ants-asdf-system, abstract-arrays, adp, ahungry-fleece, amb, april, babel, bdef, binary-parser, binpack, bp, cells, cffi, ci, cl+ssl, cl-advice, cl-bcrypt, cl-bnf, cl-cffi-gtk, cl-collider, cl-colors2, cl-confidence, cl-conspack, cl-containers, cl-cxx, cl-data-structures, cl-dbi, cl-digraph, cl-enumeration, cl-etcd, cl-fix, cl-gamepad, cl-git, cl-glib, cl-gobject-introspection, cl-gobject-introspection-wrapper, cl-graph, cl-gserver, cl-hamcrest, cl-i18n, cl-indentify, cl-jpeg, cl-lambdacalc, cl-mathstats, cl-megolm, cl-migratum, cl-mixed, cl-naive-store, cl-oju, cl-opencl-utils, cl-patterns, cl-pdf, cl-prevalence, cl-protobufs, cl-rashell, cl-rdkafka, cl-rfc4251, cl-ssh-keys, cl-steamworks, cl-store, cl-str, cl-telegram-bot, cl-trie, cl-unification, cl-utils, cl-vorbis, cl-webkit, cl-wol, cl-xkb, cl-yacc, clack, clad, clast, clath, clavier, clazy, climacs, climc, clingon, clip, clog, closer-mop, cluffer, clx, cmd, colorize, common-doc, consfigurator, croatoan, cserial-port, ctype, database-migrations, defenum, definer, dense-arrays, deploy, depot, dexador, djula, doc, docparser, docs-builder, dsm, duologue, eclector, extensible-compound-types, fare-csv, fare-utils, file-attributes, file-select, filesystem-utils, find-port, fiveam-matchers, float-features, for, functional-trees, glacier, gtirb-capstone, gtirb-functions, harmony, hashtrie, helambdap, http2, hu.dwim.syntax-sugar, in-nomine, journal, js, jsonrpc, lack, lass, lift, lisp-unit2, lmdb, local-time, log4cl-extras, lunamech-matrix-api, maiden, markup, mcclim, metabang-bind, mfiano-utils, mgl-mat, mgl-pax, mito, mk-defsystem, mmap, more-cffi, multiposter, music-spelling, mutility, nail, ndebug, new-op, nfiles, nhooks, nkeymaps, nodgui, north, numerical-utilities, numericals, nyxt, omglib, ook, osc, osicat, parachute, pero, petalisp, plot, plump, polymorphic-functions, postmodern, promise, py4cl2, qtools, random-state, read-number, replic, rove, sc-extensions, sel, serapeum, shop3, simple-neural-network, sketch, slime, sly, speechless, spinneret, statistics, stepster, stumpwm, swank-client, swank-crew, teepeedee2, ten, testiere, tfeb-lisp-hax, tooter, trace-db, trivial-backtrace, trivial-coerce, trivial-download, trivial-mimes, trivial-package-locks, trivial-shell, trivial-timeout, type-i, typo, uax-9, vellum, vellum-postmodern, vk, vp-trees, wayflan, with-contexts, with-user-abort, wookie, workout-timer, xhtmlambda, yason, zippy.

Removed projects: cl-sane, cluster, vellum-binary.

To get this update, use (ql:update-dist "quicklisp"). Enjoy!

It's already been a month since Kandria released! Woah, time sure flies these days, huh?

Kandria Launch

Well, in case you missed the launch, the game is now officially available on Steam, Itch, and directly on our Website.

The reviews we've gotten have been very positive, both from press, on Steam, and on the streams I've caught! I'm really happy that people have been enjoying the game, and am very thankful for all of the support and well wishes.

The first two weeks after launch were spent furiously fixing things, with a patch being released almost every day. Things have calmed down a lot since, and while there's a few polish things left over that we know about, overall the game is now very stable, and even got the official Steam Deck Verified rating!

But, the work isn't done yet.

Upcoming Attractions

Back in the Kickstarter we reached three stretch goals. One of those was an extra questline, which is already in the released game. But, the two other goals I'm hammering out now:

• Level Editor
  This is actually already in... sort of. If you open the game and press the section/tilde key below Escape, you'll toggle the editor during gameplay. If you want, you can mess about with it! But, it's still a bit rough around the edges and especially sharing levels with others is not easy enough for my tastes. So, that's what this update is about. Alongside the update we'll also organise a little contest with rewards, so look forward to that!

• Modding Support
  And this one is quite exciting as well. Part of the modding support update is the source code release of Kandria, which has already happened. But, the more important and difficult parts are a stable API and mod loading mechanism, the integration with the mod.io API, and a convenient user interface to manage the mods. An extremely rudimentary API already exists now, but it will be quite a while before things become stable.

Alongside both of those updates I'll also be writing documentation on the editor and the modding, to ensure people have an easier entry point. The editor should be usable without any coding knowledge, too.

I can't promise any dates for the two features, but if things go really well the editor will be out sometime next month!

But, there is yet another update: there'll be a Japanese translation for Kandria! As someone that's been trying (and mostly failing) to learn Japanese for many years, this is quite exciting for me. If all goes well it should be out by June.

And that's all I can say for now. If you're interested in the level editor and modding stuff, please hop on by our Discord! You'll be able to ask questions there and chat with other interested folks.

While SLIME is most of the time able to indent Common Lisp correctly, it will sometimes trip on custom forms. Let us see how we can customize indentation.

In the process of writing my PostgreSQL client in Common Lisp, I wrote a READ-MESSAGE-CASE macro which reads a message from a stream and execute code depending on the type of the message:

(defmacro read-message-case ((message stream) &rest forms)
  `(let ((,message (read-message ,stream)))
     (case (car ,message)
       (:error-response
        (backend-error (cdr ,message)))
       (:notice-response
        nil)
       ,@forms
       (t
        (error 'unexpected-message :message ,message)))))

This macro is quite useful: all message loops can use it to automatically handle error responses, notices, and signal unexpected messages.

But SLIME does not know how to indent READ-MESSAGE-CASE, so by default it will align all message forms on the first argument:

(read-message-case (message stream)
                   (:authentication-ok
                     (return))
                   (:authentication-cleartext-password
                     (unless password
                       (error 'missing-password))
                     (write-password-message password stream)))

While we want it aligned the same way as HANDLER-CASE:

(read-message-case (message stream)
  (:authentication-ok
    (return))
  (:authentication-cleartext-password
    (unless password
      (error 'missing-password))
    (write-password-message password stream)))

Good news, SLIME indentation is defined as a list of rules. Each rule associates an indentation specification (a S-expression describing how to indent the form) to a symbol and store it as the common-lisp-indent-function property of the symbol.

You can obtain the indentation rule of a Common Lisp symbol easily. For example, executing (get 'defun 'common-lisp-indent-function) (e.g. in IELM or with eval-expression) yields (4 &lambda &body). This indicates that DEFUN forms are to be indented as follows:

• The first argument of DEFUN (the function name) is indented by four spaces.
• The second argument (the list of function arguments) is indented as a lambda list.
• The rest of the arguments are indented based on the lisp-body-indent custom variable, which controls the indentation of the body of a lambda form (two spaces by default).

You can refer to the documentation of the common-lisp-indent-function Emacs function (defined in SLIME of course) for a complete description of the format.

We want READ-MESSAGE-CASE to be indented the same way as HANDLER-CASE, whose indentation specification is (4 &rest (&whole 2 &lambda &body)) (in short, an argument and a list of lambda lists). Fortunately there is a way to specify that a form must be indented the same way as another form, using (as <symbol>).

Let us first define a function to set the indentation specification of a symbol:

(defun g-common-lisp-indent (symbol indent)
  "Set the indentation of SYMBOL to INDENT."
  (put symbol 'common-lisp-indent-function indent))

Then use it for READ-MESSAGE-CASE:

(g-common-lisp-indent 'read-message-case '(as handler-case))

While it is in general best to avoid custom indentation, exceptions are sometimes necessary for readability. And SLIME makes it easy.

Table of Contents

1. Introduction
2. Defining method combinations - the short form
3. Defining method combinations - the long form
   1. The Hooker
   2. The Memoizer
4. Conclusions

Update [2023-01-23]

Christophe Rhodes pointed out that "The Hooker" method combination is not conforming because there are multiple methods with the same "role" that can't be ordered and that have different qualifiers:

Note that two methods with identical specializers, but with different qualifiers, are not ordered by the algorithm described in Step 2 of the method selection and combination process described in Section 7.6.6 (Method Selection and Combination). Normally the two methods play different roles in the effective method because they have different qualifiers, and no matter how they are ordered in the result of Step 2, the effective method is the same. If the two methods play the same role and their order matters, an error is signaled. This happens as part of the qualifier pattern matching in define-method-combination.

http://www.lispworks.com/documentation/HyperSpec/Body/m_defi_4.htm

So instead of using qualifier patterns we should use qualifier predicates. They are not a subject of the above paragraph because of its last sentence (there is also an example in the spec that has multiple methods with a predicate). So instead of

(define-method-combination hooker ()
  (... (hook-before (:before*)) ...) ...)

the method combination should use:

(defun hook-before-p (method-qualifier)
  (typep method-qualifier '(cons (eql :before) (cons t null))))

(define-method-combination hooker ()
  (... (hook-before hook-before-p) ...) ...)

and other "hook" groups should also use predicates.

Another thing worth mentioning is that both ECL and SBCL addressed issues with the qualifier pattern matching and :arguments since the publication of this blog post.

Introduction

Method combinations are used to compute the effective method for a generic function. An effective method is a body of the generic function that combines a set of applicable methods computed based on the invocation arguments.

For example we may have a function responsible for reporting the object status and each method focuses on a different aspect of the object. In that case we may want to append all results into a list:

(defgeneric status (object)
  (:method-combination append))

(defclass base-car ()
  ((engine-status :initarg :engine :accessor engine-status)
   (wheels-status :initarg :wheels :accessor wheels-status)
   (fuel-level :initarg :fuel :accessor fuel-level))
  (:default-initargs :engine 'ok :wheels 'ok :fuel 'full))

(defmethod status append ((object base-car))
  (list :engine (engine-status object)
        :wheels (wheels-status object)
        :fuel (fuel-level object)))

(defclass premium-car (base-car)
  ((gps-status :initarg :gps :accessor gps-status)
   (nitro-level :initarg :nitro :accessor nitro-level))
  (:default-initargs :gps 'no-signal :nitro 'low))

(defmethod status append ((object premium-car))
  (list :gps (gps-status object)
        :nitro (nitro-level object)))

CL-USER> (status (make-instance 'premium-car))
(:GPS NO-SIGNAL :NITRO LOW :ENGINE OK :WHEELS OK :FUEL FULL)

CL-USER> (status (make-instance 'base-car))
(:ENGINE OK :WHEELS OK :FUEL FULL)

The effective method may look like this:

(append (call-method #<method status-for-premium-car>)
        (call-method #<method status-for-base-car>   ))

Note that append is a function so all methods are called. It is possible to use other operators (for example a macro and) and then the invocation of particular methods may be conditional:

(and (call-method #<method can-repair-p-for-premium-car>)
     (call-method #<method can-repair-p-for-base-car>   ))

Defining method combinations - the short form

The short form allows us to define a method combination in the spirit of the previous example:

(OPERATOR (call-method #<m1>)
          (call-method #<m2>)
          ...)

For example we may want to return as the second value the count of odd numbers:

(defun sum-and-count-odd (&rest args)
  (values (reduce #'+ args)
          (count-if #'oddp args)))

(define-method-combination sum-and-count-odd)

(defclass a () ())
(defclass b (a) ())
(defclass c (b) ())

(defgeneric num (o)
  (:method-combination sum-and-count-odd)
  (:method sum-and-count-odd ((o a)) 1)
  (:method sum-and-count-odd ((o b)) 2)
  (:method sum-and-count-odd ((o c)) 3)
  (:method :around ((o c))
    (print "haa!")
    (call-next-method)))

(num (make-instance 'b)) ;; (values 3 1)
(num (make-instance 'c)) ;; (values 6 2)

Note that the short form supports also around methods. It is also important to note that effective methods are cached, that is unless the generic function or the method combination changes, the computation of the effective method may be called only once per the set of effective methods.

Admittedly these examples are not very useful. Usually we operate on data stored in instances and this is not a good abstraction to achieve that. Method combinations are useful to control method invocations and their results. Here is another example:

(defmacro majority-vote (&rest method-calls)
  (let* ((num-methods (length method-calls))
         (tie-methods (/ num-methods 2)))
    `(prog ((yes 0) (no 0))
        ,@(loop for invocation in method-calls
                append `((if ,invocation
                             (incf yes)
                             (incf no))
                         (cond
                           ((> yes ,tie-methods)
                            (return (values t yes no)))
                           ((> no ,tie-methods)
                            (return (values nil yes no))))))
        (error "we have a tie! ~d ~d" yes no))))

(define-method-combination majority-vote)

(defclass a () ())
(defclass b (a) ())
(defclass c (b) ())
(defclass d (c) ())

(defgeneric foo (object param)
  (:method-combination majority-vote)
  (:method majority-vote ((o a) param) nil)
  (:method majority-vote ((o b) param) t)
  (:method majority-vote ((o c) param) t)
  (:method majority-vote ((o d) param) nil))

(foo (make-instance 'a) :whatever) ; (values nil 0 1)
(foo (make-instance 'b) :whatever) ; #<error tie 1 1>
(foo (make-instance 'c) :whatever) ; (values t 2 0)
(foo (make-instance 'd) :whatever) ; #<error tie 2 2>

Defining method combinations - the long form

The long form is much more interesting. It allows us to specify numerous qualifiers and handle methods without any qualifiers at all.

The Hooker

Here we will define a method combination that allows us to define named hooks that are invoked before or after the method. It is possible to have any number of hooks for the same set of arguments (something we can't achieve with the standard :before and :after auxiliary methods):

(defun combine-auxiliary-methods (primary around before after)
  (labels ((call-primary ()
             `(call-method ,(first primary) ,(rest primary)))
           (call-methods (methods)
             (mapcar (lambda (method)
                       `(call-method ,method))
                     methods))
           (wrap-after (the-form)
             (if after
                 `(multiple-value-prog1 ,the-form
                    ,@(call-methods after))
                 the-form))
           (wrap-before (the-form)
             (if before
                 `(progn
                    ,@(call-methods before)
                    ,the-form)
                 the-form))
           (wrap-around (the-form)
             (if around
                 `(call-method ,(first around)
                               (,@(rest around)
                                (make-method ,the-form)))
                 the-form)))
    (wrap-around (wrap-after (wrap-before (call-primary))))))

(define-method-combination hooker ()
  ((normal-before (:before))
   (normal-after  (:after)
                  :order :most-specific-last)
   (normal-around (:around))
   (hook-before   (:before *))
   (hook-after    (:after  *)
                  :order :most-specific-last)
   (hook-around   (:around *))
   (primary () :required t))
  (let ((around (append hook-around normal-around))
        (before (append hook-before normal-before))
        (after  (append normal-after hook-after)))
    (combine-auxiliary-methods primary around before after)))

With this we may define a generic function and associated methods similar to other functions with an extra feature - we may provide named :before, :after and :around methods. Named auxiliary methods take a precedence over unnamed ones. Only after that the specialization is considered. There is one caveat - PCL-derived CLOS implementations (clasp, cmucl, ecl, sbcl) currently ([2023-01-18 śro]) have a bug preventing wildcard qualifier pattern symbol * from working. So better download ccl or wait for fixes. Here's an example for using it:

;;; The protocol.
(defgeneric note-buffer-dimensions-changed (buffer w h)
  (:method (b w h)
    (declare (ignore b w h))
    nil))

(defgeneric change-dimensions (buffer w h)
  (:method-combination hooker))

;;; The implementation of unspecialized methods.
(defmethod change-dimensions :after (buffer w h)
  (note-buffer-dimensions-changed buffer w h))

;;; The stanard class.
(defclass buffer ()
  ((w :initform 0 :accessor w)
   (h :initform 0 :accessor h)))

;;; The implementation for the standard class.
(defmethod change-dimensions ((buffer buffer) w h)
  (print "... Changing the buffer size ...")
  (setf (values (w buffer) (h buffer))
        (values w h)))

(defmethod note-buffer-dimensions-changed ((buffer buffer) w h)
  (declare (ignore buffer w h))
  (print "... Resizing the viewport ..."))

;;; Some dubious-quality third-party code that doesn't want to interfere with
;;; methods defined by the implementation.
(defmethod change-dimensions :after system (buffer w h)
  (print `(log :something-changed ,buffer ,w ,h)))

(defmethod change-dimensions :after my-hook ((buffer buffer) w h)
  (print `(send-email! :me ,buffer ,w ,h)))

CL-USER> (defvar *buffer* (make-instance 'buffer))
*BUFFER*
CL-USER> (change-dimensions *buffer* 10 30)

"... Changing the buffer size ..." 
"... Resizing the viewport ..." 
(LOG :SOMETHING-CHANGED #<BUFFER #x30200088220D> 10 30) 
(SEND-EMAIL! :ME #<BUFFER #x30200088220D> 10 30) 
10
30

The Memoizer

Another example (this time it will work on all implementations) is optional memoization of the function invocation. If we define a method with the qualifier :memoize then the result will be cached depending on arguments. The method combination allows also "normal" auxiliary functions by reusing the function combine-auxiliary-methods from the previous section.

The function ensure-memoized-result accepts the following arguments:

• test: compare generations
• memo: a form that returns the current generation
• cache-key: a list composed of a generic function and its arguments
• form: a form implementing the method to be called

When the current generation is nil that means that caching is disabled and we remove the result from the cache. Otherwise we use the test to compare the generation of a cached value and the current one - if they are the same, then the cached value is returned. Otherwise it is returned.

(defparameter *memo* (make-hash-table :test #'equal))
(defun ensure-memoized-result (test memo cache-key form)
  `(let ((new-generation ,memo))
     (if (null new-generation)
         (progn
           (remhash ,cache-key *memo*)
           ,form)
         (destructuring-bind (old-generation . cached-result)
             (gethash ,cache-key *memo* '(nil))
           (apply #'values
                  (if (,test old-generation new-generation)
                      cached-result
                      (rest
                       (setf (gethash ,cache-key *memo*)
                             (list* new-generation (multiple-value-list ,form))))))))))

The method with the qualifier :memoize is used to compute the current generation key. When there is no such method then the function behaves as if the standard method combination is used. The method combination accepts a single argument test, so it is possible to define different predicates for deciding whether the cache is up-to-date or not.

(define-method-combination memoizer (test)
  ((before (:before))
   (after  (:after) :order :most-specific-last)
   (around (:around))
   (memoize (:memoize))
   (primary () :required t))
  (:arguments &whole args)
  (:generic-function function)
  (let ((form (combine-auxiliary-methods primary around before after))
        (memo `(call-method ,(first memoize) ,(rest memoize)))
        (ckey `(list* ,function ,args)))
    (if memoize
        (ensure-memoized-result test memo ckey form)
        form)))

Now let's define a function with "our" method combination. We will use a counter to verify that values are indeed cached.

(defparameter *counter* 0)

(defgeneric test-function (arg &optional opt)
  (:method-combination memoizer eql))

(defmethod test-function ((arg integer) &optional opt)
  (list* `(:counter ,(incf *counter*)) arg opt))

CL-USER> (test-function 42)
((:COUNTER 1) 42)
CL-USER> (test-function 42)
((:COUNTER 2) 42)
CL-USER> (defmethod test-function :memoize ((arg integer) &optional (cache t))
           (and cache :gen-z))
#<STANDARD-METHOD TEST-FUNCTION :MEMOIZE (INTEGER)>
CL-USER> (test-function 42)
((:COUNTER 3) 42)
CL-USER> (test-function 42)
((:COUNTER 3) 42)
CL-USER> (test-function 42 nil)
((:COUNTER 4) 42)
CL-USER> (test-function 42)
((:COUNTER 3) 42)
CL-USER> (test-function 43)
((:COUNTER 5) 43)
CL-USER> (test-function 43)
((:COUNTER 5) 43)
CL-USER> (defmethod test-function :memoize ((arg (eql 43)) &optional (cache t))
           (and cache :gen-x))
#<STANDARD-METHOD TEST-FUNCTION :MEMOIZE ((EQL 43))>
CL-USER> (test-function 43)
((:COUNTER 6) 43)
CL-USER> (test-function 43)
((:COUNTER 6) 43)
CL-USER> (test-function 42)
((:COUNTER 3) 42)

Conclusions

Method combinations are a feature that is often overlooked but give a great deal of control over the generic function invocation. The fact that ccl is the only implementation from a few that I've tried which got method combinations "right" doesn't surprise me - I've always had an impression that it shines in many unexpected places.

I was working on the terminal output for a Common Lisp logger, and I realized that SLIME does not interpret ANSI escape sequences.

This is not the end of the world, but having at least colors would be nice. Fortunately there is a library to do just that.

First let us install the package, here using use-package and straight.el.

(use-package slime-repl-ansi-color
  :straight t)

While in theory we are supposed to just add slime-repl-ansi-color to slime-contribs, it did not work for me, and I add to enable the minor mode manually.

If you already have a SLIME REPL hook, simply add (slime-repl-ansi-color-mode 1). If not, write an initialization function, and add it to the SLIME REPL initialization hook:

(defun g-init-slime-repl-mode ()
  (slime-repl-ansi-color-mode 1))
  
(add-hook 'slime-repl-mode-hook 'g-init-slime-repl-mode)

To test that it works as intended, fire up SLIME and print a simple message using ANSI escape sequences:

(let ((escape (code-char 27)))
  (format t "~C[1;33mHello world!~C[0m~%" escape escape))

While it is tempting to use the #\Esc character, it is part of the Common Lisp standard; therefore we use CODE-CHAR to obtain it from its ASCII numeric value. We use two escape sequences, the first one to set the bold flag and foreground color, and the second one to reset display status.

If everything works well, should you see a nice bold yellow message:

Job posting: https://jobs.lever.co/dodmg/85221f38-1def-4b3c-b627-6ad26d4f5df7?lever-via=CxJdiOp5C6

HRL has been on the leading edge of technology, conducting pioneering research and advancing the state of the art. This position is integrated with a growing team of scientists and engineers on HRL's quantum computing research program.

GENERAL DESCRIPTION:

As a DevOps/DevSecOps engineer, you’ll be focused on maintaining reliable systems for testing and delivery of HRL’s quantum software. (You will not be directly responsible for developing the software or its tests.)

Specifically, you will be responsible for:

* Monitoring the status of CI/CD infrastructure on open and air-gapped networks.

* Building and maintaining infrastructure for synchronizing software between open and air-gapped networks.

* Working closely with developers and IT staff to ensure continued reliability of integration and deployment infrastructure.

* Tracking and vetting software dependencies.

* Looking for and implementing improvements to DevSecOps practices.

Among other candidate requirements, we highly value expertise in Lisp, Python, and C++.

CONFIDENTIALITY NOTICE: The information transmitted in this email, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential, proprietary and/or privileged material exempt from disclosure under applicable law. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this message in error, please contact the sender immediately and destroy any copies of this information in their entirety.

While I mostly use SBCL for Common Lisp development, I regularly switch to CCL or even ECL to run tests.

This is how I do it with SLIME.

Starting implementations

SLIME lets you configure multiple implementations using the slime-lisp-implementations setting. In my case:

(setq slime-lisp-implementations
   '((sbcl ("/usr/bin/sbcl" "--dynamic-space-size" "2048"))
     (ccl ("/usr/bin/ccl"))
     (ecl ("/usr/bin/ecl"))))

Doing so means that running M-x slime will execute the first implementation, i.e. SBCL. There are two ways to run other implementations.

First you can run C-u M-x slime which lets you type the path and arguments of the implementation to execute. This is a bit annoying because the prompt starts with the content of the inferior-lisp-program variable, i.e. "lisp" by default, meaning it has to be deleted manually each time. Therefore I set inferior-lisp-program to the empty string:

(setq inferior-lisp-program "")

Then you can run C-- M-x slime (or M-- M-x slime which is easier to type) to instruct SLIME to use interactive completion (via completing-read) to let you select the implementations among those configured in slime-lisp-implementations.

To make my life easier, I bind C-c C-s s to a function which always prompt for the implementation to start:

(defun g-slime-start ()
  (interactive)
  (let ((current-prefix-arg '-))
    (call-interactively 'slime)))

Using C-c C-s as prefix for all my global SLIME key bindings helps me remember them.

Switching between multiple implementations

Running the slime function several times will create multiple connections as expected. Commands executed in Common Lisp buffers are applied to the current connection, which is by default the most recent one.

There are two ways to change the current implementation:

1. Run M-x slime-next-connection.
2. Run M-x slime-list-connections, which opens a buffer listing connections, and lets you choose the current one with the d key.

I find both impractical: the first one does not let me choose the implementation, forcing me to run potentially several times before getting the one I want. The second one opens a buffer but does not switch to it.

All I want is a prompt with completion. So I wrote one.

First we define a function to select a connection among existing one:

(defun g-slime-select-connection (prompt)
  (interactive)
  (let* ((connections-data
          (mapcar (lambda (process)
                    (cons (slime-connection-name process) process))
                  slime-net-processes))
         (completion-extra-properties
          '(:annotation-function
            (lambda (string)
              (let* ((process (alist-get string minibuffer-completion-table
                                         nil nil #'string=))
                     (contact (process-contact process)))
                (if (consp contact)
                    (format "  %s:%s" (car contact) (cadr contact))
                  (format "  %S" contact))))))
         (connection-name (completing-read prompt connections-data)))
    (let ((connection (cl-find connection-name slime-net-processes
                               :key #'slime-connection-name
                               :test #'string=)))
      (or connection
          (error "Unknown SLIME connection %S" connection-name)))))

Then use it to select a connection as the current one:

(defun g-slime-switch-connection ()
  (interactive)
  (let ((connection (g-slime-select-connection "Switch to connection: ")))
    (slime-select-connection connection)
    (message "Using connection %s" (slime-connection-name connection))))

I bind this function to C-c C-s c.

In a perfect world, we could format nice columns in the prompt and highlight the current connection, but the completing-read interface is really limited, and I did not want to use an external package such as Helm.

Stopping implementations

Sometimes it is necessary to stop an implementations and kill all associated buffers. It is not something I use a lot; but when I need it, it is frustrating to have to switch to the REPL buffer, run slime-quit-lisp, then kill the REPL buffer manually.

Adding this feature is trivial with the g-slime-select-connection defined earlier:

(defun g-slime-kill-connection ()
  (interactive)
  (let* ((connection (g-slime-select-connection "Kill connection: "))
         (repl-buffer (slime-repl-buffer nil connection)))
    (when repl-buffer
      (kill-buffer repl-buffer))
    (slime-quit-lisp-internal connection 'slime-quit-sentinel t)))

Finally I bind this function to C-c C-s k.

It is now much more comfortable to manage multiple implementations.

I often find myself wanting a simple case-like macro where the keys are regular expressions. regex-case is an attempt at this.

I use CL-PPCRE for the usual things regular expressions are useful for, and probably for some of the things they should not really be used for as well. I often find myself wanting a case like macro, where the keys are regular expressions. There is a contributed package for Trivia which will do this, but Trivia is pretty overwhelming. So I gave in and wrote regex-case which does what I want.

regex-case is a case-like macro. It looks like

(regex-case <thing>
  (<pattern> (...)
   <form> ...)
  ...
  (otherwise ()
   <form> ...))

Here <pattern> is a literal regular expression, either a string or in CL-PPCRE’s s-expression parse-tree syntax for them. Unlike case there can only be a single pattern per clause: allowing the parse-tree syntax makes it hard to do anything else. otherwise (which can also be t) is optional but must be last.

The second form in a clause specifies what, if any, variables to bind on a match. As an example

(regex-case line
  ("fog\\s+(.*)\\s$" (:match m :registers (v))
    ...)
  ...)

will bind m to the whole match and v to the substring corresponding to the first register. You can also bind match and register positions. A nice (perhaps) thing is that you can not bind some register variables:

(regex-case line
  (... (:registers (_ _ v))
   ...)
  ...)

will bind v to the substring corresponding to the third register. You can use nil instead of _.

The current state of regex-case is a bit preliminary: in particular I don’t like the syntax for binding thngs very much, although I can’t think of a better one. Currently therefore it’s in my collection of toys: it will probably migrate from there at some point.

Currently documentation is here and source code is here.

Kandria is now finally available for purchase and play!

I recommend buying it on Steam, as the algorithm there will help us bring the game in front of more people, as well. However, if that isn't a possibility for you, there's also options on Itch.io and through Xsolla on our webpage:

• Steam

• Itch.io

• Xsolla

I am also live on Steam, Twitch, and YouTube right now, to celebrate the launch! Come on and hang out in the chat: https://stream.shinmera.com

I hope you all enjoy the game, and thank you very much for sticking with us for all this time!

​Kandria launches tomorrow, on Wednesday the 11th, at 15:00 CET / 9:00 EST!

There'll be a launch stream for the occasion as well. It'll be live on Twitch! I'll be happy to answer any questions you may have about the game, and hope to see you there!​

Last opportunity to wishlist the game, too: https://kandria.com/steam

And 2022 is over. The Common Lisp language and environment are solid and stable, yet evolve. Implementations, go-to libraries, best practices, communities evolve. We don’t need a “State of the Ecosystem” every two weeks but still, what happened and what did you miss in 2022?

This is my pick of the most exciting, fascinating, interesting or just cool projects, tools, libraries and articles that popped-up during that time (with a few exceptions that appeared in late 2021).

This overview is not a “State of the CL ecosystem” (HN comments (133)) that I did in 2020, for which you can find complementary comments on HN.

I think this article (of sorts) is definitely helpful for onlookers to Common Lisp, but doesn’t provide the full “story” or “feel” of Common Lisp, and I want to offer to HN my own perspective.

And, suffice to say, I tried to talk about the most important things, but this article (of sorts) is by no means a compilation of all new CL projects or all the articles published on the internet. Look on Reddit, Quicklisp releases, Github, and my favourite resources:

• Awesome-cl - a curated list of libraries (there might be more than you think)
• the CL Cookbook

If I had to pick 3 achievements they would be:

• SBCL developments: SBCL is now callable as a shared library. See below in “Implementations”.
• a new 3D graphics project: Kons-9: “The idea would be to develop a system along the lines of Blender/Maya/Houdini, but oriented towards the strengths of Common Lisp”. And the project progresses at a good pace.
• CLOG, the Common Lisp Omnificent GUI. It’s like a GUI framework to create web apps. Based on websockets, it offers a light abstraction to create fully-dynamic web applications, in Common Lisp. It has lots of demos to create websites, web apps, games, and it ships a complete editor. For development, we can connect our Lisp REPL to the browser, and see changes on the fly. The author had a similar commercial product written in Ada, discovered Common Lisp, and is now super active on this project.

Let’s go for more.

Thanks to @k1d77a, @Hexstream, @digikar and @stylewarning for their feedback.

Table of Contents

• Documentation
• Implementations
• Jobs
• Projects
  • Language libraries
  • Editors, online editors, REPLs, plugins
  • Concurrency
  • Databases
  • Delivery tools
  • Games
  • Graphics, GUIs
    • Kons-9, a new 3D graphics project
  • Interfaces with other languages
  • Numerical and scientific
  • Web
    • CLOG
  • New releases
  • (re) discoveries
• Articles
  • Graphics
  • Tooling
  • Around the language
  • Web related
  • Other articles
• Screencasts and podcasts
• Other discussions
  • Community
  • Learning Lisp
  • Common Lisp VS ...

Documentation

A newcomer to Lisp came, asked a question, and suddenly he created a super useful rendering of the specification. Check it out!

• 🚀 Common Lisp CommunitySpec (CLCS) - a rendition of the Common Lisp ANSI Specification draft.
  • Github: https://github.com/fonol/cl-community-spec
  • It is readable, it has syntax highlighting for code snippets, an interactive search bar, it looks modern, it is open.

But that’s not all, he also started work on a new Common Lisp editor, built in Rust and Tauri, see below.

We continue to enrich the Common Lisp Cookbook. You are welcome to join, since documention is best built by newcomers for newcomers.

• New CL Cookbook EPUB and PDF release - mainly readability and file format improvements - thanks to the 13 contributors

A resurrected project:

• 🚀 Learn Lisp the Hard Way is back online

Also:

• the revamped Common Lisp Document Repository (CDR) site, “a repository of documents that are of interest to the Common Lisp community. The most important property of a CDR document is that it will never change”
  • comments
• I am creating a Common Lisp video course on Udemy 🎥
  • read more about my motivation and follow the project on Github: https://github.com/vindarel/common-lisp-course-in-videos/
  • the course has some free videos. If you are a student, drop me a line for a free link.
  • direct link to the course.
  • Thanks for your support!

Implementations

We saw achievements in at least 7 8 implementations.

• SBCL continues to ship monthly. In 2022:
  • 🚀 SBCL is now callable as a shared library. See sbcl-librarian below.
  • 🚀 sb-simd - SIMD programming in SBCL.
  • core compression uses zstd instead of zip: compression is about 4 times faster, decompression about two times, compression saves ±10% of size.
  • trace now supports tracing macro functions, compiler-macro functions, individual methods and local functions (flet and labels) (SBCL 2.2.5)
  • the SBCL repository reached 20,000 commits.
  • Prebuilt SBCL binary for Android (Termux) (unofficial)
• Clasp 2.0.0 released
  • This is Common Lisp on LLVM.
  • Christian Schafmeister talk - brief update about his “molecular lego” supported by his Lisp compiler
  • there’s less funding than in the 80s, but still funding: “CLASP was supported by The Defense Threat Reduction Agency, The National Institutes of Health, The National Science Foundation”.
• ECL:
  • LQML: a lightweight ECL binding to QML (both Qt5 and Qt6) derived from EQL5
  • tested on the following platforms: Linux, macOS, android, iOS.
  • ECL targetting WASM via Emscripten - preliminary support
• LispWorks Personal Edition updated to version 8.0.1, incl. native Apple Silicon version
  • additional important LispWorks 8 patch for macOS 12.6
• Major New release of Allegro Common Lisp Express Edition for 10.1
  • Browser-based IDE for Linux and macOS
  • no syntax highlighting for the editor though :S
  • Applications built using Common Graphics can use browsers for delivery. Firefox, Chrome, Safari, Edge and many other browsers are supported.
  • New platforms: aarch64, x86_64
  • download: https://franz.com/downloads/clp/survey
• ABCL 1.9.0 was released in May.
  • “ABCL 1.9.0 has been best tested on the openjdk8, openjdk11, and openjdk17 runtimes. It will run other places but those are the best supported.”
• GNU Common Lisp 2.6.13 released, after 8 years.

New implementation! It’s 2022 and people start new CL implementations.

• NPT - an implementation of ANSI Common Lisp in C.
  • comments

See also:

• LCL, Lua Common Lisp - The goal of this project is to provide an implementation of Common Lisp that can be used wherever an unmodified Lua VM is running.
  • not a complete implementation.

They are doing great work to revive a Lisp machine:

• 2022 Medley Interlisp Annual Report

Medley Interlisp is a project aiming to restore the Interlisp-D software environment of the Lisp Machines Xerox produced since the early 1980s, and rehost it on modern operating systems and computers. It’s unique in the retrocomputing space in that many of the original designers and implementors of major parts of the system are participating in the effort.

Paolo Amoroso blog post: my encounter with Medley Interlisp.

Jobs

I won’t list expired job announces, but this year Lispers could apply for jobs in: web development(WebCheckout, freelance announces), cloud service providers (Keepit), big-data analysis (Ravenpack, and chances are they are still hiring)), quantum computing (HLR Laboratories), AI (Mind AI, SRI International), real-time data aggregration and alerting engines for energy systems (3E); for a startup building autism tech (and using CLOG already); there have been a job seeking to rewrite a Python backend to Common Lisp (RIFFIT); there have been some bounties; etc.

Prior Lisp experience was not 100% necessary. There were openings for junior and senior levels, remote and not remote (Australia for “a big corp”, U.S., Spain, Ukraine...).

Comes a question:

• Any Ideas for strategies to find jobs with lisp?

I remind the reader that most Lisp jobs do not have a public job posting, instead candidates are often found organically on the community channels: IRC, Twitter, Discord, Reddit... or teams simply train their new developer.

In 2022 we added a few companies to the ongoing, non-official list on awesome-lisp-companies. If your company uses Common Lisp, feel free to tell us on an issue or in the comments!

For example, Feetr.io “is entirely Lisp”.

Lisp was a conscious decision because it allows a small team to be incredibly productive, plus the fact that it’s a live image allows you to connect to it over the internet and poke and prod the current state, which has really allowed a much clearer understanding of the data.

They post SLY screenshots on their Twitter^^

Evacsound (HN):

We’re using CL in prod for an embedded system for some years now, fairly smooth sailing. It started out as an MVP/prototype so implementation was of no concern, then gained enough velocity and market interest that a rewrite was infeasible. We re-train talent on the job instead.

Pandorabots, or barefootnetworks, designing the Intel Tofino programmable switches, and more.

Projects

• Ultralisp now supports tags. We can browse a list of projects under a tag.
  • Ultralisp is a Quicklisp distribution that ships every five minutes.
  • see also CLPM for a new package manager.

Language libraries

• Typo: A portable type inference library for Common Lisp, by Marco Heisig.
• Testing:
  • fiveam-matchers - An extensible, composable matchers library for FiveAM.
  • testiere - TDD-like dev for Common Lisp. Tests are included at the top of a defun/t form. When you recompile your functions interactively, the tests are run.
  • melisgl/try test framework. - “it is what we get if we make tests functions and build a test framework on top of the condition system.”
  • journal - A Common Lisp library for logging, tracing, testing and persistence
  • 40ants-critic - a wrapper around LISP-CRITIC which provides a better interface to analyze ASDF systems and a command-line interface.
• CLEDE - the Common Lisp Emacs Development Environment
  • “The idea is to supply features that other language with and static analyzer have, like refactoring and code generation.”
• easy-macros - An easy way to write 90% of your macros.
• Quicklisp doctor - a program that examines the quicklisp installation.
• more-cffi: Additional helper macros for the cffi project

Editors, online editors, REPLs, plugins

• 🚀 Parrot - A cross-platform Common Lisp editor.
  • built in Rust with Tauri, CodeMirror, the Slynk server.
  • “A hobby project started in Summer 2022. It aims to be an editor for Common Lisp (SBCL), that mostly works out of the box.”
  • in development, untested on Linux and Mac.
• Alive LSP for VSCode v0.1.9 · Add initial step debugger support
• Common Lisp at Riju, a fast online playground for every programming language.
• Codewars (code training platform) now has Common Lisp (SBCL 2.0.9)
• Mobile app “cl-repl” (LQML) to replace “CL REPL” (EQL5)
• ☆ slime-star - SLIME configuration with some extensions pre-installed.
  • a Lisp System Browser
  • SLIME Doc Contribs
  • Quicklisp Systems browsers
  • Quicksearch utility
  • Slime breakpoints - Inspect objects from their printed representation in output streams.
  • custom utilities and menus.
• parachute-browser: A lightweight UI for using the Parachute testing framework in LispWorks

New releases:

• Lem editor 1.10.0: lsp-mode by default, multiple cursors, sql mode, and more.
  • Lem is a general purpose editor written in Common Lisp. It works for many languages thanks to its LSP client.

Concurrency

• 🚀 New version of the Sento Actor Framework released with a few new goodies in future handling. Nicer syntax and futures can now be mapped.
  • in v2.2.0: stashing and replay of messages.
  • in v1.12: “Shutdown and stop of actor, actor context and actor system can now wait for a full shutdown/stop of all actors to really have a clean system shutdown.”
• Writing distributed apps with cl-etcd

See also lisp-actors, which also does networking. It looks like more of a research project, as it doesn’t have unit-tests nor documentation, but it was used for the (stopped) Emotiq blockchain.

Discussions:

• Concurrency: Common Lisp vs Clojure
• Moving from the BEAM to Common Lisp: What are my concurrency options?

Databases

• DuckDB Lisp bindings
• ndbapi: Common Lisp bindings to the C++ NDB API of RonDB
• CLSQL released under a non-restrictive license
• Document Store/DB Implemented in Common Lisp

More choices: awesome-cl#databases.

Delivery tools

There has been outstanding work done there. It is also great to see the different entities working on this. That includes SBCL developers, Doug Katzman of Google, and people at HRL Laboratories (also responsible of Coalton, Haskell-like on top of CL).

Have you ever wanted to call into your Lisp library from C? Have you ever written your nice scientific application in Lisp, only to be requested by people to rewrite it in Python, so they can use its functionality? Or, maybe you’ve written an RPC or pipes library to coordinate different programming languages, running things in different processes and passing messages around to simulate foreign function calls.

[...] If you prefer using SBCL, you can now join in on the cross-language programming frenzy too.

• 🎉 sbcl-librarian - An opinionated interface for creating C- and Python-compatible shared libraries in Common Lisp with SBCL. Requires SBCL version >2.1.10.
  • introductory blogpost: Using Lisp libraries from other programming languages - now with SBCL.
  • HN comments (67)
• 🚀 alien-works-delivery - WIP system for delivering Common Lisp applications as executable bundles. For now it only supports AppImage format for Linux and MSIX for Windows, but .APK for Android and later MacOSX and iOS bundle formats are planned too.
• Support for compiling Common Lisp code using bazel.io, by the CL team at Google.

Games

Kandria launches on Steam on the 11th of January, in two days!

• 2022 for Kandria in review

  • https://kandria.com/
  • HN comments (32)
  • They Successfully completed their Kickstarter
  • They received the Pro Helvetia grant
  • An Overview of Kandria’s Development with Lisp

• Jettatura - old school RPG game released on Steam this October - built in Common Lisp (“a decade in the making”)

• cl-defender - Toy Game using Common Lisp and Raylib.

🎥 Kandria trailer.

Graphics, GUIs

We saw the release of fresh bindings for Gtk4.

• 🚀 Writing beautiful GUI with Common Lisp and GTK4

We had bindings for Qt5... but they are still very rough, hard to install so far.

• CommonQt 5 - Qt5 bindings.

Also:

• I made a Wayland client from scratch for Common Lisp
• vk - CFFI bindings to Vulkan

History:

• Izware Mirai is available on the Internet Archive (no source code)
  • Hacking Lisp on Mirai (screencast)

But an awesome novelty of 2022 is Kons-9.

Kons-9, a new 3D graphics project

🚀 A new 3D graphics project: Kons-9.

• reddit announce on August
  • HN comments (80)

The idea would be to develop a system along the lines of Blender/Maya/Houdini, but oriented towards the strengths of Common Lisp.

I’m an old-time 3D developer who has worked in CL on and off for many years.

I don’t consider myself an expert [...] A little about me: · wrote 3D animation software used in "Jurassic Park" · software R&D lead on "Final Fantasy: The Spirits Within" movie · senior software developer on "The Hobbit" films.

• kons-9, a Common Lisp 3D Graphics Project
• 🎥 trailer
• author’s blog posts
• and see also his screencasts below.

Interfaces with other languages

• py4cl2-cffi: CFFI based alternative to py4cl2.
  • it does one big new thing: it supports passing CL arrays by reference. That means we actually have access to numpy, scipy, and friends.
  • “If py4cl2-cffi reaches stability, and I find that the performance of (i) cffi-numpy, (ii) magicl, as well as (iii) a few BLAS functions I have handcrafted for numericals turn out to be comparable, I might no longer have to reinvent numpy.” @digikar
• Small update to RDNZL (CL .NET bridge by Edi Weitz)
  • forked project, added support for Int16, fixed Int64, re-building the supporting DLLs.
  • see also: Bike
• jclass: Common Lisp library for Java class file manipulation

For more, see awesome-cl.

Numerical and scientific

• 🚀 new Lisp Stats release
  • “emphasis on plotting and polishing of sharp edges. data-frames, array operations, documentation.”
  • HN comments (55)
  • ” I’ve been using lisp-stat in production as part of an algorithmic trading application I wrote. It’s been very solid, and though the plotting is (perhaps was, in light of this new release) kinda unwieldy, I really enjoyed using it. Excited to check out the newest release.”
  • “For example, within Lisp-Stat the statistics routines [1] were written by an econometrician working for the Austrian government (Julia folks might know him - Tamas Papp). It would not be exaggerating to say his job depending on it. These are state of the art, high performance algorithms, equal to anything available in R or Python. So, if you’re doing econometrics, or something related, everything you need is already there in the tin.”
  • “For machine learning, there’s CLML, developed by NTT. This is the largest telco in Japan, equivalent to ATT in the USA. As well, there is MGL, used to win the Higgs Boson challenge a few years back. Both actively maintained.”
  • “For linear algebra, MagicCL was mention elsewhere in the thread. My favourite is MGL-MAT, also by the author of MGL. This supports both BLAS and CUBLAS (CUDA for GPUs) for solutions.”
  • “Finally, there’s the XLISP-STAT archive. Prior to Luke Tierney, the author of XLISP-Stat joining the core R team, XLISP-STAT was the dominate statistical computing platform. There’s heaps of stuff in the archive, most at least as good as what’s in base R, that could be ported to Lisp-Stat.”
  • “Common Lisp is a viable platform for statistics and machine learning. It isn’t (yet) quite as well organised as R or Python, but it’s all there.”
• numericals - Performance of NumPy with the goodness of Common Lisp
  • numericals is “a high performance basic math library with vectorized elementary and transcendental functions. It also provides support for a numpy-like array object through dense-arrays and static-dispatch through the CLTL2 API.”
  • the post is about numericals, dense-arrays, magicl, numcl, py4cl/2...
  • the author’s comparison and wish-list of features for a Common Lispy approach to a (better) Numpy
• MGL-MAT - a library for working with multi-dimensional arrays which supports efficient interfacing to foreign and CUDA code. BLAS and CUBLAS bindings are available.
• hbook - Text-based histograms in Common Lisp inspired by the venerable HBOOK histogramming library from CERN.

New releases:

• Maxima 5.46 was released.
  • “Maxima is a Computer Algebra System comparable to commercial systems like Mathematica and Maple. It emphasizes symbolic mathematical computation: algebra, trigonometry, calculus, and much more.”
  • see its frontends, for example WxMaxima.

Call to action:

• 📢 Uncle Stats Wants You

Web

Screenshotbot (Github) was released. It is “a screenshot testing service to tie with your existing Android, iOS and Web screenshot tests”.

It is straightforward to install with a Docker command. They offer more features and support with their paid service.

LicensePrompt was released. It is “a single place to track all recurring software and IT expenses and send relevant reminders to all interested people”. It’s built in CL, interface with HTMX.

• Lisp Interpreter in a browser using WASM
• Show HN: Common Lisp running natively over WebAssembly for the first time
  • HN comments (64)

Libraries:

• jingle: Common Lisp web framework with bells and whistles (based on ningle)
  • jingle demo: OpenAPI 3.x spec, Swagger UI, Docker and command-line interface app with jingle.
• ciao: Ciao is an easy-to-use Common Lisp OAuth 2.0 client library. It is a port of the Racket OAuth 2.0 Client to Common Lisp.
• stepster: a web scraping library, on top of Plump and Clss (new in QL)
• openrpc: Automatic OpenRPC spec generation, automatic JSON-RPC client building
• HTTP/2 implementation in Common Lisp

Skeletons:

• cl-cookieweb: my project skeleton to start web projects. Demo in video. I am cheating, the bulk of it was done in 2021.
  • “Provides a working toy web app with the Hunchentoot web server, easy-routes, Djula templates, styled with Bulma, based on SQLite, with migrations and an example table definition.”
  • if you don’t know where to start for web dev in CL, enjoy all the pointers of this starter kit and find your best setup.
  • see also this web template by @dnaeon, and check out all his other Lisp libraries.

Bindings:

• 👍 lisp-pay: Wrappers around various Payment Processors (Paypal, Stripe, Coinpayment)
• lunamech-matrix-api: Implementation of the Matrix API, LunaMech a Matrix bot

Apps:

• Ackfock - a platform of mini agreements and mini memos of understanding (built with CLOG, closed source).
• todolist-cl: a nice looking todolist with a web UI, written in Common Lisp (and by a newcomer to CL, to add credit)

I don’t have lots of open-source apps to show. Mines are running in production and all is going well. I share everything on my blog posts. I also have an open-source one in development, but that’s for the 2023 showcase :D

CLOG

🚀 The awesome novelty of 2022 I spoke of in the introduction is CLOG, the Common Lisp Omnificent GUI:

• Native Desktop Executables for CLOG
• CLOG and CLOG Builder Release 1.8 - IDE for Common Lisp and CLOG
• CLOG on Android, APK download

I know of one open-source consequent CLOG app: mold-desktop, in development.

I’m developing a programmable desktop and a bookmarks manager application with CLOG. I think I know one of the things that make CLOG user interfaces so easy to develop. It is that they are effortlessly composable. That’s it for now :)

@mmontone

New releases

There are lots of awesome projects in music composition, including OpusModus and OpenMusic which saw new releases. I also like to cite ScoreCloud, a mobile app built with LispWorks, where you whistle, sing or play your instrument, and the app writes the music score O_o

• 🎵 Opusmodus 3.0, Music Composition System, macOS Intel & Apple Silicon native, based on LispWorks, just released
• 🎵 OpenMusic 7.0, now also native for M1 Macs, visual programming language designed for music composition
• Consfigurator, a Common Lisp based declarative configuration management system, reaches v1.0
• April 1.0 released - APL in Common Lisp.
• cl-cmark approaching stable release
• cl-git: a Common Lisp CFFI interface to the libgit2 library
• tinmop 0.9.9.1, a terminal client for gopher, gemini, kami and pleroma.
  • look here for how to build full-screen terminal applications in Common Lisp.
• clingon - new release, new features (command-line options parser)
  • a full-featured options parser. Supports sub-commands, bash and zsh completions, many arguments types...

See awesome-cl and Cliki for more.

(re) discoveries

• The math pastebin Mathb.in is written in Common Lisp
• TIL that PTC’s Creo Elements/Direct 3D CAD modeling software has a free version for Windows. “7+ million lines of Common Lisp code”, used by Eterna for their watches.
  • that’s a huge software. 7 million lines. There’s a free version to try out!

Articles

Graphics

• Playing with raycasting
• Playing with graphics - a simple game with SDL2
• Multipass Translator for CLIM
• McCLIM: Implementing a simpleminded REPL from scratch

Tooling

• Debugging Lisp: trace options, break on conditions
  • HN comments (28), I learned, again, new tips.
• Developing Common Lisp using GNU Screen, Rlwrap, and Vim
  • again, learned new tips. The --remember flag of rlwrap allows to TAB-complete whatever was previously typed at the prompt. That’s dumb autocompletion, but autocompletion nonetheless. My summary.
• Configuring Slime cross-referencing
• SLIME Compilation Tips
• How to save lisp and die from Slime or Sly. trivial-dump-core
  • that may be super useful, at least if answers a question everybody has one time or another. I should try it more.
• How to approach a Lisp sandbox environment
• log4cl questions
• A StumpWM debugger
• Windows environment for SBCL
• Securing Quicklisp through mitmproxy
  • because Quicklisp doesn’t use HTTPS. Here’s how to add security. The new CLPM uses HTTPS.
• Lisp in Vim

Scripting

• Day 3: Roswell: Common Lisp scripting, by E. Fukamachi.
• Qlot tutorial with Docker
  • Qlot v1.0.0 was officially released.
• Day 4: Roswell: How to make Roswell scripts faster
• Day 5: Roswell: Hidden feature of “-s” option
• Writing scripts in lisp?
  • a legitimate question. Many tips.
  • where I show my very new and un-released CIEL scripting facility. Batteries included for Common Lisp. That’s for 2023 but you can be an early adopter :p
• Lisp for Unix-like systems
  • in particular, read this answer by /u/lispm to learn about past and present attempts and solutions.
  • will the OP manage to make WCL work? “WCL: Delivering efficient Common Lisp applications under Unix”, an implementation “tailored to zillions of small Lisp programs. Make sure that much of Lisp is shared memory. Be able to create shared memory libraries/applications”.
• (SBCL) Is there a way to detect if lisp code was run by –script vs interactively?

Around the language

• 🚀 Using Coalton to Implement a Quantum Compiler
  • Coalton brings Haskell-like type checking on top of Common Lisp. Groundbreaking. They use it for their quantum compiler.
  • it is still under development and there is their Discord to talk about it.
• Eliminating CLOS for a 4.5x speedup
  • sometimes it’s easy, use a struct.
• The empty list. “Someone has been getting really impressively confused and cross on reddit about empty lists, booleans and so on in Common Lisp, which led us to a discussion about what the differences really are. Here's a summary which we think is correct.”
• Are there any tutorials or examples people would recommend to get started with unit testing common lisp?
• Fun with Macros: Do-File
• Series tips and tricks
• STATIC-LET, Or How I Learned To Stop Worrying And Love LOAD-TIME-VALUE

History:

• Old LISP programs still run in Common Lisp

Web related

• Common Lisp web development and the road to a middleware
• 🚀 Lisp for the web: building one standalone binary with foreign libraries, templates and static assets
  • what I managed to do here, with the help of the community, represents a great step forward for my Lisp web stack.
  • I can build a standalone binary for my web app, containing all static assets (a patch was required for the Djula HTML templating engine), so I can just rsync it to my server and it works.
  • a standalone binary is easy to integrate into an Electron window. Stay tuned.
• Lisp for the web: deploying with Systemd, gotchas and solutions
  • all the little gotchas I faced are now google-able.
• Woo: a high-performance Common Lisp web server, by E. Fukamachi.
• HTTP over unix sockets in Common Lisp. By the same author:
  • Using TailwindCSS in Common Lisp web apps without Node.js tooling
  • Using SVGs in Common Lisp web apps with Djula
  • Create a Common Lisp Web app using ningle
  • Using environment variables with cl-dotenv in Common Lisp
  • Mito: An ORM for Common Lisp
  • Running docker commands from Common Lisp REPLs
  • he also put up several little examples of a Common Lisp web app with HTMX, such as cl-beer. His stack: Caveman, Djula templates, HTMX for the interactivity. Thanks again, they are super useful.
• A new static site generator
• Web Development with CL Backend and ClojureScript Frontend
• Update on gRPC Server - gRPC now has the ability to create servers based on protocol buffer service descriptors.
• 🚀 Almost 600,000 entries per second from Lisp to Apache Accumulo over Apache Thrift
• Writing an interactive web app in Common Lisp: Hunchentoot then CLOG
• Review of CL Json Libraries, updated 15 Jan 2022

Call for action:

• 📢 JACL call for collaboration

Other articles

• Astronomical Calculations for Hard Science Fiction in Common Lisp, by Fernando Borretti
  • HN comments (42) (yes, 42)
  • we miss you Fernando. Oh, he just released a new library: lcm: “Manage your system configuration in Common Lisp. Think of it as Ansible, for your localhost”
• 👀 A LISP REPL Inside ChatGPT
• FreeBSD Jail Quick Setup with Networking for a Common Lisp environment
• Proceedings, Seventeenth International Workshop on the ACL2 Theorem Prover and its Applications, 2022.
  • ACL2 v8.5 was released this year too.
• Case study - house automation tool - part 2 - getting serial

Screencasts and podcasts

New videos by me:

• 🚀 Debugging Lisp: fix and resume a program from any point in stack 🎥
• How to request the GitHub API: demo of Dexador, Jonathan, Shasht (and Serapeum), livecoding in Emacs and SLIME.
• How to create, run, build and load a new Lisp project, with my fully-featured project skeleton.
• Interactively Fixing Failing Tests in Common Lisp

by Gavin Freeborn:

• Why Learn Lisp In 2022?
• Why Are Lisp Macros So Great!?
• Lisp Is More Than Just A Language It’s An Environment
• Rewrite Your Scripts In LISP - with Roswell
• Creating Your First Lisp Project - Quicklisp, asdf, and Packages
• Unleash the REPL with SLY
• Lem: what if Emacs was multithreaded

KONS-9 series:

• Kaveh’s Common Lisp Lesson 01: points and shapes
• Common Lisp OpenGL programming tutorial #12 - Animation & Hierarchies

CLOG series:

• CLOG Extra 3 - The CLOG Project System for CLOG and NON-CLOG projects
• CLOG Extra 4 - All about panels
• Repl Style. Dev visually with CLOG Builder
  • “This is amazing work! Having Pharo/Smalltalk capabilities with Common Lisp is fascinating.”

CL study group:

• Roswell (part II)

Others:

• Common Lisp Game of Life visualisation with LispWorks and CAPI
• Nyxt on GambiConf (The hacker’s power-browser, written in Common Lisp)
• Far and Away the Simplest Tutorial on Macros

and of course, find 3h48+ of condensed Lisp content on my Udemy video course! (I’m still working on new content, as a student you get updates).

Aside screncasts, some podcasts:

• CORECURSIVE #076: LISP in Space, With Ron Garret ← a podcast, trascribed with pictures
  • HN comments (98)
  • same topic: NASA Programmer Remembers Debugging Lisp in Deep Space
• The Array Cast - Andrew Sengul, creator of the April language tells us about the advantages of combining Lisp and APL

Other discussions

Community

• 💌 Impressions of Common Lisp and the Lisp Community
  • “So, thank you. I'm glad I found this group (and some other Lisp groups) on Reddit. You guys are a real blessing.”
• CL Community(?) Norms
• In what domains is common lisp used in 2022?
• What are you working on?
• A brief interview with Common Lisp creator Dr. Scott Fahlman

Learning Lisp

• Advice on professional development in Common Lisp?
• Trying to get into Lisp, Feeling overwhelmed
• Looking for good common lisp projects on github to read?
  • this GitHub advanced search searches for defclass in 4 users projects: Shinmera, 40ants, Ruricolist, edicl (maintained Edi Weitz libraries):
• Teach lisp to high schoolers?
• Why buy LispWorks?
  • it’s expensive (there’s a free limited edition), but all this feedback is much interesting.
• Writing robust software in CL
  • talking Erlang, Actors libraries.
  • Moira - Monitor and restart background threads.
• What should a new programmer writing in Common Lisp know about security?

Common Lisp VS ...

• what were the lessons you learned from programming in Lisp that carried over to other languages (more on the imperative side)?
• For serious/industrial/business use, in what ways do you think Common Lisp beat Clojure, and vice versa?
  • and: When should I choose Common Lisp over Clojure (for business), and vice versa?
• Common Lisp VS Racket (openly biased over CL, with testimonies of lispers knowing both)
  • HN comments (143): “I’m a heavy user of Common Lisp, and I only dabble in Racket from time to time. While Common Lisp is my tool of choice for a lot of reasons stated in the post, since the post largely skews favorably toward Common Lisp, I’ll offer two things that Racket shines at.”
• Why not: from Common Lisp to Julia
  • HackerNews comments (200) (for this post and the post it responds to)
  • The author of the first post said he was “migrating from Common Lisp to Julia as [his] primary programming language”. He was starting “to grow increasingly frustrated with various aspects of the language” and the CL ecosystem. Many of his arguments were harsh towards CL and didn’t mention existing better alternatives.
  • Two months later, in another blog post, he admits the previous article “was unfair to both languages”, and while “[he] is still mostly using Julia”, “Common Lisp is still on the table for [him]“.
  • On LiberaChat, he says he’s back from Julia to Common Lisp: “As of yesterday, I’m back to making libraries in CL. I cannot consider Julia as a serious language anymore, for it has deceived me after 7 years of research.”
  • Welcome back and thanks to him for the past and future CL libraries.
• Is Lisp too malleable to use for serious development?
• Why Lisp? (version française). “We will show how our own LispE interpreter was implemented.”
  • HN comments (200)

Thanks everyone, happy lisping and see you around!

All my code is stored in various Git repositories. When Git formats a diff between two objects, it generates a list of hunks, or groups of changes.

Each hunk can be displayed with a title which is automatically extracted. Git ships with support for multiple languages, but Lisp dialects are not part of it. Fortunately Git lets users configure their own extraction.

The first step is to identify the language using a pattern applied to the filename. Edit your Git attribute file at $HOME/.gitattributes and add entries for both Emacs Lisp and Common Lisp:

*.lisp diff=common-lisp
*.el diff=elisp

Then edit your Git configuration file at $HOME/.gitconfig and configure the path of the Git attribute file:

[core]
    attributesfile = ~/.gitattributes

Finally, set the regular expression used to match a top-level function name:

[diff "common-lisp"]
    xfuncname="^\\((def\\S+\\s+\\S+)"
    
[diff "elisp"]
    xfuncname="^\\((((def\\S+)|use-package)\\s+\\S+)"

For Lisp dialects, we do not just identify function names: it is convenient to identify hunks for all sorts of top-level definitions. We use a regular expression which captures the first symbol of the form and the name that follows.

Of course you can modifiy these expressions to identify more complex top-level forms. For example, for Emacs Lisp, I also want to identify use-package expressions.

You can see the result in all tools displaying Git diffs, for example in Magit with Common Lisp code:

Or for my Emacs configuration file:

Hunk titles, highlighted in blue, now contain the type and name of the top-level construction the changes are associated with.

A simple change, but one which really helps reading diffs.

In case you missed the yearly update last week: Kandria will release in one week from today, on January 11th, 15:00 CET / 09:00 EST. I hope you're as excited to play it as we are to finally get it into your hands!

Please remember to wishlist it on Steam to make sure you don't miss it!

It's that time of the year again! The end of it. And what a year it's been for Kandria. We're now less than two weeks away from the release. Yikes! Or should I say, woah! Well, let's take a moment and look at some of all of the things that happened, before we look at what the future may possibly hold in store for us. At least, if I have anything to say about it.

Honestly, so many things happened that I barely remember most of them. I had to go back through the monthly reviews to remember all of it. But then again, I've always been rather terrible at remembering things that far back in any chronologically complete manner. I won't be going over stuff in chronological order, either, but instead will touch on a bunch of individual topics. Let's start out with

Conferences

In 2022 we were present in person at quite a number of conferences:

• European Lisp Symposium in Portugal

• Digital Dragons in Poland

• Develop: Brighton in England

• Gamescom & Devcom in Germany

• HEROFest in Switzerland

We got a lot of useful feedback from random people trying the game out at the events, and also got to meet a lot of friendly and great developers from around the World. That all said, these conferences are also quite taxing and costly. We got the booth sponsored for all of them, but travel expenses are still not nothing, not to mention the work time. Travelling is also quite exhausting to me in general, so I hope I won't have to zip around the place quite as much next year.

However, I can already say that - unforeseen circumstances notwithstanding - I will be at the European Lisp Symposium in the Netherlands, and Tokyo Game Show in Japan.

Kickstarter & Steam Next Fest

In July we had a big double whammy of our Kickstarter and the Steam Next Fest, both launched at the same time. This was also our first big attempt at pushing for some marketing. We tried out Facebook ads, which weren't of much use at all. We also contacted a number of streamers and influencers, a few of which actually gave the new demo we had at the time a shot. It was a lot of fun to watch them play through it and chat with them as they did so.

Leading up to release and during it I imagine we'll have a few more such stream appearances. If you see a streamer playing Kandria, please don't hesitate to notify us in the Discord and I'd be happy to drop by in chat. Assuming I'm not asleep at the time, of course!

Anyway, the Kickstarter went rather well for us, and we managed to get funded in the first week. After that it was mostly coasting along, giving an update every now and again to keep spirits up and push for those stretch goals (more on that later).

I'm really happy that things weren't as hectic as they are often described as being, as I was still able to focus on developing the game. Losing a month of work would have made things quite a bit more troublesome later on.

Still, I'm also very well aware that the reason things went so well for us is mostly down to the fact that we had a rather low goal set, and that we had a lot of support from the programming, and especially Common Lisp community, many of which chipped in rather large sums. Thank you all very much!

I'm not sure that launching your Kickstarter alongside the Next Fest was a good idea. It's definitely a good idea to have a demo available for your Kickstarter, so that people can trust in your abilities to deliver a complete product, but I don't know if the cross-promotion idea worked out. It might be better to have the next fest part way through the Kickstarter or even at the end of it, or entirely separate them, to have two marketing beats rather than one bigger one. Still, it's impossible to say whether it would have gone better or worse overall if we had done it differently, so I'm not complaining. More just thinking about how I'd do this if there's going to be another similar thing in the future.

Development

At the start of the year we didn't even have the full game map ready yet, let alone all the assets, quests, or dialogue. A number of important features were also missing still, both in the engine and in the game itself.

Thinking back to it now it is kind of insane how much of the game was still missing. I know there's folks that can put together a full game in less than a year total, but that's usually much larger teams, or far smaller games.

There were quite a few painful stretches of arduous work. Filling out the entire map with interesting challenges was one, then going back and tiling it all was another. And finally going back again to add details and flairs everywhere was yet another. But, the game feels a lot more livelier and interesting now, so it was definitely worth all that extra work.

If you want to read up on all the nitty gritty of the development that happened during the year, you can browse back in time on the blog, or for even more detail, hop on by the mailing list.

One of the coolest parts for me was finally getting a Steam Deck (they're still not officially available in Switzerland), and seeing Kandria just... work for the most part. Having it be portable is really, really sweet. And it only took a couple of tweaks with the menuing to make it all run well. I would still love to also have the game on Switch, but we'll have to see about that later down the road.

Working up to Release

The game's been pretty much done since the end of November, and in the remaining time since then I've been working on translating the game into German. That took quite a bit of work, there's some 60'000 words, and I'm not the best at translating to begin with. The first draft of that is now done, and we should be good ready to get the game out there in both English and German by the release date.

Unfortunately there won't be any other languages for the foreseeable future. My funds have run very dry, and I need to save up again to be able to support the development of the next game (more on that later). However, if you're interested in localising the game yourself, you'll be able to do so soon. Please keep your eyes and ears open!

Aside from localisation work I've also ironed out some more bugs, cleaned up some stuff in the code base, developed an independent key distribution system so I can sell copies without being attached to a third-party platform, and added some more minor enhancements and changes along the way.

I really hope the release will go well, as far as I know there's only very minor outstanding issues.

The Release

So. Kandria is releasing on Wednesday, January 11th, 15:00 CET. It'll be released on Steam, Itch.io, and our website. All versions will be DRM-free, though we get the biggest cut of the revenue if you buy it directly from our website.

In addition to this, the Soundtrack will also be available on Steam, Itch.io, and Bandcamp, and on various streaming services such as Spotify.

If you were a backer of the Kickstarter campaign, you will receive your keys for the game and OST in the coming days.

Immediately on release I will be streaming the game on Steam and Twitch, so please join me there for a little celebration. After that I will be looking at any and all feedback that's coming in, and working on patches to address any fires that may be unveiled. And after I've addressed what I can, I think I'll take some more holidays to recenter myself and consider the coming year properly.

2023

Even after the release, my work on Kandria will not be done quite yet. There's two big post-release updates thanks to the Kickstarter stretch goals that will be coming:

1. Level Editor. The initial release already includes the development level editor, but it is a bit rough around the edges and needs more usability and stability improvements. Once that is done, there'll be another big patch update along with a community event to encourage people to make and share their own levels.

2. Modding Support. While the game's source code will be available on release already, the second post-launch update will focus on two things: presenting an explicit API for mods, documentation for people to write their own mods for Kandria, and an in-game mod browser supported by mod.io.

I cannot yet make any promises about when these updates will land, especially as I also need to start gearing up work for the next game project. That's right, I'm already planning and working on the next game, and I'm really excited about it. I don't want to reveal anything about it yet, but I think you'll be positively surprised when I do!

Since things are still a bit under covers at the moment I don't know if I'll be able to keep doing monthly roundups like this, though rest assured that I will keep you in the loop with any important developments, don't you worry about that.

You

I wanted to reserve this last section right at the very end of both the article and the year here just for you. Thank you so much. I know this is sappy, and I know this is cliche, and I know it is all of these things and many others, but I do genuinely feel blessed at this moment to have you reading about my work, and following along for such a long time. And the better you know me, the better you'll know how rare it is for me to express such genuine positivity, so I hope you will take it to heart and believe me when I say that I am very thankful to you, and I hope that you'll continue to follow my endeavours in the future as well.

Before I go, I have one last favour to ask of you: please share Kandria with your friends, colleagues, and groups. I know it may not seem like much, and I know it can feel awkward, but it is invaluable for someone like me that's just starting out in this industry. Even just a few more people can make a big difference. So please, share the Steam page, itch site, or our website with people.

And again, thank you. I hope you have a great new year.

The SLIME Emacs package for Common Lisp supports cross-referencing: one can list all references pointing to a symbol, move through this list and jump to the source code of each reference.

Removing automatic reference jumps

While cross-referencing is very useful, the default configuration is frustrating: moving through the list in the Emacs buffer triggers the jump to the reference under the cursor. If you are interested in a reference in the middle of the list, you will have to move to it, opening multiple buffers you do not care about as a side effect. I finally took the time to fix it.

Key bindings for slime-ref-mode mode are stored in the slime-xref-mode-map keymap. After a quick look in slime.el, it is easy to remove bindings for slime-xref-prev-line and slime-xref-next-line:

(define-key slime-xref-mode-map (kbd "n") nil)
(define-key slime-xref-mode-map [remap next-line] nil)
(define-key slime-xref-mode-map (kbd "p") nil)
(define-key slime-xref-mode-map [remap previous-line] nil)

If you are using use-package, it is even simpler:

(use-package slime
  (:map slime-xref-mode-map
      (("n")
       ([remap next-line])
       ("p")
       ([remap previous-line]))))

Changing the way references are used

SLIME supports two ways to jump to a reference:

1. With return or space, it spawns a buffer containing the source file and close the cross-referencing buffer.
2. With v, it spawns the source file buffer but keeps the cross-referencing buffer open and keeps it current.

This is not practical to me, so I made a change. The default action, triggered by return, now keeps the cross-referencing buffer open and switches to the source file in the same window. This way, I can switch back to the cross-referencing buffer with C-x b to select another reference without spawning buffers in other windows (I do not like having my windows hijacked by commands).

To do that, I need a new function:

(defun g-slime-show-xref ()
  "Display the source file of the cross-reference under the point
in the same window."
  (interactive)
  (let ((location (slime-xref-location-at-point)))
    (slime-goto-source-location location)
    (with-selected-window (display-buffer-same-window (current-buffer) nil)
      (goto-char (point))
      (g-recenter-window))))

Note the use of g-recenter-window, a custom function to move the current point at eye level. Feel free to use the builtin recenter function instead.

I then bind the function to return and remove other bindings:

(define-key slime-xref-mode-map (kbd "RET") 'g-slime-show-xref)
(define-key slime-xref-mode-map (kbd "SPC") nil)
(define-key slime-xref-mode-map (kbd "v") nil)

Much better now!

My friend Zyni pointed out that someone has been getting really impressively confused and cross on reddit about empty lists, booleans and so on in Common Lisp, which led us to a discussion about what the differences between CL and Scheme really are here. Here’s a summary which we think is correct.

A peculiar object in Common Lisp¹

In Common Lisp there is a single special object, nil.

• This represents both the empty list, and the special false value, all other objects being true.
• This object is a list and is the only list object which is not a cons.
• As such this object is an atom, and again it is the only list object which is an atom.
• You can take the car and cdr of this object: both of these operations return the object itself.
• This object is also a symbol, and it is the only object which is both a list and a symbol.
• The empty list when written as an empty list, (), is self-evaluating.

Some comments.

• It is necessary that there be a special empty-list object which is a list but not a cons: the things which are not necessary are that it be a symbol, and that it represent falsity.
• Combining the empty list and the special false object can lead to particularly good implementations perhaps.
• The implementation of this object is always going to be a bit weird.
• It is clear that the empty list cannot be any kind of compound form so requiring it to be quoted — requiring you to write '() really — serves no useful purpose. Nevertheless I (Tim) would probably rather CL did that.
• Not having to quote nil on the other hand is not at all strange: any symbol can be made self-evaluating simply by (defconstant s 's), for instance.
• The graph of types in CL is a DAG, not a tree: it is not at all strange that there is an object whose type is both list and symbol.

Some entirely mundane things in Common Lisp

• There is a symbol, t which represents the canonical true value. Nothing is magic about this symbol in any way: it could be defined by (defconstant t 't).
• There is a type, boolean which could be defined by (deftype boolean () '(member nil t)), except that it is required that boolean be a recognisable subtype of symbol. All implementations we have tried recognise (member nil t) as a subtype of symbol, but the standard does not require them to do so. Additionally (type-of 't) must return boolean we think.
• There is a type, null, which could be defined by (deftype null () '(member nil)) or (deftype null () '(eql nil)), with the same caveats as above, and (type-of nil) should return null.
• There are types named t (top of the type graph) and nil (bottom of type graph).

These mundane things are just that: they don’t require implementational magic at all.

Three peculiar objects in Scheme

In Scheme there is an object, ().

• () is the special object that represents the empty list.
• It does not represent false.
• It is not a symbol.
• It is the only list object which is not a pair (cons): list? is true of it but pair? is false.
• You can’t take the car or cdr of it.
• It is not self-evaluatiing.

There is another object, #f.

• #f is the distinguished false value and is the only false value in Scheme, all other objects being true.
• It is not a symbol or a list but satisfies the boolean? predicate.
• It is self-evaluating.

There is another object, #t.

• #t represents the canonical true value, but all objects other than #f are true.
• It is not a symbol or a list but satisfies the boolean? predicate.
• It is self-evaluating.

Some comments. - Scheme does not have such an elaborate type system as CL and, apart from numbers, doesn’t really have subtype relations the way CL does.

A summary

CL’s treatment of nil clearly makes some people very unhappy indeed. In particular they seem to think CL is somehow inconsistent, which it clearly is not. Generally this is either because they don’t understand how it works, because it doesn’t work the way they want it to work, or (usually) both. Scheme’s treatment is often cited by these people as being better. But CL requires precisely one implementationally-weird object, while Scheme requires two, or three if you count #t which you probably should. Both languages have idiosyncratic evaluation rules around these objects. Additionally it’s worth understanding that things like CL’s boolean type mean essentially nothing implementationally: boolean is just a name for a set of symbols. The only thing preventing you from defining a type like this yourself is the requirement for type-of to return the type.

Is one better than the other? No: they’re just not the same. Certainly the CL approach carries more historical baggage. Equally certainly it is perfectly consistent, and changing it would break essentially all CL programs that exist.

Thanks to Zyni for most of this: I’m really writing it up just so we can remember it. We’re pretty confident about the CL part, less so about the Scheme bit.

Paredit is an Emacs package for structural editing. It is particularly useful in Lisp languages to manipulate expressions instead of just characters.

One of the numerous little features of Paredit is the automatic insertion of a space character before a delimiting pair. For example, if you are typing (length, typing ( will have Paredit automatically insert a space character before the opening parenthesis, to produce the expected (length ( content.

Paredit is smart enough to avoid doing so after quote, backquote or comma characters, but not after an unquote-splicing sequence (,@) which is quite annoying in languages such as Scheme or Common Lisp. As almost always in Emacs, this behaviour can be customized.

Paredit decides whether to add a space or not using the paredit-space-for-delimiter-p function, ending up with applying a list of predicates from paredit-space-for-delimiter-predicates.

Let us add our own. For more flexibility, we will start by defining a list of prefixes which are not to be followed by a space:

(defvar g-paredit-no-space-prefixes (list ",@"))

We then write our predicate which simply checks if we are right after one of these prefixes:

(defun g-paredit-space-for-delimiter (endp delimiter)
  (let ((point (point)))
    (or endp
        (seq-every-p
         (lambda (prefix)
           (and (> point (length prefix))
                (let ((start (- point (length prefix)))
                      (end point))
                  (not (string= (buffer-substring start end) prefix)))))
         g-paredit-no-space-prefixes))))

Finally we add a Paredit hook to append our predicate to the list:

(defun g-init-paredit-space-for-delimiter ()
  (add-to-list 'paredit-space-for-delimiter-predicates
               'g-paredit-space-for-delimiter))

(add-hook 'paredit-mode-hook 'g-init-paredit-space-for-delimiter)

Not only does it fix the problem for unquote-slicing, but it makes it easy to add new prefixes. For example I immediately added #p (used for pathnames in Common Lisp, e.g. #p"/usr/bin/") to the list.

I recently went back to Common Lisp to solve the daily problems of the Advent of Code. Of course it started with installing and configuring SLIME, the main major mode used for Common Lisp development in Emacs.

The most useful feature of SLIME is the ability to load sections of code into the Common Lisp implementation currently running. One can use C-c C-c to evaluate the current top-level form, and C-c C-k to reload the entire file, making incremental development incredibly convenient.

However I found the default configuration frustrating. Here are a few tips which made my life easier.

Removing the compilation error prompt

If the Common Lisp implementation fails to compile the file, SLIME will ask the user if they want to load the fasl file (i.e. the compiled form of the file) anyway.

I cannot find a reason why one would want to load the ouput of a file that failed to compile, and having to decline every time is quite annoying.

Disable the prompt by setting slime-load-failed-fasl to 'never:

(setq slime-load-failed-fasl 'never)

Removing the SLIME compilation buffer on success

When compilation fails, SLIME creates a new window containing the diagnostic reported by the Common Lisp implementation. I use display-buffer-alist to make sure the window is displayed on the right side of my three-column split, and fix my code in the middle column.

However if the next compilation succeeds, SLIME updates the buffer to indicate the absence of error, but keeps the window open even though it is not useful anymore, meaning that I have to switch to it and close it with q.

One can look at the slime-compilation-finished function to see that SLIME calls the function referenced by the slime-compilation-finished-hook variable right after the creation or update of the compilation buffer. The default value is slime-maybe-show-compilation-log which does not open a new window if there is no error, but does not close an existing one.

Let us write our own function and use it:

(defun g-slime-maybe-show-compilation-log (notes)
  (with-struct (slime-compilation-result. notes successp)
      slime-last-compilation-result
    (when successp
      (let ((name (slime-buffer-name :compilation)))
        (when (get-buffer name)
          (kill-buffer name))))
    (slime-maybe-show-compilation-log notes)))
    
(setq slime-compilation-finished-hook 'g-slime-maybe-show-compilation-log)`

Nothing crazy here, we obtain the compilation status (in a very SLIME-specific way, with-struct is not a standard Emacs Lisp macro) and kill the compilation buffer if there is one while compilation succeeded.

Making compilation less verbose

Common Lisp specifies two variables, *compile-verbose* and *load-verbose*, which control how much information is displayed during compilation and loading respectively.

My implementation of choice, SBCL, is quite chatty by default. So I always set both variables to nil in my $HOME/.sbclrc file.

However SLIME forces *compile-verbose*; this is done in SWANK, the Common Lisp part of SLIME. When compiling a file, SLIME instructs the running Common Lisp implementation to execute swank:compile-file-for-emacs which forces *compile-verbose* to t around the call of a list of functions susceptible to handle the file. The one we are interested about is swank::swank-compile-file*.

First, let us write some Common Lisp code to replace the function with a wrapper which sets *compile-verbose* to nil.

(let ((old-function #'swank::swank-compile-file*))
  (setf (fdefinition 'swank::swank-compile-file*)
        (lambda (pathname load-p &rest options &key policy &allow-other-keys)
          (declare (ignore policy))
          (let ((*compile-verbose* nil))
            (apply old-function pathname load-p options)))))

We save it to a file in the Emacs directory.

In Emacs, we use the slime-connected-hook hook to load the code into the Common Lisp implementation as soon as Slime is connected to it:

(defun g-slime-patch-swank-compilation-function ()
  (let* ((path (expand-file-name "swank-patch-compilation-function.lisp"
                                 user-emacs-directory))
         (lisp-path (slime-to-lisp-filename path)))
    (slime-eval-async `(swank:load-file ,lisp-path))))
    
(add-hook 'slime-connected-hook 'g-slime-patch-swank-compilation-function)

Quite a hack, but it works.