Planet Lisp

Zach BeaneNew version of ZS3 supports AWS4 auth

· 2 days ago

I just published ZS3 1.2.8. It’s available on my website and will be in the next Quicklisp dist update in May. The main difference is support for the latest AWS authentication system. This makes ZS3 work properly with the latest AWS regions like Frankfurt and Seoul.

If you have any trouble using it, please let me know!

(This work was for a paying customer; if you are interested in specific updates and features in ZS3 or any of my other software, get in touch.)

Daniel KochmańskiCreating a project homepage with the SCLP

· 9 days ago


In this short tutorial I'll describe how to bootstrap easily a project website. In fact that's what I did today with the Embeddable Common-Lisp website in order to provide the RSS feed and make putting there the news easier.

Additionally I'm showing here, how to create a standalone executable for coleslaw with clon after providing quicklisp independant bundle of systems.

Quick start

First clone the repository:

$ cd /home/p/ecl
$ git clone website
$ cd website

Now you should adjust the appropriate files. Edit .coleslawrc (file is self-explanatory), static pages and posts.

Each file with the extension *.page is a static page. pages/ is an example template with a static page - don't forget to link it in the .coleslawrc's sitenav section. Exact URL of the page is declared in the file's header.

Files named *.post represent blog/news posts which appear in the RSS feed. They are indexed and accessible from the root URL. Supported file formats are markdown, html and cl-who (if enabled).

When you're done, you could just load coleslaw with your favorite CL implementation, using Quicklisp load coleslaw and call the function main on the website directory:

(ql:quickload 'coleslaw)
(coleslaw:main "/home/p/ecl/website/")

We will take more ambitious road - we'll create a standalone executable with a proper command line arguments built from a clean bundle produced by Zach Bane's Quicklisp. CLI arguments will be handled by Clon - the Command-Line Options Nuker, an excellent deployment solution created by Didier Verna.

Creating the bundle

Bundle is a self-containing tree of systems packed with their dependencies. It doesn't require internet access or Quicklisp and is a preferred solution for the application deployment.

Some dependencies aren't correctly detected - Quicklisp can't possibly know, that our plugin will depend on the cl-who system, and it can't detect cl-unicode's requirement during the build phase - flexi-streams (this is probably a bug). We have to mention these systems explicitly.

Clon is added to enable the clonification (keep reading).

(ql:bundle-systems '(coleslaw flexi-streams
                     cl-who cl-fad
                   :to #P"/tmp/clw")

Clonifying the application

(in-package :cl-user)
(require "asdf")

(load "bundle")
(asdf:load-system :net.didierverna.clon)
(asdf:load-system :coleslaw)
(asdf:load-system :cl-fad)

(use-package :net.didierverna.clon)
(defsynopsis (:postfix "DIR*")
  (text :contents "Application builds websites from provided directories.")
  (flag :short-name "h" :long-name "help"
        :description "Print this help and exit."))

(defun main ()
  "Entry point for our standalone application."
  (when (getopt :short-name "h")
  (print (remainder))
  (handler-case (mapcar
                 #'(lambda (p)
                      (cl-fad:pathname-as-directory p)))
    (error (c) (format t "Generating website failed:~%~A" c)))

(dump "coleslaw" main)

You may generate the executable with sbcl and ccl (ecl has some problems with the coleslaw dependency - esrap, I'm working on it). I have used ccl, because it doesn't "derp" on the symbol exit and produces slighly smaller executable than sbcl.

Issue the following in the bundle directory (/tmp/clw):

ccl -n -l clonify.lisp

This command should create native executable named coleslaw in the same directory. On my host ccl produces binary with the approximate size 50M.

Executable usage

This is a very simple executable definition. You may extend it with new arguments, more elaborate help messages, even colors.

To generate a websites with sources in directories /tmp/a and /tmp/b you call it as follows:

./coleslaw /tmp/a /tmp/b

That's all. Deployment destination is set in the .coleslawrc file in each website directory.

Adding GIT hooks

You may configure a post-receive hook for your GIT repository, so your website will be automatically regenerated on each commit. Let's assume, that you have put the coleslaw standalone executable in place accessible with the PATH environment variable. Enter your bare git repository and create the file hooks/post-receive:

cd website.git

cat > hooks/post-receive <<EOF
########## CONFIGURATION VALUES ##########



if cd `dirname "$0"`/..; then
    cd $OLDPWD || exit 1
    exit 1

git clone $GIT_REPO $TMP_GIT_CLONE || exit 1

while read oldrev newrev refname; do
    if [ $refname = "refs/heads/master" ]; then
        echo -e "\n  Master updated. Running coleslaw...\n"
        coleslaw $TMP_GIT_CLONE


That's all. Now, when you push to the master branch your website will be regenerated. By default .gitignore file lists directory static/files as ignored to avoid keeping binary files in the repository. If you copy something to the static directory you will have to run coleslaw by hand.


Coleslaw is a very nice project simplifying managing project website with easy bootstrapping the site without any need to maintain working lisp process on the server (this is static content which may be served with nginx or apache) and allowing easy blogging (write a post in markdown and push to the repository).

Sample Common-Lisp Project is a pre-configured website definition with a theme inspired by the projects themes with some nice features, like RSS feed and blog engine (thanks to coleslaw).

We have described the process of creating a simple website, creating a standalone executable (which may be shared by various users) and chaining it with git hooks.


ECL NewsNew website look

· 9 days ago

I've imported the old archives and genearated ECL website with help of the coleslaw and the sclp. Now we have a proper RSS feed and posting news is less annoying then before.

For posterity, here is the ugly hack I've used to import archives from JSON:

(defparameter *archives-template*
title: ~A
date: ~A
author: ~A
format: md


(setf *json-posts*
  (with-open-file (f #P"/home/jack/linki/repo/ecl-website/static/files/misc/news-ecl-backup-2015-08-25.json"
                     :direction :input
                     :external-format '(:line-termination :cr :character-encoding :utf-8))
    (cl-json:decode-json f)))

(mapcar (let ((cnt 0))
          #'(lambda (post)
              (with-open-file (f (format nil "/tmp/archives/" (incf cnt))
                                 :direction :output
                                 :if-exists :supersede
                                 :external-format (make-external-format :line-termination :unix))
                (format f *archives-template*
                        (cdr (assoc :title post))
                        ;; (cdr (assoc :labels post))
                        (substitute #\- #\/
                                    (subseq (cdr (assoc :url post)) 40 47))
                        (let ((author (cdr (assoc :author post))))
                          (if (string-equal author "dkochmanski")
                              "jackdaniel" author))
                        (remove #\Return (cdr (assoc :text post)))))))
        (cdar *json-posts*))

You may find a guide how to use the Sample Common Lisp Project template for your own project here. The clnet theme is inspired by the css in most of the projects.

Best regards, Daniel

Quicklisp newsApril 2016 Quicklisp dist update now available.

· 9 days ago
Quicklisp's 67th monthly update is now available!

 Thank you to all the people who signed up for the recurring Supporter Club in the last month. The "big" fundraiser is still in the works, and I'll let you know more when there's more info to share.

New projects:
  • caveman2-widgets — Weblocks like widgets for caveman2. — LLGPL
  • cl-gamepad — Bindings to libstem_gamepad, allowing the handling of gamepad input. — Artistic
  • cl-geos — A CFFI wrapper of GEOS for performing geometric operations in Lisp. — Lisp-LGPL
  • cl-hash-table-destructuring — Hash table destructuring utils — WTFPL
  • cl-statsd — Statsd client in Common Lisp — MIT
  • cl-vhdl — My attempt to understand VHDL, and basicly make VHDL with Lisp-macro — MIT
  • electron-tools — Download, extract, and run Electron binaries. — MIT
  • flare — Easy particle systems with fine grained control. — Artistic
  • inlined-generic-function — MOP implementation of the fast inlinable generic functions dispatched in compile-time — LLGPL
  • liblmdb — Low-level LMDB bindings. — MIT
  • lmdb — Bindings to LMDB. — MIT
  • oclcl — oclcl is a library S-expression to OpenCL C. — LLGPL
  • prbs — A library of higher-order functions for generating Pseudo-Random Binary Sequences of (practically) any degree — MIT
  • — Common Lisp client — MIT
  • random-state — Portable random number generation. — Artistic
  • remote-js — Send JavaScript from Common Lisp to a browser. — MIT
  • sketch — Sketch is a Common Lisp framework for the creation of electronic art, computer graphics, visual design, game making and more. It is inspired by Processing and OpenFrameworks. — MIT
  • tm — Formalized Iteration Library for Common LISP — MIT
  • trivial-compress — Compress a directory. — MIT
  • trivial-string-template — A trivial string template library, inspired by Python's string.Template — MIT
  • trivial-ws — Trivial WebSockets. — MIT
Updated projects3d-vectorsalexandriaarchitecture.service-providerarrow-macrosasdf-flvasteroidsbinfixburgled-batteriesceramiccl+sslcl-anacl-asynccl-autowrapcl-bsoncl-geometrycl-hash-utilcl-itertoolscl-jpegcl-l10ncl-lexercl-llvmcl-marklogiccl-mockcl-mtgnetcl-mysqlcl-ohmcl-openglcl-opsresearchcl-pangocl-rabbitcl-rethinkdbcl-sdl2cl-slugcl-string-matchcl-stringscl-tasuketecl-tetris3dcl-wordcutcl-yaclyamlclack-errorsclassimpclinchcloser-mopclxcommon-doccommon-doc-plumpcommon-htmlcommonqtcopy-directorycroatoandartsclemailaddressdeedsdefpackage-plus,dissectdocparseresrapesrap-liquidfare-utilsfast-ioform-fiddlegendlhyperluminal-memjp-numeralkenzolakelisp-interface-librarylocal-timemacrodynamicsmcclimmitonorthosicatpaiprolog,parse-jspath-parseprojecturedqtoolsqtools-uiquickutilrpmrtg-mathrutilssafe-queuescalplsdl2kitserapeumsimple-date-timesimple-tasksskittersmugsnarksouthspinneretstaplestumpwmsxql,teepeedee2temporal-functionstriviatrivial-channelstrivial-extractusocketutilities.print-treewhichworkout-timer.

To get this update, use (ql:update-dist "quicklisp").

Michael MalisHow to Generate Self-Referential Programs

· 11 days ago

In this post, I am going to show you how to write programs that are self-referential. By self-referential, I mean programs which are able to obtain their own source code without any external input. In other words, they won’t just read from their own files. This post is based on section 6.1 of the book Introduction to the Theory of Computation.

Before we can start generating self-referential programs we are first going to need some techniques for generating programs in general. The first technique we need is a method of taking a given program and writing a second program that outputs the given program. As an example, given (+ 2 2), we would need to write a program that outputs (+ 2 2). In most languages this is easy. One way to do it in Lisp is to put a quote in front of the program:

'(+ 2 2)
=> (+ 2 2)

We are also going to need a function that automates this process. Such a function would take a program as its argument and return a new program that when ran, outputs the program that was originally passed to the function. In most languages doing this is fairly tricky. In Lisp, we can write this function easily through backquote:

(defun code-that-generates (program)

(code-that-generates '(+ 2 2))
=> '(+ 2 2)

If you don’t understand how backquote works, you can read this. Even though it’s for Emacs Lisp, everything there is still applicable to other Lisps. Just make sure that you understand that code-that-generates can be used to generate a program that outputs a given program.

Now that we have these two techniques, we can begin writing programs that are able to refer to themselves. The first self-referential program we will write will be an example of a quine. If you don’t know, a quine is a program that outputs its own source code. The quine we are going to write is made up of two parts, part A and part B, where part A is a function that is applied to part B:

(A B)

To describe how the quine works, it is easiest to start with part B. All that part B needs to do is return the source code of part A:

(A 'A)

Part A’s job is to take its own source code, and use it to obtain the source code of the entire quine. Since B is a program that outputs A, A can use code-that-generates on its own source code in order to obtain the source code of B. Once A has the source code of both A and B, it becomes trivial to combine the two to obtain the source code of the entire quine. Here is the complete quine, with the call to code-that-generates inlined:

((lambda (a)
   (let ((b `',a))
     `(,a ,b)))
 '(lambda (a)
    (let ((b `',a))
      `(,a ,b))))
((lambda (a)
   (let ((b `',a))
     `(,a ,b)))
 '(lambda (a)
    (let ((b `',a))
      `(,a ,b))))

Now this is where things start getting interesting. A quine can be thought of as a program that generates its own source code, and immediately returns it. What if instead of immediately returning its own source code, the quine applied a function to it first, and then returned the result of that. The steps for building such a program are almost exactly the same as the steps we took for building the quine. This time, there is a third part F, for the function we want to call. The structure of the program will look like the following:

(F AB)

Where AB has a similar structure to our quine. After breaking AB into the two parts, A and B, the program looks like the following:

(F (A B))

Part B in the above program has the same responsibilities as B in the quine, it returns the source code for A:

(F (A 'A))

Then once A has the source code for itself, it can use code-that-generates to obtain the source code for B. Now that it has the source of A and B, it is easy for it to construct AB. Once part A has the code for AB, it can easily generate the source of the entire program. Here is what the program becomes after filling in everything except F:

 ((lambda (a)
    (let ((b `',a))
      (let ((ab `(,a ,b)))
        `(F ,ab))))
  '(lambda (a)
     (let ((b `',a))
       (let ((ab `(,a ,b)))
         `(F ,ab))))))

What makes this so awesome is that F can be any function we want, and the above program will run F with the source code of the entire program! For example, replacing F with identity causes the program to become a quine:

 ((lambda (a)
    (let ((b `',a))
      (let ((ab `(,a ,b)))
        `(identity ,ab))))
  '(lambda (a)
    (let ((b `',a))
      (let ((ab `(,a ,b)))
        `(identity ,ab))))))
 ((lambda (a)
    (let ((b `',a))
      (let ((ab `(,a ,b)))
        `(identity ,ab))))
  '(lambda (a)
    (let ((b `',a))
      (let ((ab `(,a ,b)))
        `(identity ,ab))))))

But we can also do some much more impressive things. We can replace F with a function that lists its argument twice, and get a program that returns a list containing its own source code twice:

((lambda (x) (list x x))
 ((lambda (a)
    (let ((b `',a))
      (let ((ab `(,a ,b)))
        `((lambda (x) (list x x)) ,ab))))
  '(lambda (a)
     (let ((b `',a))
       (let ((ab `(,a ,b)))
         `((lambda (x) (list x x)) ,ab))))))


(((lambda (x) (list x x))
  ((lambda (a)
     (let ((b `',a))
       (let ((ab `(,a ,b)))
         `((lambda (x) (list x x)) ,ab))))
   '(lambda (a)
      (let ((b `',a))
        (let ((ab `(,a ,b)))
          `((lambda (x) (list x x)) ,ab))))))
 ((lambda (x) (list x x))
  ((lambda (a)
     (let ((b `',a))
       (let ((ab `(,a ,b)))
         `((lambda (x) (list x x)) ,ab))))
   '(lambda (a)
      (let ((b `',a))
        (let ((ab `(,a ,b)))
          `((lambda (x) (list x x)) ,ab)))))))

To make writing these self-referential programs easier, we can define a function that fills in F for us. It just requires a little nested backquote trickery.1

(defun self-referential-version-of (f)
     ((lambda (a)
        (let ((b `',a))
          (let ((ab `(,a ,b)))
            `(,',f ,ab))))
       '(lambda (a)
          (let ((b `',a))
            (let ((ab `(,a ,b)))
              `(,',f ,ab)))))))

(self-referential-version-of '(lambda (x) (list x x))
((lambda (x) (list x x))
 ((lambda (a)
    (let ((b `',a))
      (let ((ab `(,a ,b)))
        `(,'(lambda (x) (list x x)) ,ab))))
  '(lambda (a)
     (let ((b `',a))
       (let ((ab `(,a ,b)))
         `(,'(lambda (x) (list x x)) ,ab))))))

Now that we’ve got a function that can generate self-referential programs for us, I am going to show you how to build something called a quine-relay. A quine-relay is like a normal quine, except it passes through multiple languages. The quine-relay we are going to write is a Lisp program that outputs a C program that outputs the original Lisp program. All we have to do is write a function that takes its argument and writes a C program that prints the argument it was given. Then we can pass that function to self-referential-version-of to get the quine-relay! That’s it! Here is a program that will generate the quine-relay:

  '(lambda (self)
     (format t

"#include <stdio.h>~%int main(){printf(\"%s\",~(~s~));}"

             (remove #\newline (prin1-to-string self)))))

I’ve omitted the actual quine-relay for brevity, but you can find it here if you are curious. There are a few idiosyncrasies in the above program and in the quine-relay because of the differences in behavior between Lisp and C. For example, in C you can’t have multi-line strings, so it becomes easier to remove all of the newlines from the Lisp program, than it is to keep them.

And that’s all it takes to write self-referential programs. After seeing how easy it is to generate a quine-relay, it shouldn’t be hard to imagine how to write one with many more steps. You may even be able to get up to 100 if you work at it long enough.

The post How to Generate Self-Referential Programs appeared first on Macrology.

Wimpie NortjeKeep Quicklisp and Qlot out of your application binary.

· 17 days ago

'How do I create a Lisp application binary without including Quicklisp or Qlot?'

Quicklisp helps you work faster because it knows where to find libraries and Qlot keeps you mostly out of dependency hell. Without these two tools application development would be dead slow. However, applications distributed as standalone binaries should not contain them.

Quicklisp adds unnecessary code and it will likely try to create a ~/quicklisp directory on your user's machine. This can fail for any number of reasons (such as file permissions or lack of internet access) which will only cause headaches.

Qlot also adds unnecessary code and it will cause the same problems as Quicklisp.

The process of building a binary using Buildapp and Quicklisp without actually having them inside the final application is as follows:

  1. Load your project using the normal Quicklisp method, i.e. (ql:quickload).
  2. Export a manifest file which lists the absolute path to every ASDF system currently loaded into the Lisp image. This is done using (ql:write-asdf-manifest-file).
  3. Exit from Lisp.
  4. Use Buildapp to create your binary. The manifest file must be passed as a parameter, --load-system must be used to load your system and Quicklisp must NOT be loaded.

@xach explained the procedure in this Stack Overflow answer.

When Qlot is employed in a project, the above process becomes:

  1. Load Qlot.
  2. Load your project using Qlot's method, i.e. (qlot:quickload).
  3. Export the manifest file using (qlot:with-local-quicklisp (ql:write-asdf-manifest-file)).
  4. Exit from Lisp.
  5. Use Buildapp to create your binary.

Since Quicklisp is not aware of Qlot (ql:write-asdf-manifest) must be wrapped with (qlot:with-local-quicklisp) in order to generate paths to the project-local Qlot managed libraries instead of the system wide Quicklisp managed libraries.


This is an excerpt of a makefile from a project which uses Qlot. The makefile is used to create a standalone binary that does not contain Quicklisp or Qlot. The prep-quicklisp.lisp script is used to setup Quicklisp for loading the project system from an arbitrary location.

# CCL Flags for manifest build
MANIFEST_FLAGS =  --no-init 
MANIFEST_FLAGS += --batch 
MANIFEST_FLAGS += --load prep-quicklisp.lisp
MANIFEST_FLAGS += --eval '(ql:quickload :qlot)'
MANIFEST_FLAGS += --eval '(qlot:install :$(QL_SYSTEM))'
MANIFEST_FLAGS += --eval '(qlot:quickload :$(QL_SYSTEM))'
MANIFEST_FLAGS += --eval '(qlot:with-local-quicklisp :$(QL_SYSTEM) (ql:write-asdf-manifest-file \#P"$(MANIFEST)" :if-exists :supersede :exclude-local-projects nil))'
MANIFEST_FLAGS += --eval '(quit)'

# Buildapp settings
B_FLAGS =  --output $(OUTDIR)/$(TARGET)
B_FLAGS += --manifest-file $(MANIFEST)
B_FLAGS += --load-system $(QL_SYSTEM)
B_FLAGS += --entry app:main



Michael MalisLoops in Lisp Part 4: Series

· 18 days ago

This is part four of Loops in Lisp. Follow one of the following links for part one, two, or three).

One of the many advantages of programming in a functional style (by this, I mean manipulating your data through the operations, map, fold, and filter) is that your program winds up being made up a bunch of tiny and composable pieces. Since each piece is so small, usually only a few lines each, it becomes trivial to unit test the entire program. Additionally, it is easy to express new features as just the composition of several existing functions. One disadvantage of programming through map and friends, is that there is fairly large time penalty for allocating the intermediate results. For example, every time filter is called on a list, a new list needs to be allocated. These costs add up pretty quickly and can make a functional program much slower than its imperative equivalent.

One solution to this problem is laziness. Instead of allocating a new list every time an operation is performed on a list, you instead keep track of all of the transformations made on the list. Then when you fold over the list, you perform all of the transformations as you are folding over it. By doing this, you don’t need to allocate intermediate lists. Although laziness doesn’t allocate any intermediate lists, there is still a small cost for keeping track of the laziness. An alternative solution that makes functional programming just as fast as imperative programming is provided by the Series library.1 Series lets you write your program in a functional style without any runtime penalty at all!

Personally, the Series library is my favorite example of the magic that can be pulled off with macros. In short, Series works by taking your functional code and compiling it down into a single loop. In this loop, there is one step per transformation performed on the original list. The loop iterates over the values of the original sequence on at a time. On each iteration, the loop takes a single element, performs all of the transformations performed on the list on that single element, and then accumulates that value into the result according to the folding operation. This loop requires no additional memory allocation at runtime, and their is no time penalty either! As an example, here is a program that sums the first N squares, written using Series:

(defun integers ()
  "Returns a 'series' of all of the natural numbers."
  (declare (optimizable-series-function))
  (scan-range :from 1))

(defun squares ()
  "Returns a 'series' of all of the square numbers."
  (declare (optimizable-series-function))
  (map-fn t 
          (lambda (x) (* x x)) 

(defun sum-squares (n)
  "Returns the sum of the first N square numbers."
  (collect-sum (subseries (squares) 0 n)))

(sum-squares 10)
=> 385

The above code certainly looks functional, there are no side effects in sight. Now let’s look at the code generated by Series. Here is what the macroexpansion of collect-sum looks like:

(common-lisp:let* ((#:out-969 n))
  (common-lisp:let ((#:numbers-966
                     (coerce-maybe-fold (- 1 1) 'number))
                    (#:index-965 -1)
                    (#:sum-959 0))
    (declare (type number #:numbers-966)
             (type (integer -1) #:index-965)
             (type number #:sum-959))
       (setq #:numbers-966
             (+ #:numbers-966
                (coerce-maybe-fold 1 'number)))
       (setq #:items-967
             ((lambda (x) (* x x)) #:numbers-966))
       (incf #:index-965)
          (declare (type nonnegative-integer #:index-965))
         (if (>= #:index-965 #:out-969)
             (go end))
         (if (< #:index-965 0)
             (go #:ll-970)))
       (setq #:sum-959 (+ #:sum-959 #:items-967))
       (go #:ll-970)

What series does it looks at the entire lifetime of the sequence from its creation until it is folded. It uses this information to build the above loop which simultaneously generates the original sequence, maps over it, filters elements out of it, and folds it into the final result. Here is the breakdown of the expansion. Lines 1-9 are just initialization. They define all of the variables the loop will be using and set them to their starting values. The important variables to keep track of are #:NUMBERS-966, #:ITEMS-967, and #:SUM-959. As the code “iterates” over the original sequence, #:NUMBERS-966 is the value of the original sequence, #:ITEMS-967 is the square of that value, and #:SUM-959 is the sum of the squares so far. The rest of the code is the actual loop.

The loop first takes #:NUMBERS-966, the previous value of the sequence, and increments it in order to set it to current value of the sequence (since the sequence is the range from 1 to infinity). Next the loop takes the square of #:NUMBERS-966 to get the ith square number and stores that in #:ITEMS-967. Then the loop checks if it ha taken more than N elements out of the sequence, and if so, terminates. Finally the loop takes the value in #:ITEMS-967 and accumulates that into #:SUM-959.

Although the imperative version is equivalent to the original functional code, it is much faster than the functional code if the functional code were to allocate intermediate results or use laziness. This idea of turning transformations on a list into a loop doesn’t just work for this simple example, it also works for much more complicated programs. I just find it incredible that Series is able to take such pretty code and compile it into code that is extremely fast.

The post Loops in Lisp Part 4: Series appeared first on Macrology.

LispjobsFRONT-END DEVELOPER (CLOJURESCRIPT), Vital Labs, San Francisco

· 19 days ago


We are looking for a generalist developer with a front-end focus and strong interest in working with the Clojure ecosystem who can lead or contribute to all aspects of a modern technology stack. Our production systems are built almost entirely in Clojure/Clojurescript with help from Java/Javascript libraries as needed. We leverage Cassandra and Datomic as our data storage infrastructure and orchestrate all these tools with Ansible. We build our web interfaces with React, as well as our current mobile interfaces. We will be expanding our platforms into native iOS/Android and are exploring new tools such as React Native.

We emphasize simplicity and elegance to combat the inherently complex nature of healthcare. If you are a jack-of-all-trades with a deep interest in building front-end systems that involve complex client-server IO, data visualization, and helping people make sense of machine learning dat, then we'd love to talk to you.


  • 4+ years working on front-end systems (desktop or mobile).
  • 2+ years working on production distributed systems projects with strong exposure to the back-end.
  • Experience working with UX and UI designers.
  • Strong knowledge of web standards, cross-browser issues, and CSS
  • Comfortable with one or more dynamic languages such as Python, Ruby, or Clojure.


  • Comfort working with React or React Native
  • Experience with data visualization or animation in a web context
  • A track record contributing to open source software


  • Experience with the Java/Clojure ecosystem
  • Exposure to Healthcare IT technologies and challenges
  • Exposure to AI – expert systems, statistical modeling, etc.
  • A UX/UI design background and/or interest

Wimpie NortjeExplaining version priority handling in Qlot.

· 24 days ago

How do I specify a library version that is newer than the specified Quicklisp distribution?

While struggling to match a database, Linux libraries, database drivers and all the correct Quicklisp library versions one question I struggled a lot with was 'How do I specify a library version that is newer than the specified Quicklisp distribution?'

The Qlot documentation says about library priorities: 'If multiple distributions provide the same library, lower one would take priority over higher ones.'

This can be interpreted in more than one way which can lead to confusion. The correct interpretation is that the 'lower one' in the documentation refers to the order of the declarations in the qlfile.

To explain Qlot's priority handling more clearly: The order of the declarations in qlfile determines the priority. When multiple distributions provide the same functionality, later statements override the the earlier ones.

The two examples demonstrate the Qlot priority handling.

Using Clack from the latest Quicklisp distribution.

ql :all 2014-01-13
ql clack :latest

Using Clack from the 2014-01-13 distribution.

ql clack :latest
ql :all 2014-01-13

Fernando BorrettiUsing LMDB from Common Lisp

· 24 days ago

LMDB is a fast key-value store. This talk is useful for those who want to learn more. The design of LMDB means many of the things that are standard in other databases - write-ahead logs, and all the filesystem housekeeping necessary to implement concurrent transactions - is unnecessasry in LMDB.

The bindings are implemented as two separate libraries: liblmdb, a low-level, autogenerated CFFI binding; and lmdb, the high-level CLOS binding.


Using LMDB requires some work. A simple query requires setting up (and tearing down) a whole stack of objects, namely:

  1. An environment, which is, essentially, a collection of databases. You create the environment object by passing a directory where LMDB will store its data.
  2. A transaction within that environment. All queries have to take place inside a transaction.
  3. A database access object, which is created from a transaction, after the transaction's been started. The database object keeps the name of the database we're accessing within the environment.


LMDB lifecycle diagram

When you have the database object, you can set, retrieve and delete key-value pairs. For more complex operations, you have to use cursors, which add another level of lifecycle management within databases.


Starting by loading LMDB and Alexandria,

CL-USER> (ql:quickload '(:lmdb :alexandria))
To load "lmdb":
  Load 1 ASDF system:
; Loading "lmdb"
To load "alexandria":
  Load 1 ASDF system:
; Loading "alexandria"


We'll store the database in your home directory under lmdb-test/, and use a hardcoded named LMDB database:

CL-USER> (defparameter +directory+
           (merge-pathnames #p"lmdb-test/" (user-homedir-pathname)))

CL-USER> (defparameter +db-name+ "mydb")

First, let's abstract away all of the housekeeping:

CL-USER> (defmacro with-db ((db) &body body)
           (alexandria:with-gensyms (env txn)
             `(let ((,env (lmdb:make-environment +directory+)))
                (lmdb:with-environment (,env)
                  (let ((,txn (lmdb:make-transaction ,env)))
                    (lmdb:begin-transaction ,txn)
                    (let ((db (lmdb:make-database ,txn +db-name+)))
                      (lmdb:with-database (,db)
                          (lmdb:commit-transaction ,txn)))))))))

We can retrieve keys using the get function:

CL-USER> (with-db (db)
           (lmdb:get db #(1)))

Obviously this returns NIL, since we haven't actually set anything. To add or overwrite a key value pair, you use put:

CL-USER> (with-db (db)
           (lmdb:put db #(1) #(1 2 3)))
#(1 2 3)

CL-USER> (with-db (db)
           (lmdb:get db #(1)))
#(1 2 3)

That's better. But raw byte vectors are unwieldy: how can we store actual data?

First, let's get rid of this key/value pair so we can get back to a blank slate. We use the del function for that:

CL-USER> (with-db (db)
(lmdb:del db #(1))) T

CL-USER> (with-db (db)
           (lmdb:get db #(1)))

Alright, so, real data. These bindings only handle byte vectors: fancier datatypes are explicitly anti-features. Serialization of more complex data structures to byte vectors should be done by a higher-level library - maybe I'll write a Moneta clone for Common Lisp.

Storing strings is pretty simple, all you need is the trivial-utf-8 library:

CL-USER> (ql:quickload :trivial-utf-8)
To load "trivial-utf-8":
  Load 1 ASDF system:
; Loading "trivial-utf-8"


CL-USER> (defun str->vec (str)
           (trivial-utf-8:string-to-utf-8-bytes str))

CL-USER> (defun vec->str (vec)
           (trivial-utf-8:utf-8-bytes-to-string vec))

Now we can use this like this:

CL-USER> (with-db (db)
           (lmdb:put db (str->vec "Common Lisp")
                        (str->vec "An ANSI-standarized Lisp dialect")))
#(65 110 32 65 78 83 73 45 115 116 97 110 100 97 114 105 122 101 100 32 76 105
  115 112 32 100 105 97 108 101 99 116)

CL-USER> (with-db (db)
           (vec->str (lmdb:get db (str->vec "Common Lisp"))))
"An ANSI-standarized Lisp dialect"

How about integers? We use bit-smasher for that:

CL-USER> (ql:quickload :bit-smasher)
To load "bit-smasher":
  Load 1 ASDF system:
; Loading "bit-smasher"

CL-USER> (defun int->vec (int)
           (bit-smasher:int->octets int))

CL-USER> (defun vec->int (vec)
           (bit-smasher:octets->int vec))

And usage:

CL-USER> (with-db (db)
           (lmdb:put db (str->vec "Common Lisp/age")
                        (int->vec 21)))

CL-USER> (with-db (db)
           (vec->int (lmdb:get db (str->vec "Common Lisp/age"))))

This works with Common Lisp's arbitrary-precision integers, as well. Let's try ten to the three hundredth power1:

CL-USER> (expt 10 300)

CL-USER> (integer-length *)

Nine hundred and ninety seven bits is larger than the average machine word, and will be until we start dismanting planets into computers2. Let's see how it works:

CL-USER> (with-db (db)
           (lmdb:put db (str->vec "big integer")
                        (int->vec (expt 10 300))))
#(23 228 60 136 0 117 155 165 156 8 225 76 124 215 170 216 106 74 69 129 9 249
  28 33 197 113 219 232 77 82 217 54 244 74 190 138 61 91 72 193 0 149 157 157
  11 108 200 86 179 173 201 59 103 174 168 248 224 103 210 200 208 75 193 119
  247 180 40 122 110 63 205 163 111 163 179 52 46 174 180 66 225 93 69 9 82 244
  221 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0)

CL-USER> (with-db (db)
           (vec->int (lmdb:get db (str->vec "big integer"))))

So all of this is fine, but what if we don't know the contents of the database? That's what cursors are for, but we don't need to deal with them directly because this wrapper abstracts them:

CL-USER> (with-db (db)
           (lmdb:do-pairs (db key value)
             (format t "~A: ~A~%~%" key value)))
#(67 111 109 109 111 110 32 76 105 115 112): #(65 110 32 65 78 83 73 45 115 116
                                               97 110 100 97 114 105 122 101
                                               100 32 76 105 115 112 32 100 105
                                               97 108 101 99 116)

#(67 111 109 109 111 110 32 76 105 115 112 47 97 103 101): #(21)

#(98 105 103 32 105 110 116 101 103 101 114): #(23 228 60 136 0 117 155 165 156
                                                8 225 76 124 215 170 216 106 74
                                                69 129 9 249 28 33 197 113 219
                                                232 77 82 217 54 244 74 190 138
                                                61 91 72 193 0 149 157 157 11
                                                108 200 86 179 173 201 59 103
                                                174 168 248 224 103 210 200 208
                                                75 193 119 247 180 40 122 110
                                                63 205 163 111 163 179 52 46
                                                174 180 66 225 93 69 9 82 244
                                                221 16 0 0 0 0 0 0 0 0 0 0 0 0
                                                0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                                                0 0 0 0 0 0 0 0 0)


Which is not very informative, since these are just byte vectors.

Finally, you don't want to keep the database directory:

CL-USER> (uiop:delete-directory-tree +directory+ :validate t)


  1. I replaced the actual integer representation from the REPL with 1e300 for brevity.

  2. A 1024-bit wide machine word is probably overkill even then.

Wimpie NortjeFree your project from dependency hell.

· 31 days ago

'How do you manage libraries when different versions of same one is used by different projects?'

It is possible to override a library in Quicklisp's release distribution by placing the desired version in quicklisp/local-projects/ or by ensuring that it is present in ASDF's load path.

One can implement some scheme to override the required libraries in order to set up a Lisp environment with all the correct library versions for each project. This will be time consuming and error prone.

The easy solution to this problem is to use Qlot.

The easy solution to this problem is to use Qlot. It is a dependency management tool built on Quicklisp for Quicklisp libraries. It installs and maintains a separate Quicklisp installation independent from the main installation. One can specify the source and version for each library individually. Unspecified libraries are loaded from the project's default Quicklisp repository, which itself can be specified.

Projects which do not need dependency management simply use Quicklisp as before without even being aware of the existence of Qlot.

Library dependencies are specified in a qlfile and the project system is loaded much like plain Quicklisp.


# "qlfile" of "myapp"
git clack
github datafly fukamachi/datafly :branch v0.7.x
ql log4cl 2014-03-17

Loading a system

;; Installing libraries project-locally.
(qlot:install :myapp)

;; Loading a project with its project-local quicklisp.
(qlot:quickload :myapp)

By using Qlot in your projects you can guarantee that it will build with the correct libraries irrespective of the build time or place.

Wimpie NortjeUse Quicklisp to load personal projects from arbitrary locations.

· 34 days ago

'How does one load a personal project using Quicklisp?'

That is a question which keeps on popping up even though it has been answered many times.

The answers usually involve moving your project to quicklisp/local-projects/, use symlinks from local-projects/ to your system or modifying ASDF search paths to include your custom location.

I suspect people keep on asking the question because the usual recommendations have one or more of the following problems:

  1. Programmers have their own methods to organize code. They don't want to move their code around to accommodate their tools.

  2. Successfully building a project should not depend on the source code being in a specific location.

  3. Building a project on a new computer should not require non-standard environment setups.

  4. It is not possible to work on multiple branches of the same project in parallel if the branches are all located in local-projects/.

  5. ASDF is a lower level tool than Quicklisp. Messing with the ASDF functionality on which Quicklisp depends has the potential to cause problems later.

It is possible to load an ASDF system from any arbitrary location and to avoid all these problems caused by the other methods with the example code below.

(pushnew (truename "/projects/app/") ql:*local-project-directories* )
(ql:quickload :app)

Didier VernaELS 2016 registration now open!

· 38 days ago

Some people do seem to be in the starting blocks for the 9th European Lisp Symposium. One person has already booked his flight, while 3 people from Poland registered yesterday, with the web form that I was still testing. All of this, without the final programme being available, and hell, we're still missing a couple of reviews!

Thank you for the confidence! The registration page is now officially open! :-)

Zach BeaneForcing higher safety levels in SBCL

· 38 days ago

SBCL has a system to let you force compilation with high safety. It overrides any other declarations.

Imagine you have some code with some (speed 3) (safety 0) declarations, and it’s landing you in the debugger without useful variable or location info.

Calling (sb-ext:restrict-compiler-policy 'safety 3) before recompiling the code will force the compiler policy to high safety, ignoring the (safety 0) declarations.

If you use slime, the C-u prefix before compilation commands will also force high safety for the duration of that compilation. I use it for C-c C-c (slime-compile-defun) and C-c C-k (slime-compile-and-load-file) all the time. It’s a quick way to get a lot more debug info when I hit sldb.

This feature was created by Nikodemus Siivola in 2007. Thanks, Nikodemus!

Quicklisp newsRecovering from a breaking update

· 40 days ago
The March Quicklisp update has usocket 0.6.4, an update that unfortunately does not build on older versions of SBCL due to IPv6-related symbols. In response, Chun Tian quickly released usocket, which builds fine on older SBCLs.

It's only been a few days since the last Quicklisp dist release, so I'm not going to make a new one just for the usocket problem.

If you're hit by this issue, what can you do?

The best option, if available, is to upgrade to a recent SBCL, which has many new and exciting features and works well with the usocket in the latest Quicklisp release.

If that's not feasible, you can also manually get usocket (or newer) from the usocket download page, unpack it in ~/quicklisp/local-projects/, and go from there.

Finally, you can revert to last month's Quicklisp dist. I first wrote about this in "Going back in (dist) time", but here's a short overview:

* (in-package :ql-dist-user)

* (second (available-versions (dist "quicklisp")))
. "")

* (install-dist (cdr *) :replace t :prompt nil)
lots of output

In particular, the second element of available-versions will always be the second most-recent Quicklisp dist, so that's generally the one to use if you want to go one dist backward.

(Topic for the future: how better prerelease testing will catch these kinds of problems.)

Zach BeaneSketch looks really cool

· 40 days ago

Sketch, by Danilo Vidovic, looks pretty great. He describes it as “a Common Lisp environment for the creation of electronic art, visual design, game prototyping, game making, computer graphics, exploration of human-computer interaction and more.”

The examples make it look pretty easy to pop up a window and make colorful marks with it. That’s a great way to have a visually interesting feedback loop of development. I’m looking forward to trying it out.

Nicolas HafnerA Case for Documentation - Confession 61

· 43 days ago

header I thought I was done typing for today after spending all day writing up the documentation for North, but I guess I was wrong. Now, there's something that I've noticed every now and again when I take a peek at some of the new projects that pop up on Quicklisp.

Every so often there's one or two libraries that sound interesting or useful to me from their description. But then when I take a look at their project page, way too frequently I have to discover that there's either no README at all, it's just a template copy, or has a single, lonely paragraph out in the cold white space in it. It's also pretty common that docstrings are either completely absent, or only partially existent. This is really sad, because there's a fair chance I would have tried the library out, or in the very least kept it in mind for when I need it later.

Now, I understand, writing documentation takes time. A lot of time. And a lot of people are simply here to get their work done quickly. That's absolutely fine by me, I'm a very impatient person myself. However, when it comes to submitting a library to Quicklisp, it seems to me that the idea behind that would be that the library might prove useful to somebody else, so that they don't have to spend their time writing the same thing again.

And this is where Common Lisp's strength of rapid prototyping becomes a severe flaw, because it pushes people to simply roll their own thing again as they can then fit it perfectly to their needs. Plus, it's probably much more fun or even quicker to do than trying to grind through someone else's undocumented code, desperately trying to figure out how to use it or make it work in the first place.

Writing documentation takes a long time in part because it's hard to formalise and ratify things to fit into text so that hopefully somebody in the world can understand what the thought process was. Now imagine trying to do exactly that, but you did not actually write any part of the code that the documentation should describe. Trying to reverse-engineer the missing documentation from the code is a huge lot harder than writing documentation itself.

As such it doesn't surprise me much to find a lot of people distraught with the current library situation in Lisp, even though over a thousand of them are now readily available by a mere command. A large part of that particular pie is definitely also attributed to the fact that it's simply hard to discover libraries that would help you solve the problem you're tackling, but I cannot imagine that the insufficient amount of -or complete lack of documentation is not also a part of what deters people and makes them think there's nothing useful around.

Obviously there is a lot of useful code around, a lot of time and talent is hidden away in all the projects that were written in CL. It's simply such a shame that a lot of it is so difficult to get a handle on. So please, I implore you, take some time out of your busy schedule and write some documentation for your library before you submit it to Quicklisp. Even just a few paragraphs with a "How To" section that explains how to get the most basic task done helps tremendously to make your library more welcoming.

But even if you don't care about anyone else and simply submit your code to Quicklisp to profit off of its ease of distribution, I still implore you to write documentation- at least for your own sake. Trying to read back into the code of a library that's even just as little as a few months old without any documentation to go by is a huge headache and time-waster. Figuring out some way to reconstruct your past self's ideas and thoughts is very difficult, no matter how much you think you know yourself.

Writing good documentation goes beyond writing a README for your project however. Good and complete documentation includes docstring coverage of all definitions in your code, including those that are not exported, it includes writing not just about the surface on how to make use of your library, but also about its inner workings, how the machinery fits together and what kind of concepts and ideas were realised to make it all work, what kind of problems it aims to solve and how. I would even go so far as to say that good documentation is also the organisation of your source files, how you divide the code and spread it over files, packages, and systems, what you name each of them, and how you reference things across them. All of these things are -if done right- a huge help for anyone that attempts to gain understanding of your code.

Such a level of documentation takes a lot of effort, consideration, and time to do. However, while I know that I too have not reached this level for all of my libraries, I find that the ones I did the most work for are also the ones that I can find bugs the fastest in and have the least trouble getting back into when trying to rework or extend them.

So, please. Please consider taking some time out of your busy schedule to write some documentation.

P.s: If you're one of the people who thinks that docstrings in the definitions are horrendously cluttering up your source code and make it hard to see the forest for the trees in there, have a look at documentation-utils which aims to make it easy to define documentation outside of the associated definition forms.

Quicklisp newsMarch 2016 Quicklisp dist update now available

· 43 days ago
This update, Quicklisp's 66th, is brought to you by me! I've been fortunate to have Clozure sponsor a lot of Quicklisp work in the past, but now I'm self-employed. Working on Quicklisp takes time. In April, I'm going to try to raise a lump sum of money so I can work on a number of get-out-of-beta features, like better download security, provenance info, a manual, etc. Beyond that onetime fundraising, I'd like to enlist more regular monthly supporters for funding ongoing dist updates and maintenance. Do you think Quicklisp is worth it? I'm looking forward to finding out shortly.

If you have any feedback on fundraising, feel free to get in touch.

Back to your regularly scheduled update announcement...

New projects:
  • agm — A high-performance game math library. — MIT
  • cepl — Fast lispy way to work with OpenGL — BSD 2 Clause
  • — A camera implementation for CEPL — BSD 2 Clause
  • cepl.devil — A couple of helper functions to load from devil straight to cepl c-arrays or textures — BSD 2 Clause
  • cepl.sdl2 — SDL2 host for cepl — BSD 2 Clause
  • cepl.skitter — Plumbing to use skitter with cepl — BSD 2 Clause
  • cl-bunny — Common Lisp RabbitMQ client based on IOLib — MIT
  • cl-iterative — Generic iterative algorithm with multiple controls — LGPLv3
  • cl-llvm — CFFI bindings to the LLVM libraries. — MIT
  • cl-ohm — An object-hash mapping for Redis in Common Lisp — MIT
  • cl-sdl2-image — Bindings for sdl2_image using autowrap — MIT
  • cl-strings — A set of utilities for manipulating strings in CL. — MIT
  • cl-tetris3d — Yet another 3D Tetris clone — MIT (also see COPYING file for details)
  • cl-wordcut — Word segmentation tools for ASEAN languages written in Common Lisp — LLGPL
  • cl-zmq — Zero MQ 3 bindings — LGPLv3
  • dartsclemailaddress — Parsing and formatting email addresses (RFC 5322 compliant) — MIT
  • dendrite — Master package for all dendrite packages — BSD 2 Clause
  • documentation-utils — A few simple tools to help you with documenting your library. — Artistic
  • file-local-variable — File-local variable independent from ASDF — LLGPL
  • hu.dwim.bluez — Common Lisp FFI wrapper for libbluetooth, aka Bluez, which is a Linux Bluetooth stack. — BSD or Bugroff
  • hu.dwim.sdl — Common Lisp FFI wrapper for libSDL2. — BSD or Bugroff
  • lambda-gtk — Unspecified — LLGPL
  • legit — CL interface to the GIT binary. — Artistic
  • livesupport — Some helpers that make livecoding with slime/sly a little easier — BSD 2 Clause
  • magicffi — cffi interface to libmagic(3) — Simplified BSD License
  • mito — Abstraction layer for DB schema — LLGPL
  • mito-attachment — Mito mixin class for file management — LLGPL
  • mito-auth — User authorization for Mito classes — LLGPL
  • north — oAuth 1.0a server and client implementation, the successor to South. — Artistic
  • org-sampler — Extract docstrings as Emacs org-mode files — LLGPL 3.latest
  • perfpiece — CL:TIME on steroids. — MIT
  • rt-events — A simple real-time events API. — MIT
  • skitter — An event system for games — BSD 2 Clause
  • snark — The Snark Theorem Prover — MPL 1.1, see file LICENSE
  • trivialib.bdd — BDD and ZDD implementation using Trivia — LLGPL
  • what3words — Describe what3words here — BSD
Updated projectsarchitecture.service-providerarchivebinfixbirchbit-smasherbordeaux-threadsburgled-batteries.syntaxcavemanceramiccffichipzchronicitycl+sslcl-amqpcl-anacl-arrowscl-autowrapcl-charmscl-dbicl-events,cl-fusecl-gambolcl-gobject-introspectioncl-graphcl-itertoolscl-libuvcl-marklogiccl-opsresearchcl-pdfcl-quickcheckcl-rabbitcl-rethinkdbcl-sdl2cl-sdl2-ttfcl-strftimecl-tasuketecl-tgacl-xulclackclack-errorsclaviercloser-mop,clxcolleencontextlcroatoancurry-compose-reader-macroscxml-rngdeedsdelta-debugdjuladocparsereazy-projecteruditeesrapexternal-programfemlispfiltered-functionsfmtfngendlglkithu.dwim.defhu.dwim.perec,hu.dwim.quasi-quotehu.dwim.rdbmshu.dwim.syntax-sugarhu.dwim.utilhu.dwim.web-serverhunchentootia-hash-tableieee-floatsinquisitorintegralintel-hexiolibjskenzolacklakelisp-executablelisp-interface-librarylocal-time,lw-compatmcclimninglenstosicatparenscriptpathname-utilsperiodsplumpplump-sexpproc-parseqlotqt-libsqtoolsqurirtg-mathscalplsdl2kitserapeumsimple-taskssip-hashslimestmxstumpwmsxqltemporal-functions,triviatrivia.balland2006trivial-shellusocketvarjovomwoo.

Removed projects: buffalo (disappeared from github), cl-fgraph (per author's request), cl-quakeinfo (no longer builds on non-Allegro).

To get this update, use (ql:update-dist "quicklisp").


Wimpie NortjeIronclad's password hashing needs initialization.

· 45 days ago

Update (2016-04-02): @abeaumont pointed out that the hashing issue has already been fixed in the Github repo. As of today Quicklisp provides Ironclad version 0.33.0 which dates back to 2014. Until Quicklisp provides an updated version the information in this post is still relevant.

Few people debate whether user passwords must be securely hashed before being stored on disk. Ironclad provides the tools to easily hash and check passwords with the PBKDF2 algorithm.

One of the basic requirements of PBKDF2 is that the salt value be obtained from a cryptographically secure pseudo-random number generator.

Ironclad's password hashing function ... does not initialize the random state ...

Ironclad's password hashing function uses the standard (random) function which is not cryptographically secure. Worse, it does not initialize the random state and it gives no warning or indication about this fact. The effect is that every time the application is started it uses the same sequence of pseudo-random numbers, i.e. after restart the same password generates the same hash.

It is possible to initialize *random-state* properly before using Ironclad's password hashing. This is still not cryptographically secure but at least the pseudo-random sequence is different after every restart which should be sufficient for general purpose credential protection. It will probably not be good enough for a targeted attack.

This code initializes the random state properly:

(defparameter *package-random-state* nil)

(defun hash (password)
  (unless *package-random-state*
    (setf *package-random-state* (make-random-state t)))
  (let* ((*random-state* (make-random-state *package-random-state*))
         (hashed (ironclad:pbkdf2-hash-password-to-combined-string
    (setf *package-random-state* (make-random-state nil))

Though Ironclad's password hashing is very easy to use, the cl-pass library makes it even easier. Cl-pass sets strong (in 2016) default parameters and it handles the random state initialization transparently.

Quicklisp newsFebruary 2016 Quicklisp download stats

· 46 days ago

Here are the top 100 downloads by raw download count.

 5607 alexandria
4586 cl-ppcre
4388 trivial-features
4119 babel
3369 split-sequence
3332 bordeaux-threads
3268 cl+ssl
3243 cffi
3208 anaphora
2725 closer-mop
2692 trivial-gray-streams
2635 trivial-garbage
2620 let-plus
2551 flexi-streams
2524 usocket
2515 cl-fad
2491 cl-colors
2382 cl-base64
2318 prove
2286 cl-ansi-text
2255 cl-utilities
2238 quri
2218 chunga
2203 local-time
2140 cl-html-parse
2113 puri
2102 iterate
1956 chipz
1873 nibbles
1871 drakma
1799 named-readtables
1763 ironclad
1612 cl-interpol
1551 md5
1520 trivial-backtrace
1517 cl-unicode
1418 slime
1343 lack
1336 clack
1330 trivial-types
1309 cl-annot
1295 cl-syntax
1208 fast-io
1161 trivial-utf-8
1134 salza2
1114 cl-json
1104 proc-parse
1092 static-vectors
1053 hunchentoot
992 rfc2388
990 jonathan
981 parse-number
965 cl-mysql
957 fast-http
916 clx
911 uiop
903 xsubseq
875 metabang-bind
865 clsql
836 smart-buffer
835 ieee-floats
829 lparallel
825 osicat
802 postmodern
770 closure-common
766 quicklisp-slime-helper
759 cxml
753 log4cl
699 optima
686 uuid
679 mcclim
672 array-utils
645 cl-reexport
644 cl-sqlite
633 trivial-indent
631 zpng
615 http-body
611 yason
609 fiveam
605 trivial-mimes
602 plump
589 swap-bytes
523 symbol-munger
512 cl-containers
505 cl-yacc
502 ningle
501 vom
501 simple-date-time
498 map-set
494 myway
490 lisp-unit
483 lquery
479 dexador
476 asdf-system-connections
473 cl-cookie
470 function-cache
466 cl-csv
466 iolib
466 metatilities-base
464 cl-dbi

Wimpie NortjeSending large files from Clack

· 47 days ago

For one of my projects I encountered the issue of sending a large file as an HTTP response from Clack.

Clack supports sending files as a response with Hunchentoot, FCGI and Wookie back-ends. If you use Toot as back-end you will have to implement your own method.

To send a file, return the file's path as a PATHNAME (#P"/path/to/file") as the response body.

The different back-ends use different methods to send a file. This has other implications which you may want to consider.

Server Method Notes
Hunchentoot HTTP Chunking
  • Optimal for large files.
  • Does not load whole file into RAM.
  • Multiple small messages.
Wookie Byte stream
  • Does not load whole file into RAM.
  • Single large message.
FCGI Array
  • Loads the whole file into RAM.
  • Single large message.
Toot DIY

Wimpie NortjeStoring Clack session information.

· 49 days ago

Clack comes with separate middleware modules for session management and for storing session information in a database. I discovered the storage middleware shortly after implementing my own version.

This post describes what I learnt about these two options. Though the code to implement this feature is minimal, it took me much too long to figure out the details necessary to do so.

Writing your own storage layer has the following advantages:

  • You can use storage not supported by cl-dbi, i.e. MySQL, PostgreSQL and SQLite.
  • You determine the table layout.

My suggestion is to use the included version. It will save you time. I only created my own version because I did not know Clack already included the necessary module.

Using Clack's storage middleware

This information is correct at least up to commit c4f3e09 (2015-10-27)

The class name is <clack-session-store-dbi> located at v1-compat/src/contrib/session/store/.

The code below creates a Clack application which handles the storage of session information completely automatically.

   :store (make-instance '<clack-session-store-dbi>
                         :connect-args '(:postgresql
                                         :database-name "DBNAME"
                                         :username "ROLE"
                                         :password "PASSWORD")))

A hash table containing the session information is available inside request handlers. It is obtained with (getf env :clack.session).

The hash table content is serialized before being stored to disk so it can be used to store any arbitrary data.

Session tracking is done using cookies which contain only a session ID. All other session information are kept server side.

Writing your own persistence middleware

The code below implements middleware for a storage layer. The actual database access is done by the model package.

(in-package :cl-user)
  (:use :cl

(cl-syntax:use-syntax :annot)

(defclass <sltv-session-store> (<clack-session-store>)

(defmethod fetch ((this <sltv-session-store>) sid)
  (let* ((session (model:fetch-session-as-hashtable sid)))
    (setf (gethash :id session) sid)

(defmethod store-session ((this <sltv-session-store>) sid session)
  (model:store-session sid session))

(defmethod remove-session ((this <sltv-session-store>) sid)
  (model:remove-session sid))


  • sid is the session ID generated by Clack.
  • The session hash table must contain a key ID which is set to the sid not the database record's ID field.
  • The session must be defined with (make-hash-table :test 'equal).
  • fetch obtains an existing session from the database or creates a new entry if it doesn't exist.
  • store-session stores the session hash table and sid in the database.
  • remove-session deletes the database record.

When an HTTP request is received the session middleware fetches the session for the received sid. If the request did not include an sid (i.e. no cookie) or it included an expired sid, Clack generates a new one and fetches (which should create an new record) the session.

The request handler has three option settings with which it can control some of the session middleware's actions.

Do not store the session object. It is still created in the database and available in the request handler but the database entry is deleted before the response is returned.
Expire the session so that a new one is generated next time the user visits.
Create a new session and populate it with the current data.

An option can be changed with the form

(setf (gethash :change-id (getf env :clack.session.options)) t)

Wimpie NortjeGenerating HTTP access logs with Clack.

· 51 days ago

After using Hunchentoot as my HTTP server for a while I decided to switch to Clack. One of the first things I needed to do was to enable access logging. This stumped me for a while because it is not as trivial as with Hunchentoot. After digging around I found the relevant documentation.

By default Clack does not log anything. It is distributed with the <clack-middleware-accesslog> middleware which can log HTTP accesses but must be explicitly enabled.

Without any customization the code below will log to *standard-output*.


The middleware provides three methods for customization.

  1. *time-format*
    A variable to change the format used to display time.

  2. formatter
    A slot which takes a function of three parameters. It returns the formatted string which will be logged.

    The function prototype is:
    (defun formatter (clack-env http-result-code current-time))

  3. logger
    A slot which takes a function of one parameter. The parameter is the string to be logged. This function performs the actual logging action.

One possible customization is this:

      :logger #'model:http-access-log))

Wimpie NortjeFix out-of-sequence printing when using (format).

· 57 days ago

"Why doesn't format simply do its job and print immediately?" - Redditor

Many programmers new to Lisp have been baffled when they use (format) to write to *STANDARD-OUTPUT* and the text appears after inexplicable delays or output from multiple threads appear out of sequence.

The reason is that *STANDARD-OUTPUT* is a buffered stream by default. Text written to such a stream is buffered until an event occurs which triggers the process of actually writing to the output. The usual trigger events are:

  • the buffer is full,
  • the buffer is closed, or
  • a #\NEWLINE is written.

There are two options to print the buffer's content immediately without otherwise interfering with the stream: (finish-output) and (force-output).

(finish-output) is a blocking function. It ensures that all buffered output is written to the output stream before returning.

(force-output) is a non-blocking function. It initiates the writing process and then returns immediately without waiting for the buffer to be completely written.

A code example:

(format t "Debug a value: ~A" 5)

ECL NewsBountysource Salt account

· 58 days ago

We have set up an account on the bountysource to simplify the financial contributions to the project.

Additionally you may send money contributions with a PayPal account:

Eugene ZaikonnikovThe Grand SYMBOL-PLIST Treatise

· 61 days ago

Plists are handy simple key/value stucures, stored in a single-linked list. A plist (a 1 b 2 c 3) holds three numeric values for properties a through c. Lisp dialects have standard functions for conveniently adding, looking up and removing the properties.

All operations are thus O(n), but shorter plists (say a dozen or two entries) typically beat hash tables due to substantial constant overhead of the latter.

Symbol-plists specifically are plists attached to a Lisp symbol via dedicated language mechanism.


The earliest mentions of property lists predate first Lisp manuals. In his 1958 Programs with Common Sense paper[1], John McCarthy had to say the following:

For example, to most people, the number 3812 is not an object: they have nothing to say about it except what can be deduced from its structure. On the other hand, to most Americans the number 1776 is an object because they have filed somewhere the fact that it represents the year when the American Revolution started. In the advice taker each object has a property list in which are listed the specific things we have to say about it.

Some remarkable facts can be deduced from this short passage:

  • Symbol-plists predate Lisp symbols conceptually. Here, a number is used as an object with a list of properties.
  • The intent is providing a form of associative memory, a key-value store for the procedural evaluator. It's like noSQL before SQL!
  • Symbol-plists are the original sin of functional programming. They were meant for side effect in FP before FP was a thing. Thus classic Lisp was never meant to be purely functional; state is not an oversight but a deliberate design decision.
  • Even though the precise storage model is unspecified, a list is implied. LISP at the time was still a study in progress and hash tables were known since 1953, but they would have been too costly in terms of memory.

The earliest known object orientation mechanism

The use of word object in the quote above is not coincidental. The sets of properties were defining the object entities. The objects in early Lisp would manifest as symbols. LISP 1.5 implementation hinged on symbol-plists, with four pre-defined properties:

  • PNAME, the print name of the symbol
  • EXPR, a user defined associated function, stored as S-expression
  • SUBR, pointer to machine code built in subroutine
  • APVAL, the global scope storage, confusingly referred as "constant value" (as opposed to value within the function's dynamic scope)

This scheme was later split into symbol-name, symbol-function and symbol-value cells.

Plists and symbols were thus providing

  • User definable object properties
  • Encapsulation mechanism with accessors
  • Pre-defined system properties
  • Dedicated property for associated function

Granted, there was no explicit this/self-reference to be used in stored function, but that's just sugar trivia:

(defmacro mydefun (name args &rest body)
  `(defun ,name ,args
	(let ((this ',name))
(mydefun foo ()
(foo) => FOO

..the reason it's not there in the spec is that the "modern" OO model was not a design concern. Advice Taker was meant to be procedural logic manipulator, not an agency. This conceptual difference would lead to eventual split of McCarthy-Minsky cooperation, and we'll see Minsky introducing the object-referencing methods with his frames effort.

It is important though that Lisp symbols were true object entities existing at run-time from day zero.

Symbol-plists as namespaces

Symbol-plists can be thought as defining factor of lisp arity. In the degraded case of Scheme we get lisp-1 with one name/value pair only.

Lisp 1.5 is an lisp-n, where n is the size of the largest symbol plist in the system.

Common Lisp is a lisp-(c + n), where c is implementation-dependent number of key-value pairs (or cells) factored out of original symbol-plist. Funny enough, symbol-plist is now one of the 4 standard mandated cells itself. The advantage here is the system cells are safe from accidental plist smashing and don't impose a run-time penalty for user's plist manipulating code.

Use and disuse

Nearly all early AI programs make extensive use of symbol-plists. As a didactic sample I would recommend annotated tree implemenation in Steve Tanimoto's ID3-like inducer. However, with gradual changes in Lisp implemenation landscape and programming mindset, symbol-plists fell into disuse.

An obvious contributing factor to symbol-plists decline was the removal of system properties into separate cells. However, the bigger affecting trend was the move from symbolic manipulation in Lisp code throughout 1980-1990s. This had several reasons. The symbol manipulating, computationalist AI effort has largely collapsed facing advances in statistical ML techniques. The proliferation of viable macro systems and native compilation obliterated demand to code-as-data manipulation. This was amplified by the influx of Algol tradition programmers, where symbol entities are simply not a concept and hence the whole approach remained alien.

There is still acceptance of detached plists for transient key/value records, but unsurprisingly, symbol-plists are viewed as obsolete, almost counter-cultural thing. Now it's certainly personal, but plists without symbols are like naked singularities: they might occur in nature but still are abominations. Or like garden slugs: sad, vulnerable version of a snail without a place to belong. As plists have no inherent type in Common Lisp[2], their interpretation is contextual, and outside of symbol-plist realm the context is not necessarily clear.

[1] Published in vol.1 1959 proceedings of Mechanisation of Thought Processes conference. A remarkable print from low hanging fruit times of Computer Science, with submissions from McCarthy, Minsky, Ershov, Hopper, Backus, McCulloh and other pioneers of note.

[2] Interestingly, in Lisp 1.5 plists were tagged by -1 value in CAR of the list. There were other storage optimizing techniques too, such as special format for flags, the valueless properties.

ECL NewsECL 16.1.2 release

· 62 days ago

We are happy to inform that the official ECL 16.1.2 release is available for download:

Pascal CostanzaCloser project repositories moved to github

· 62 days ago
I have moved the Closer project repositories to github. All details around this move are listed at the Closer project website, especially see the list of repositories. Since github provides dedicated means for reporting issues, I'm shutting down the mailing lists for the Closer project as well.

Wimpie NortjeWhen Google does not answer your Common Lisp questions.

· 66 days ago

Many Common Lisp programmers have found that Google is not very useful in answering their questions. It is not uncommon to hear "I can't find answers on Google" from CL programmers. Even the famous Xach1 (Zach Beane) has said that.

The reason for this problem is likely that CL questions are addressed in forums and mailing lists rather than in expansive blog posts.

Zach wrote an in-depth post about how and where to get help with your Common Lisp questions. It is not an exhaustive list but it covers the most important options. The sidepanel on subreddit /r/lisp contains a more complete list of resources and naturally the blogs about Lisp will increase as more people start to use it.

Below is a summary of Zach's information for the benefit of those in a hurry. It is worth reading his complete post at least once to get more insight on each of them.

For general Lisp questions go here:

For questions about a specific library, start with its own mailing list2 and then try the list above.

For questions about your implementation, go to its own mailing list. (Zach gives links to the most prominent ones.)

When you are about to post a question, take Zach's advice on how to ask a question. It will likely save you time.

  1. If you don't know about Xach you have not yet installed Quicklisp.

  2. Some libraries you are likely to use:

CL Test Gridquicklisp 2016-02-08

· 68 days ago
The difference between this and previous months:

Grouped by lisp implementation first and then by library:

Grouped by library first and then by lisp impl:

(Both reports show the same data, just arranged differently)

The diff contains mostly improvements, but there are several regressions: varjo, gendl, lake, buffalo, binfix, data-table and others.

If you're interested in some particular failure or need help investigating something, just ask in the comments or on the mailing list.

For older items, see the Planet Lisp Archives.

Last updated: 2016-04-28 16:42