Planet Lisp

Zach BeaneQuicklisp: beyond beta

· 30 minutes ago

I gave a talk at ELS 2015 in London in April about the past, present, and future of Quicklisp. The "future" part detailed several things I'd like to accomplish before removing the beta label from Quicklisp.

The slides and script of the talk are available on github. But since there are over a hundred slides and about twenty pages of script, I thought I'd summarize things in this post.

First, what's Quicklisp for, anyway? The slogan I put in the talk is: Make it easy to confidently build on the work of others. (Also, work on as many Common Lisps on as many platforms as possible.) 

Quicklisp achieves part of that already. It runs well on almost all Common Lisp implementations on all platforms. It's easy to install, use, and update. Things are tested together so everything usually works. But if something breaks on update, you can revert to a previous working version. And with those features you can build on hundreds of libraries already.

If it can do all that, why not drop the "beta" already? There are still a number of things that I want Quicklisp to do before I'm ready to say "Here, this is what I had in mind from the start."

First, I'd like to improve the confidence in the code you download and run. By adding HTTPS, signature checking, and checksum validation, you can be sure that there is nobody intercepting and modifying the software provided by Quicklisp. The signature and archive integrity checks must be made complete and automatic to have the best results.

Second, I'd like to add comprehensive user and developer documentation. For users, that means being able to learn each command, feature, and behavior of Quicklisp, to be able to use it to its fullest. For developers, that means being able to build your own solutions on a Quicklisp foundation without starting from scratch.

Third, I'd like to make it easy to find the project that does what you need, evaluate its quality and popularity, and find out if its license is compatible with your goals. If you want to make changes to a project, I want it to be easy to get the original source of a project and send fixes or improvements to the upstream maintainer.

Fourth, I'd like to make it easy to hook into the archive-fetching component of Quicklisp in a way that makes it easy to support additional integrity checks, support local development policies, and add local mirrors or caches for Quicklisp software.

These changes and improvements will take time. When they're done, I'll be happy to drop "beta" from Quicklisp.

Didier VernaDeclt 1.1 is released

· 2 days ago


as promised last week, I've just released a new version of Declt, my reference manual generator for ASDF systems. This new version (1.1) is now able to document Clon again (the documentation of which has been updated on the website).

New in this release:

  • Declt now properly handles and documents complex system and component dependencies, such as :feature :require and :version statements,
  • Declt also documents a system's :if-feature if any.

But the most important addition is the ability to document several ASDF systems in the same reference manual. More precisely, Declt now documents not only the main system but also all its subsystems. A subsystem is defined as a system on which the main one depends on in any way, and which is also part of the same distribution (under the same directory tree). Declt also understands multiple system definitions from the same .asd file.


Paul KhuongLinear-log Bucketing: Fast, Versatile, Simple

· 3 days ago

There’s a couple code snippets in this post (lb.lisp, bucket.lisp, bucket-down.lisp, bin.c). They’re all CC0.

What do memory allocation, histograms, and event scheduling have in common? They all benefit from rounding values to predetermined buckets, and the same bucketing strategy combines acceptable precision with reasonable space usage for a wide range of values. I don’t know if it has a real name; I had to come up with the (confusing) term “linear-log bucketing” for this post! I also used it twice last week, in otherwise unrelated contexts, so I figure it deserves more publicity.

I’m sure the idea is old, but I first came across this strategy in jemalloc’s binning scheme for allocation sizes. The general idea is to simplify allocation and reduce external fragmentation by rounding allocations up to one of a few bin sizes. The simplest scheme would round up to the next power of two, but experience shows that’s extremely wasteful: in the worst case, an allocation for \(k\) bytes can be rounded up to \(2k - 2\) bytes, for almost 100% space overhead! Jemalloc further divides each power-of-two range into 4 bins, reducing the worst-case space overhead to 25%.

This sub-power-of-two binning covers medium and large allocations. We still have to deal with small ones: the ABI forces alignment on every allocation, regardless of their size, and we don’t want to have too many small bins (e.g., 1 byte, 2 bytes, 3 bytes, ..., 8 bytes). Jemalloc adds another constraint: bins are always multiples of the allocation quantum (usually 16 bytes).

The sequence for bin sizes thus looks like: 16, 32, 48, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320, 384, ... (0 is special because malloc must either return NULL [bad for error checking] or treat it as a full blown allocation).

I like to think of this sequence as a special initial range with 4 linearly spaced subbins (0 to 63), followed by power-of-two ranges that are again split in 4 subbins (i.e., almost logarithmic binning). There are thus two parameters: the size of the initial linear range, and the number of subbins per range. We’re working with integers, so we also know that the linear range is at least as large as the number of subbins (it’s hard to subdivide 8 integers in 16 bins).

Assuming both parameters are powers of two, we can find the bucket for any value with only a couple x86 instructions, and no conditional jump or lookup in memory. That’s a lot simpler than jemalloc’s implementation; if you’re into Java, HdrHistogram’s binning code is nearly identical to mine.

Common Lisp: my favourite programmer’s calculator

As always when working with bits, I first doodled in SLIME/SBCL: CL’s bit manipulation functions are more expressive than C’s, and a REPL helps exploration.

Let linear be the \(\log\sb{2}\) of the linear range, and subbin the \(\log\sb{2}\) of the number of subbin per range, with linear >= subbin.

The key idea is that we can easily find the power of two range (with a BSR), and that we can determine the subbin in that range by shifting the value right to only keep its subbin most significant (nonzero) bits.

I clearly need something like \(\lfloor\log\sb{2} x\rfloor\):

(defun lb (x)
  (1- (integer-length x)))

I’ll also want to treat values smaller than 2**linear as though they were about 2**linear in size. We’ll do that with

n-bits := (lb (logior x (ash 1 linear))) === (max linear (lb x))

We now want to shift away all but the top subbin bits of x

shift := (- n-bits subbin)
sub-index := (ash x (- shift))

For a memory allocator, the problem is that the last rightward shift rounds down! Let’s add a small mask to round things up:

mask := (ldb (byte shift 0) -1) ; that's `shift` 1 bits
rounded := (+ x mask)
sub-index := (ash rounded (- shift))

We have the top subbin bits (after rounding) in sub-index. We only need to find the range index

range := (- n-bits linear) ; n-bits >= linear

Finally, we combine these two together by shifting index by subbin bits

index := (+ (ash range subbin) sub-index)

Extra! Extra! We can also find the maximum value for the bin with

size := (logandc2 rounded mask)

Assembling all this yields

(defun bucket (x linear subbin)
  (let* ((n-bits (lb (logior x (ash 1 linear))))
         (shift (- n-bits subbin))
         (mask (ldb (byte shift 0) -1))
         (rounded (+ x mask))
         (sub-index (ash rounded (- shift)))
         (range (- n-bits linear))
         (index (+ (ash range subbin) sub-index))
         (size (logandc2 rounded mask)))
    (values index size)))

Let’s look at what happens when we want \(2\sp{2} = 4\) subbin per range, and a linear progression over \([0, 2\sp{4} = 16)\).

CL-USER> (bucket 0 4 2)
0 ; 0 gets bucket 0 and rounds up to 0
CL-USER> (bucket 1 4 2)
1 ; 1 gets bucket 1 and rounds up to 4
CL-USER> (bucket 4 4 2)
1 ; so does 4
CL-USER> (bucket 5 4 2)
2 ; 5 gets the next bucket
CL-USER> (bucket 9 4 2)
CL-USER> (bucket 15 4 2)
CL-USER> (bucket 17 4 2)
CL-USER> (bucket 34 4 2)

The sequence is exactly what we want: 0, 4, 8, 12, 16, 20, 24, 28, 32, 40, 48, ...!

The function is marginally simpler if we can round down instead of up.

(defun bucket-down (x linear subbin)
  (let* ((n-bits (lb (logior x (ash 1 linear))))
         (shift (- n-bits subbin))
         (sub-index (ash x (- shift)))
         (range (- n-bits linear))
         (index (+ (ash range subbin) sub-index))
         (size (ash sub-index shift)))
     (values index size)))
CL-USER> (bucket-down 0 4 2)
0 ; 0 still gets the 0th bucket 
0 ; and rounds down to 0
CL-USER> (bucket-down 1 4 2)
0 ; but now so does 1
CL-USER> (bucket-down 3 4 2)
0 ; and 3
CL-USER> (bucket-down 4 4 2)
1 ; 4 gets its bucket
CL-USER> (bucket-down 7 4 2)
1 ; and 7 shares it
CL-USER> (bucket-down 15 4 2)
3 ; 15 gets the 3rd bucket for [12, 15]
CL-USER> (bucket-down 16 4 2)
CL-USER> (bucket-down 17 4 2)
CL-USER> (bucket-down 34 4 2)

That’s the same sequence of bucket sizes, but rounded down in size instead of up.

The same, in GCC

static inline unsigned int
lb(size_t x)
        /* I need an extension just for integer-length (: */
        return (sizeof(long long) * CHAR_BIT - 1) - __builtin_clzll(x);

 * The following isn't exactly copy/pasted, so there might be
 * transcription bugs.
static inline size_t
bin_of(size_t size, size_t *rounded_size,
    unsigned int linear, unsigned int subbin)
        size_t mask, range, rounded, sub_index;
        unsigned int n_bits, shift;

        n_bits = lb(size | (1ULL << linear));
        shift = n_bits - subbin;
        mask = (1ULL << shift) - 1;
        rounded = size + mask; /* XXX: overflow. */
        sub_index = rounded >> shift;
        range = n_bits - linear;

        *rounded_size = rounded & ~mask;
        return (range << subbin) + sub_index;

static inline size_t
bin_down_of(size_t size, size_t *rounded_size,
    unsigned int linear, unsigned int subbin)
        size_t range, sub_index;
        unsigned int n_bits, shift;

        n_bits = lb(size | (1ULL << linear));
        shift = n_bits - subbin;
        sub_index = size >> shift;
        range = n_bits - linear;

        *rounded_size = sub_index << shift;
        return (range << subbin) + sub_index;

What’s it good for?

I first implementated this code to mimic’s jemalloc binning scheme: in a memory allocator, a linear-logarithmic sequence give us alignment and bounded space overhead (bounded internal fragmentation), while keeping the number of size classes down (controlling external fragmentation).

High dynamic range histograms use the same class of sequences to bound the relative error introduced by binning, even when recording latencies that vary between microseconds and hours.

I’m currently considering this binning strategy to handle a large number of timeout events, when an exact priority queue is overkill. A timer wheel would work, but tuning memory usage is annoying. Instead of going for a hashed or hierarchical timer wheel, I’m thinking of binning events by timeout, with one FIFO per bin: events may be late, but never by more than, e.g., 10% their timeout. I also don’t really care about sub millisecond precision, but wish to treat zero specially; that’s all taken care of by the “round up” linear-log binning code.

In general, if you ever think to yourself that dispatching on the bitwidth of a number would mostly work, except that you need more granularity for large values, and perhaps less for small ones, linear-logarithmic binning sequences may be useful. They let you tune the granularity at both ends, and we know how to round values and map them to bins with simple functions that compile to fast and compact code!

P.S. If a chip out there has fast int->FP conversion and slow bit scans(!?), there’s another approach: convert the integer to FP, scale by, e.g., \(1.0 / 16\), add 1, and shift/mask to extract the bottom of the exponent and the top of the significand. That’s not slow, but unlikely to be faster than a bit scan and a couple shifts/masks.

Vsevolod DyomkinRunning Lisp in Production at Grammarly

· 4 days ago

We have written a blog post describing almost 3 years of our Lisp in production experience at Grammarly. Here's a small abstract for it.

At Grammarly, the foundation of our business, our core grammar engine, is written in Common Lisp. It currently processes more than a thousand sentences per second, is horizontally scalable, and has reliably served in production for almost 3 years.

We noticed that there are very few, if any, accounts of how to deploy Lisp software to modern cloud infrastructure, so we thought that it would be a good idea to share our experience. The Lisp runtime and programming environment provides several unique, albeit obscure, capabilities to support production systems (for the impatient, they are described in the final chapter).

Continue to the full text »

Zach BeaneLisp in production at Grammarly

· 4 days ago

Vsevolod Dyomkin (who conducted the Lisp Hackers interviews) has an interesting blog post today about running Lisp in production at Grammarly:

At Grammarly, the foundation of our business, our core grammar engine, is written in Common Lisp. It currently processes more than a thousand sentences per second, is horizontally scalable, and has reliably served in production for almost 3 years.

We noticed that there are very few, if any, accounts of how to deploy Lisp software to modern cloud infrastructure, so we thought that it would be a good idea to share our experience.

Full article.

Didier VernaClon 1.0b24 is released -- IMPORTANT

· 6 days ago


I'm happy to announce the release of the next beta version of Clon, the Common Lisp / Command Line Options Nuker library. This release doesn't contain much change in terms of functionality, but it contains a lot of change in terms of infrastructure, plus very important and backward-incompatible modifications. So if you're a Clon user, please read on.

First of all, a huge revamp of the library's infrastructure (package hierarchy, ASDF and Make implementations) occurred. A large portion of this work is actually not mine, but Fare's (big thanks to him, 'cause the level of ASDF expertise required just means that I couldn't have done that by myself). The purpose here was twofold: first, remove all logic from the ASDF files (so that other system managers could be used; not sure that's actually useful right now) and second, split the library in two: the core, basic functionality and the non-standard platform-dependent bells and whistles (read: termio support). The result is that Clon now comes with 4 different ASDF systems! A setup system allows you to configure some stuff prior to loading the library, a core system allows you to load only the basic functionality and the regular one loads everything, autodetecting platform-dependent features as before. The fourth system is auxiliary and not to be used by hand. All of this is properly documented. For a code maniac like me, this new infrastructure is much more satisfactory, and I've learned a lot about ASDF less known features.

Next, I've moved the repository to Github. Please update your links! It seems that I've lost all my former tags in the process, but oh well...Only the Git repo has moved. The main Clon web page still contains the full history of tarballs, the preformatted documentation, and will continue to do so in the future.

Finally (I've kept this to myself until the last possible minute because I'm scared like hell to tell): I've changed the systems and packages names... The com.dvlsoft prefix has been replaced with net.didierverna. All other libraries of mine will eventually endure the same surgery. It's for the best, I apologize for it and I swear I will never ever do that again, EVER (fingers crossed behind my back).

So what's next? Before considering an official 1.0 release, there are two things that I want to do. First, cleanup some remaining Fixmes and some shaky error handling. Second, provide an even simpler way of using Clon than what the Quick Start chapter in the doc demonstrates. The idea is to just implement a main function with keyword arguments, and those argument magically become command-line options.

A side-effect of this work is that Declt now chokes on Clon, because some ASDF features that it doesn't understand are in use. So Declt has a couple of new challenges ahead, and you should expect a new release in the weeks to come.

Patrick SteinSyntactic Corn Syrup

· 15 days ago

I’ve been bouncing around between Java and C++ and C and loads of JNI cruft in between. At some point today, I accidentally used a semicolon to separate parameters in my C function declaration:

void JNI_myJNIMethod( int paramA; int paramB; int paramC )

It looked wrong to me. But, I had one of those brain-lock moments where I couldn’t tell if it was wrong. I was pretty sure that it was wrong by the time my brain locked on pre-ANSI K&R:

JNI_myJNIMethod(paramA, paramB, paramC)
  int paramA;
  int paramB;
  int paramC;

Regardless, it got me thinking about the programming maxims: Deleted code has no bugs and Deleted code is debugged code.

I never have this kind of brain-lock in Lisp. Some of that is because my Emacs configuration has been molded to my Lisp habits better than to my C/C++/Java habits. Most of it, though, is that Lisp understands the difference between syntactic sugar and syntactic cruft.

Lisp decided long ago that writing code should be easy even if it makes writing the compiler tougher. C and C++ and Java all decided that LALR(1) was more important than me. As if that weren’t bad enough, C++ and Java have thrown the lexers and parsers under the bus now, too. No one gets a free ride.

drmeisterI gave a talk on Clasp and my Chemistry at Google in Cambridge Mass last week

· 15 days ago

Here is the link for the talk.

This talk describes our unique approach to constructing large, atomically precise molecules (called “Molecular Lego” or “spiroligomers”) that could act as new therapeutics, new catalysts (molecules that make new chemical reactions happen faster) and ultimately to construct atomically precise molecular devices. Then I describe Clasp and CANDO, a new implementation of the powerful language Common Lisp. Clasp is a Common Lisp compiler that uses LLVM to generate fast machine code and it interoperates with C++. CANDO is a molecular design tool that uses Clasp as its programming language. Together I believe that these are the hardware (molecules) and the software (the CANDO/Clasp compiler) that will enable the development of sophisticated molecular nanotechnology.

For more info see:

What a great place Google was! My host, Martin Cracauer was fantastic, he made me feel really, really welcome and made sure that the talk would be recorded and put up on the web. He arranged it so that I could spend the afternoon talking with him and Doug and James, two Lisp/compiler gurus at Google. He also gave me a tour of Google, it was great.

Nicolas HafnerThe Great UI Warts - Confession 56

· 19 days ago

It's now been about a year since I first started work on Parasol. In the process, I had to learn about UI programming in Common Lisp. It pains me a lot to say this, but it is definitely not one of the great strengths of CL. It certainly wasn't back then, and it still isn't now. Since Parasol started I learned a lot about Qt and in particular the Common Lisp bindings, CommonQt. While using Qt is your best bet at writing a native GUI, it just isn't as pleasant as writing other lisp code. Too many things can break, too many brick walls are laying in wait for you to hit your head against, too many things are simply not there infrastructure wise. However, as Parasol grew, and I grew tired of CommonQt's shortcomings, I started to write more and more systems to work around these problems and make the UI experience for the developer a better one. This is the goal of Qtools.

This library started out as an innocent encapsulation of a few things I'd developed in tandem with Parasol. The first serious issue I had with Parasol was memory leaking. Since we're accessing Qt -a C++ library- we need to go back to the old times and deal with our memory by hand. This is a very arduous task and one prone to mistakes. So, a system was developed to alleviate this pain. The result of this is Qtools' finalizers. At the core of it is a generic function that takes care of cleaning up the object it is passed. So in other words, a destructor function. Using this I could ensure that foreign objects were always properly cleaned up. However, I quickly came to realise that I did one very similar thing all the time: Add a finalizer method for my widget, and call finalize on its slot values. Thanks to the Meta Object Protocol's capabilities, I was able to hide this away completely. Now there's almost never a need to write a finalizer method again. It suffices to just add :finalized T to a widget's slot, and in the case of sub-widgets, the system already does it automatically.

The next issue I had was that writing in CommonQt's style is really uncomfortable. You need to duplicate a lot of information and keep track of the slots, signals, and overrides you define in the class definition. You also need to take care of different type and naming styles that come from C++ and leak into your CL application. This spawned Qtools' widget system. Not only does it take care of mapping naming styles and types, but it also allows a much more normal-looking way of defining your widgets. Instead of having to stuff information into your class definition, you can use multiple, separated forms. Just the way it works in your usual Lisp programs. At the heart of this system lies reinitialize-instance. Thanks to this fantastic function (and the MOP), I was able to separate everything out. What happens in the back when you compile a separate form is that it appends the option that should be in the class definition onto a class property, and calls reinitialize-instance. This call subsequently computes the effective class options and injects them into the class re/initialisation, effectively making it appear as if you had indeed added an option onto the class definition itself.

With this step done, much of the awkwardness was gone. Programs looked much more naturally structured, and things could be specified in a way that felt intuitive. However, one stain remained in the picture: Qt method calls. In order to call Qt methods, CommonQt provides a reader macro: #_. Sadly, in order for this reader macro to work, you need to specify the method name as it is in Qt, including the proper capitalisation. Since it is a foreign call, you also can't inspect it, or get any documentation information out of it. Argument list validity also isn't checked. Getting rid of this and allowing some form of normal-looking function call instead was a rather tricky problem to solve. My first thought was to dynamically analyse the available methods and generate Lisp wrapper functions for them. Those wrapper definitions are dumped to file, and then loaded. Sadly, doing so results in a couple hundred thousand method wrappers and a roughly 50Mb FASL file (on SBCL). The initial compile time also suffered because of this of course. This seemed like a less than stellar solution to me, mostly because the overwhelming majority of the wrappers included would never be called by the GUI anyway. So I sought a different solution. I did find one, albeit it is rather mad.

This solution is called Q+. The first part of it is the aforementioned wrapper compiler that I previously used to generate static wrappers. Modifying it a bit, I could use the same system to generate individual wrappers for any specific method I wanted. The second part is detecting when a supposed call to a wrapper function is made. Since it is not precompiled, the CL host cannot know of it. Thus, we need to somehow intercept when such a form is compiled, dynamically compile the wrapper, and then replace it with a call to the actual wrapper function. That sounds like a macro! And indeed, the q+ macro does that. It takes a method name and an argument list, dynamically compiles the wrapper, and finally emits a call to the new wrapper. The truth is a bit trickier here, since the wrapper needs to be available when a file is merely loaded as well, which wouldn't be the case if it was only generated during macro expansion. So instead, a load-time-value form that generates the wrapper is emitted alongside the wrapper call. That way, methods are always around as needed, with no run-time overhead. The last trick to Q+ is the hiding of the q+ macro call. Using the q+ macro solved most of the problems, but it was essentially the same thing as the #_ reader macro, with a bit nicer method name handling. What I wanted instead was to be able to write the actual wrapper function names. That would also allow slime to show docstrings, arguments, and similar information. In order to make this last trick work, I had to hack into the reader.

One of the greater blemishes of the the Common Lisp standard is the inability to hook into the reader's symbol creation process. This exclusion from the standard makes it impossible to write such things as package local nicknames as a library, or make a case like mine easy. What I had to do instead was to override the ( reader macro. Q+ then reads ahead, to see whether you're trying to reference a symbol from the q+ package. If so, it reads the rest of the form, and emits a call to the q+ macro from above instead. It not, it delegates to the standard reader macro for (. Overriding this reader macro is a dirty trick, and I'd rather not have done it. However, there simply is no other way to accomplish this feat, short of writing a complete reader implementation and demanding that people use that instead of the host implementation's, which is a bit too much to ask for, in my opinion. Still, it works fine, and I haven't run into any obvious issues so far. Now Qtools applications look and read like regular lisp code.

However, how the code looks is only one of the aspects that influence writing GUIs. There's a lot more to it, like for example the initial installation and the binary deployment. Those two things are what I've worked on in the past few weeks now. Out of the first item grew qt-libs, which should ensure that the required libraries like smoke and CommonQt are available easily. This currently works fine for Linux, however I did not get enough time before the Quicklisp release to find testers for Mac OS X. Windows is another problem entirely, one that I can only solve through downloading of precompiled libraries. I've wasted the entire day today with trying to get 64bit versions of the smoke libraries compiled on Windows. Hopefully I can push through with that and allow easy setup of a Qt environment on Windows as well. Qt-libs builds fine on Mac OS X now as well, though there's currently an issue remaining in loading the libraries. I'll get that sorted out before the next Quicklisp release though.

The second part grew into Qtools' new deployment system part. This allows really convenient and easy generation of ready-to-ship binaries of your application. The only thing you have to do is update your system definition a little:

(asdf:defsystem :my-system
  :defsystem-depends-on (:qtools)
  :build-operation "qt-program-op"
  :build-pathname "binary-name"
  :entry-point "my-package:start-function-or-main-class")

Once these four lines are added, you can simply launch your implementation from a shell, invoke (asdf:operate :program-op :my-system) and it'll do all the magic -like closing foreign libraries before dump, restoring the proper library search paths after resume, reloading the foreign libraries again using the new paths, etc.- for you. All you'll get is a bin folder in your project folder that you can zip and ship. I've tried this for Halftone and it Just Works™ on Linux so far.

But, the road ahead is still long and twisted. Once deployment and installation work flawlessly, there's still a lot of code left to be written to make working with Qt itself less painful. Hopefully some day I'll be able to say that writing native GUIs in Lisp is actually a nice experience!

The Qtools documentation is long and extensive. It contains a lot of talk on both how to start using Qtools, as well as what the internals are and how they work. If you're interested, have a read.


Zach BeaneSLIME 2.13 and SBCL 1.2.12 error:&nbsp;The value NIL is not of type POLICY

· 21 days ago

If you run into this error message, there's a quick runtime fix: evaluate (sb-ext:restrict-compiler-policy 'safety)

You can also add that form to your .sbclrc.

You can also update to SLIME 2.14 or downgrade to SBCL 1.2.11. Unfortunately, SLIME 2.14 isn’t in the recent June Quicklisp update, but I might do a quick second update to fix this problem.

Quicklisp newsJune 2015 Quicklisp dist update now available

· 21 days ago
This Quicklisp update is supported by my employer, Clozure Associates. If you need commercial support for Quicklisp, or any other Common Lisp programming needs, it's available via Clozure Associates.
New projects:
  • cambl — A library for working with financial amounts involving multiple commodities. — BSD-3
  • cerberus — A Kerberos implementation — MIT
  • cl-ledger — Double-entry accounting system. — BSD-3
  • cl-libssh2 — Libssh2 bindings — MIT
  • cl-sane — Lispy library bindings for sane. — GPLv3
  • docparser — Parse documentation from Common Lisp systems. — MIT
  • fn — Some macros for lambda brevity — Public Domain
  • frpc — An ONC-RPC implementation. — MIT
  • glass — General Lisp API Security System. — MIT
  • glkit — Various utilities for OpenGL — MIT
  • integral-rest — REST APIs for Integral DAO Table. — MIT
  • legion — Simple worker threads with a queue. — BSD 2-Clause
  • lime — A high-level Swank client, like Slime, but for Common Lisp applications. — MIT
  • mathkit — Various utilities for math — MIT
  • or-glpk — Foreign interface to the GNU Linear Programming Kit. — LGPL3
  • pounds — Lisp block storage, provides portable file mappings amongst other things. — MIT
  • qt-libs — System to ensure that the necessary Qt libs are available. — Artistic
  • restful — Spin up new REST entities like madman — MIT License
  • swank-protocol — A low-level Swank client. — MIT
  • temporal-functions — A means of creating functions that have an internal concept of time — 2 Clause BSD
  • utilities.binary-dump — Formatting of binary data similar to the od(1) UNIX program. — LLGPLv3
  • varjo — Common Lisp -> GLSL Compiler — LLGPL
Updated projects: apply-argv, arrow-macros, asdf-dependency-grovel, asdf-encodings, asdf-finalizers, asdf-linguist, asdf-package-system, avatar-api, babel, bit-smasher, black-tie, blackbird, blackthorn-engine, bordeaux-fft, buffalo, burgled-batteries, burgled-batteries.syntax, caveman, cells, cffi, chanl, city-hash, cl+ssl, cl-6502, cl-abnf, cl-ana, cl-annot, cl-autowrap, cl-bencode, cl-bibtex, cl-charms, cl-cli-parser, cl-coveralls, cl-cron, cl-csv, cl-dbi, cl-dot, cl-dropbox, cl-durian, cl-emb, cl-factoring, cl-ftp, cl-fuse-meta-fs, cl-gendoc, cl-geometry, cl-glfw3, cl-gpu, cl-growl, cl-influxdb, cl-isaac, cl-launch, cl-lexer, cl-libpuzzle, cl-libusb, cl-libuv, cl-llvm, cl-marklogic, cl-memcached, cl-messagepack, cl-mlep, cl-mustache, cl-netstring-plus, cl-nxt, cl-odesk, cl-pass, cl-pdf, cl-plplot, cl-ppcre, cl-primality, cl-project, cl-protobufs, cl-qrencode, cl-quickcheck, cl-rabbit, cl-recaptcha, cl-rethinkdb, cl-rlimit, cl-rrt, cl-sam, cl-sdl2, cl-shellwords, cl-slug, cl-smtp, cl-sophia, cl-strftime, cl-string-match, cl-tk, cl-unification, clack, classimp, cletris, clim-widgets, clinch, clipper, clos-diff, closer-mop, coleslaw, colleen,, command-line-arguments, common-doc, common-doc-plump, common-html, contextl, crane, croatoan, css-selectors, daemon, dartsclhashtree, defclass-std, defpackage-plus, dissect, djula, dyna, eazy-gnuplot, eazy-process, eazy-project, eco, eos, escalator, esrap, esrap-peg, event-glue, exscribe, fare-csv, fare-memoization, fare-mop, fare-quasiquote, fare-utils, fast-io, fft, find-port, gendl, glaw, glop, glu-tessellate, hdf5-cffi, hermetic, html-template, http-parse, hu.dwim.asdf, hu.dwim.common, hu.dwim.common-lisp, hu.dwim.computed-class, hu.dwim.debug, hu.dwim.def, hu.dwim.defclass-star, hu.dwim.delico, hu.dwim.logger, hu.dwim.partial-eval, hu.dwim.perec, hu.dwim.quasi-quote, hu.dwim.rdbms, hu.dwim.reiterate, hu.dwim.serializer, hu.dwim.stefil, hu.dwim.syntax-sugar, hu.dwim.uri, hu.dwim.util, hu.dwim.walker, hu.dwim.web-server, ieee-floats, imago, inferior-shell, inner-conditional, inotify, intel-hex, ip-interfaces, jonathan, jwacs, kebab, lack, lambda-gtk, lambda-reader, lass, let-over-lambda, lfarm, linedit, lisp-executable, lisp-gflags, lisp-interface-library, lisp-invocation, lisp-namespace, lispbuilder, local-time, lparallel, lucerne, lw-compat, magicffi, md5, meta, mexpr, mgl, mgl-pax, micmac, misc-extensions, mixalot, modf, modf-fset, modularize, modularize-interfaces, myweb, named-readtables, nibbles, ningle, npg, opticl, osicat, pal, parse-js, periods, perlre, pg, plump, png-read, pooler, postmodern, projectured, protobuf, pzmq, qlot, qtools, query-fs, quri, random, rcl, readable, reader-interception, repl-utilities, retrospectiff, rfc3339-timestamp, rock, rpc4cl, rpm, rucksack, s-xml, scalpl, scriba, scribble, sdl2kit, serapeum, shuffletron, single-threaded-ccl, sip-hash, smackjack, smug, snappy, software-evolution, st-json, staple, stem, stumpwm, swank-client, swank-crew, sxql, temporary-file, thorn, trivia, trivia.balland2006, trivial-download, trivial-extract, type-i, type-r, unix-options, unix-opts, usocket, utilities.print-items, utils-kt, verbose, vertex, vgplot, websocket-driver, weft, with-c-syntax, woo, wookie, workout-timer, wuwei, xhtmlgen, zip, zlib, zs3.

Removed projects: arnesi+, asdf-contrib, asdf-project-helper, asdf-utils, until-it-dies.

arnesi+ has been removed because its repo has disappeared and its authors have not replied to inquiries in months.

asdf-contrib and asdf-utils have been removed by request of the author. asdf-project-helper has stopped working as a result.

until-it-dies has never actually worked, but was previously included because some of its auxiliary systems worked.

To get this update, use (ql:update-dist "quicklisp")

Fernando BorrettiCommon Lisp with Travis and Coveralls

· 24 days ago

Travis is a service for running unit tests in the cloud. Every commit you make and every pull request your project receives is seen by Travis, which spins up a Docker container and runs your tests. With a little work, it supports Common Lisp.

Travis is controlled by a file in your repo's root, .travis.yml. It defines some options, like the language the project is written in, the list of shell commands that have to be executed prior to running the tests, and whom to send notifications to.

YAML is an easy to read format for structured data. Tools like Ansible, Chef and Salt have made it popular for configuration management and deployment. The result is basically a shell script, but structured into different sections.

To test a Common Lisp project, we use cl-travis. This provides a script that you can tell Travis to download, which sets up the Common Lisp implementation of your choice (you can test on many - more on that below), and installs Quicklisp.

Without further ado, this is what the simplest .travis.yml looks like:

language: common-lisp
sudo: required

    - LISP=sbcl
    - LISP=ccl

  # Install cl-travis
  - curl | bash

  - cl -l fiveam
       -e '(setf fiveam:*debug-on-error* t
                 fiveam:*debug-on-failure* t)'
       -e '(setf *debugger-hook*
                 (lambda (c h)
                   (declare (ignore c h))
                   (uiop:quit -1)))'
       -e '(ql:quickload :my-project-test)'


The first two items just define the language of the project and tell it that sudo is required to run the tests. cl-travis requires sudo, so we'll have to set it to required at least for now.

Every item in the env.matrix list will create a new build with a certain configuration of environment variables. In this case, we want to test on both SBCL and CCL, so we use this:

    - LISP=sbcl
    - LISP=ccl

The install list is just a list of shell commands to execute to set up the test environment. Here, we just download and install cl-travis:

  # Install cl-travis
  - curl | bash

Projects which require system libraries to run the tests, like Crane, can install and configure those in the install list:

  # Install cl-travis
  - curl | bash
  # Install the latest versions of the major dependencies
  - git clone quicklisp/local-projects/sxql
  - git clone quicklisp/local-projects/cl-dbi
  # Update package repos
  - sudo apt-get update
  # Install SQLite
  - sudo apt-get install -y sqlite3
  # Set up Postgres
  - sudo -u postgres createdb crane_test_db
  - sudo -u postgres psql -c "CREATE USER crane_test_user WITH PASSWORD 'crane_test_user'"
  - sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE crane_test_db TO crane_test_user"

For projects that you'd like to test using the latest version of dependencies, you can clone them to the ~/lisp folder. For instance, this is the install list of the .travis.yml file for Scriba:

  # Install cl-travis
  - curl | bash
  # Clone the latest common-doc
  - git clone ~/lisp/common-doc
  # Clone the latest common-doc-plump
  - git clone ~/lisp/common-doc-plump

Finally, the script is the actual testing itself. cl-travis install CIM, a command line utility for managing and running different Lisp implementations under a common interface.

The cl command launches a Lisp image, and the -l flag can be used to Quickload a library. The -e flag lets us execute code, and here's where we set up what happens on failure and how to run the tests.

If you're using FiveAM for testing, you need to tell it to enter the debugger on test failures and errors. Then, hook up the debugger to UIOP's1 implementation-independent quit function. This ensures that on a test failure the script exits with -1, which tells Travis the tests have failed. Then, we just Quickload the test system to run the tests:

  - cl -l fiveam
       -e '(setf fiveam:*debug-on-error* t
                 fiveam:*debug-on-failure* t)'
       -e '(setf *debugger-hook*
                 (lambda (c h)
                   (declare (ignore c h))
                   (uiop:quit -1)))'
       -e '(ql:quickload :my-project-test)'

If you're using fukamachi's prove for testing, you use this:

    - cl -l prove -e '(or (prove:run :my-project-test) (uiop:quit -1))'

Enabling Travis

To use Travis, you need to sign up with your GitHub account. Then hover over your name in the upper right-hand corner of the page and go to your profile page. This will give you the following page:

Travis profile page

If you're just pushed the repo, chances are you need to click on 'Sync' to update the list of repos.

Then you click on the switch next to the repo's name to enable it, and then all you have to do is push a commit to trigger a build. Travis, like all services, has its ups and downs in terms of availability. So some times builds will start almost instantaneously, other times they'll take some time.

Coverage Testing

Code coverage is basically how many lines of source code are run by tests. SBCL supports coverage measuring, and can generate some HTML reports of coverage, but it requires some manual operation.

Enter Coveralls: This is a service that takes raw code coverage data and tracks it. It shows you covered files, which lines are executed and which are not, the evolution of coverage over time, and also tells you what a pull request will do to coverage.

Coveralls works with Travis, so now extra files are needed: You run the code coverage in the Travis build, along with the tests, and send the data to Coveralls for tracking. The library that does all this is cl-coveralls, which provides a macro that wraps some code in coverage measuring.

To add Coveralls support to the .travis.yml file, we first set the COVERALLS environment variable to true for a particular implementation (preferably SBCL):

    - LISP=sbcl COVERALLS=true
    - LISP=ccl

Then, we clone cl-coveralls:

  # Coveralls support
  - git clone ~/lisp/cl-coveralls

In the script part, we load coveralls along with our testing framework, and then wrap the code that runs tests in the with-coveralls macro:

  - cl -l fiveam -l cl-coveralls
       -e '(setf fiveam:*debug-on-error* t
                 fiveam:*debug-on-failure* t)'
       -e '(setf *debugger-hook*
                 (lambda (c h)
                   (declare (ignore c h))
                   (uiop:quit -1)))'
       -e '(coveralls:with-coveralls (:exclude (list "t"))
             (ql:quickload :my-project-test))'

Note how we used the :exclude option to prevent testing code from falling into coverage tracking.

Enabling Coveralls

The process is similar to enabling a repo for Travis:

Coveralls profile page

You flick the switch to enable or disable a repo, and if the repo is new, click on 'Sync GitHub Repos' up there near the top of the page.


Now, the whole point of this is letting users know what state the software is in. Both Travis and Coveralls give each project a status badge, and image you can put in the README to let users know upfront that the project's in working order and what the coverage status is.

Here's the Markdown for Travis and Coveralls badges:

# Project Name

[![Build Status](](
[![Coverage Status](](


Below is a (necessarily incomplete) list of projects using Travis and/or Coveralls:


  1. This is ASDF's portable tools layer. It provides a few very useful things like finding your hostname, quitting the Lisp image, or finding the system's architecture in a reliably portable way.

Zach BeaneCEPL video series on YouTube

· 26 days ago

Baggers's CEPL talk at ELS was one of the highlights. He demonstrated how a small bit of code could generate some very interesting visual effects, and it could be updated and modified on the fly from within slime.

He's started to upload new CEPL videos to YouTube. Check out his video list and subscribe if you want to keep up.

Quicklisp newsMay 2015 download stats

· 28 days ago
Here are the top 100 downloads for May, 2015:
 5093  alexandria
3865 babel
3442 cl-ppcre
3296 trivial-features
3109 cffi
3023 usocket
2979 cl+ssl
2821 bordeaux-threads
2723 flexi-streams
2720 trivial-gray-streams
2702 trivial-garbage
2621 cl-fad
2588 nibbles
2442 chunga
2390 closer-mop
2380 chipz
2326 cl-base64
2266 drakma
2247 split-sequence
2160 ironclad
2104 anaphora
2100 puri
1792 iterate
1758 trivial-backtrace
1658 slime
1618 local-time
1371 md5
1268 named-readtables
1152 metabang-bind
1104 hunchentoot
1071 let-plus
1067 cl-unicode
1009 cl-colors
953 cl-interpol
936 trivial-utf-8
874 cl-ansi-text
862 prove
851 plump
851 cl-utilities
849 optima
842 jsown
825 uuid
821 parse-number
816 trivial-indent
815 trivial-types
806 array-utils
804 lquery
794 postmodern
791 quicklisp-slime-helper
787 rfc2388
770 clss
766 lparallel
731 fiveam
723 ieee-floats
710 quri
696 asdf-system-connections
661 cl-annot
642 metatilities-base
641 cl-containers
603 cl-sqlite
573 cl-syntax
571 command-line-arguments
564 salza2
538 py-configparser
531 cl-json
525 cl-abnf
524 garbage-pools
523 cl-log
522 dynamic-classes
521 cl-markdown
517 cl-mssql
516 buildapp
507 cl-who
500 static-vectors
498 asdf-finalizers
491 clack
482 fast-io
468 zpng
466 cl-vectors
452 fast-http
449 proc-parse
408 esrap
400 osicat
397 trivial-shell
394 fare-utils
389 zpb-ttf
387 cl-csv
385 clx
371 vecto
364 jonathan
360 fare-quasiquote
354 parenscript
336 closure-common
333 cl-coveralls
327 xsubseq
322 stefil
319 ningle
312 cxml
309 cl-yacc
292 lack

Zach BeaneCorman Lisp 3.02 now available

· 29 days ago

Thanks to some great work by Chun "binghe" Tian and testing and release wrangling by Luís Oliveira, you can now download a Corman Lisp zip file from GitHub and run it directly on modern Windows systems without building anything or entering a license code. Everything I just tried on my Windows 8.1 VM on VirtualBox worked fine right out of the zip.

Thanks, binghe and luis!

Christophe Rhodeslots of jobs in computing at goldsmiths

· 30 days ago

There's an awkward truth that perhaps isn't as well known as it ought to be about academic careers: an oversupply of qualified, motivated candidates chasing an extremely limited supply of academic jobs. Simplifying somewhat, the problem is: if one tenured professor (or "lecturer" as we call them in the UK) is primarily responsible for fifteen PhDs over their career, then fourteen of those newly-minted doctors will not get permanent jobs in academia.

That wouldn't be a problem if the career aspirations of people studying for doctorates were in line with the statistics - if about one in ten to one in twenty PhD candidates wanted a job in academia, then there would be minimal disappointment. However, that isn't the case; many more doctoral students have the ambition and indeed the belief to go on and have an academic career: and when belief meets reality, sometimes things break. Even when they don't, the oversupply of idealistic, qualified and motivated candidates leads to distortions, such as a large number of underpaid sessional teaching staff, assisting in the delivery of courses to ever larger cohorts of students (see also). The sector hasn't sunk as low as the "unpaid internship" seen in other oversupplied careers (games, journalism, fashion) - though it has come close, and there are some zero-hour contract horror stories out there, as well as the nigh-on-exploitative short-term postdocs that are also part of the pyramid.

All this is a somewhat depressing way to set the scene for our way of redressing the balance: Goldsmiths Computing is hiring to fill a number of positions. Some of the positions are traditional lecturer jobs - fixed-term and permanent - and while they're good openings, and I look forward to meeting candidates and working with whoever is successful, they're not what's most interesting here. We have also allocated funds for a number of post-doctoral teaching and research fellowships: three year posts where, in exchange for helping out with our teaching, the fellows will be able to pursue their own research agenda, working in collaboration with (but not under the direction of) established members of staff. I think this is a hugely positive move, and a real opportunity for anyone interesting in the particular kinds of areas of Computing that we have strengths in at Goldsmiths: Games and Graphics, Music and Art Computing, Data and Social Computing, Human-Computer Interaction and AI, Robotics and Cognition. (And if applicants were to want to work with me on projects in Music Informatics or even involving some programming language work, so much the better!)

The complete list of positions we're hoping to fill (apply by searching for the "Computing" Department in this search form) is:

  • Lecturer in Computational Art - 0.5FTE, 3 year fixed-term
  • Lecturer in Computer Science - full-time, 3 year fixed-term
  • Lecturer in Computer Science - 0.5FTE, 3 year fixed-term
  • Lecturer in Games and Graphics - full-time, open-ended
  • Lecturer in Games Art - 0.5FTE, open-ended
  • Lecturer in Physical Computing - full-time, open-ended
  • Post-doctoral Teaching and Research Fellow - full-time, 3 year fixed-term

The deadline for applications for most of these posts is Monday 8th June, so get applying!

Luís OliveiraMy answer to the Pretty Printer Puzzle

· 31 days ago
As promised, I'll describe my solution to the Pretty Printer Puzzle I proposed last week. To recap, we wish to pretty print a Lisp form to a string and identify the textual positions of arbitrary subforms therein.

First attempt

A couple of folks proposed a clever solution that goes like this: (1) replace the CAR of each subform with some unique token (a gensym should be close enough), (2) pretty-print that, (3) find the token positions and replace them with the original CARs.

Problem with this solution: changing the form affects pretty-printing. In particular, it will no longer be able to properly indent macros and special forms.

Second attempt

Another approach is to pretty-print then read the form back and track positions by either using a custom reader that keeps track of form positions (such as hu.dwim.reader) or instrumenting the standard readtable by wrapping the #\( reader-macro and doing the reading from a so-called form-tracking-stream.

Problem with this solution: it breaks down if the form contains unreadable objects.

Third attempt, getting closer

The pretty printer is customisable through a pprint-disptach-table. It is analogous to the reader's readtable. So, we try and instrument it like in the previous approach. Each time a list is about to be pretty-printed, we store the current position in the output stream.

Problem: we have been defeated by the pretty printer's intermediate buffer. Turns out the pretty printer only writes to the output stream at the very end of the process. Back to the drawing board.

Fourth and final attempt

But these attempts have not been in vain, and my final solution involves elements from all three. It goes like this:
  1. Pretty print the form normally.
  2. Pretty print the form again, this time instrumenting the pprint-dispatch-table to wrap lists with some token identifying the subform being printed. (I decided to use the unicode range U+E000..U+F8FF which is reserved for private-use, which seemed neat.) This messes up the pretty-printing a little bit, but not too much, it turns out.
  3. Cross-reference the token positions in #2 with #1 by taking advantage of the fact these outputs differ by whitespace (and tokens) only!

And that's it!

With this tool in hand, there are some interesting tools that can be built in SLIME, but that's another blog post. :-)

Patrick SteinWho Won?

· 33 days ago

The web comic XKCD recently published the following tournament bracket featuring match-ups like ORSON WELLS vs. H.G. WELLS and VAN HALEN vs. VAN MORRISON vs. VAN WILDER.

XKCD Tournament

So, who won? It seems probable to me that XKCD’s author Randall Munroe has in mind some way to decide these matches. If he published how the matches were to be decided, I missed it. However, based on some previous XKCD comics like Geohashing and Externalities, I think it’s a safe guess that it involves hashing.

So, how could we decide this? We could take the hash of each name and then in each match, the largest hash value wins. That, however, has the unfortunate side effect that the winner of the tournament would be the same regardless of the organization of the brackets.

I opted to decide the match between OSCAR DE LA RENTA and OSCAR DE LA HOYA by taking the SHA3-512 hash of the strings OSCAR DE LA RENTA vs. OSCAR DE LA HOYA and OSCAR DE LA HOYA vs. OSCAR DE LA RENTA. The winner is the one whose name appeared first in the string with the smallest hash value. For three and four person contests, I used all permutations of the players involved (separated by vs. ).

The winner? RYAN ADAMS beat out BILL PAXTON in the final.

The code for this project was a breeze thanks to #'ALEXANDRIA:MAP-PERMUTATIONS and my TRACK-BEST library. Here is a the meat of the whole thing which uses an evaluation function (here, it’s SHA3-512 of the vs. separated player list) and a way to compare the evaluations (here, a simple #'ARRAY-LESSP) and runs through all of the players.

(labels ((rank-one-match (players depth)
           (track-best:track (first players)
                             (funcall eval-permutation players depth)))

         (find-winner (players depth)
           (track-best:with-track-best (:order-by-fn compare-permutations)
             (alexandria:map-permutations (lambda (players)
                                            (rank-one-match players depth))

Here is the full source file for the tournament: tourney.lisp. Here is a text description of the whole tournament. And, here is a graphic with the outcomes of all of the matches.


Quicklisp newsLooking for more metadata

· 33 days ago
A few days ago I linked to a report showing a lot of systems that failed to build. They failed because I added an option in the Quicklisp build environment that signals an error if a system lacks the description, author, and license metadata.

This isn't a standard feature of ASDF or Quicklisp. No projects are going to be dropped next month because of it. It's an optional piece of the build system, one that I added so I could see how many systems are missing that useful data and how likely it is that people will care.

I really want to use the :description option and show it as output in the REPL when searching for systems with something like system-apropos. I also want to make it easy to quickly determine the license of a given system, so you can figure out if it's compatible with your project. And having author information readily available will make it easier to contact someone regarding the project.

ASDF system metadata is a good choice for storing this information because it's not Quicklisp-specific. Anyone can gather and use this data if it's present in the systems. I hope that in the future every system in every project will have as much useful and accurate metadata as possible.

So what should you do if you want to help with this goal?

First, if you maintain a system and it's in the report, please update each of its system definitions with :description, :author, and :license information. A good description should be no longer than a tweet,and give an idea of what the system is for. The author information should include a name and email address. The license should be short and refer to a well-known license if possible, or give information about where to read the full license otherwise.

If you're not the maintainer of a system, but you want to file an issue or bug report, consider making a polite request to the author that they update their systems to include the extra info. (If you can, make sure nobody else has submitted the request first.)

Zach BeaneGoogle-hosted Lisp Talk in Boston

· 35 days ago

On Tuesday, June 9th, Dr. Christian Schafmeister will present his talk on Clasp at the Google office at Kendall Red Line station. It’s the talk he presented at ELS last month, and worth seeing if you can make it.

Martin Cracauer is hosting, and has written up all the details on the boston-lisp list.

Luís OliveiraPretty printer puzzle

· 38 days ago
This past week, I came across a Lisp challenge that turned out to be trickier than one might expect at first. I needed to pretty-print a Lisp form to a string and identify the positions of certain subforms. Here's an example:
CL-USER> (with-output-to-string (*standard-output*)
           (pprint '(defun factorial (x) (if (zerop x) 1 (* x (factorial (1- x)))))))
      (* X (FACTORIAL (1- X)))))"
In this, output the bounding indices of the IF form are 24 and 77. In other words:
CL-USER> (subseq * 24 77)
      (* X (FACTORIAL (1- X))))"
The challenge, then, is to write a function that, given a form and list of subforms, returns a string with the pretty-printed output and a list of bounding indices for each subform. E.g.
CL-USER> (pprinted-bounds '(defun factorial (x) #1=(if (zerop x) 1 (* x #2=(factorial (1- x)))))
                          (list '#1# '#2#))
      (* X (FACTORIAL (1- X)))))"
((24 77) (57 75))
I'll post my solution later. Have fun. :-)

drmeisterClasp has a developers mailing list

· 42 days ago

The mailing list is

If you are interested in following developments to the next release you can join at:

drmeisterTagged pointers and immediate fixnums, characters, and single-floats in Clasp

· 45 days ago

In preparation for making Clasp generate performant code, I’ve switched to using tagged pointers and immediate fixnums, characters and single-floats.  This is a common technique in dynamic language implementations where you represent small values directly within pointers rather than storing them on the heap. The tagging scheme in Clasp is as follows for 64-bit systems (32-bit is not supported at the moment).

A pointer provides 64-bits (63 … 0) to work with.

FIXNUM 0b0 The other 63 bits are used to store a signed FIXNUM. The 0b0 bit allows certain arithmetic operations (addition, subtraction, comparison) to be very fast.
OTHER 0b001 In the lower three bits represents a pointer to an object on the heap.  The pointer is 8-byte aligned in memory.
CONS 0b011 A pointer to a CONS cell. Robert Strandh suggested this so that Common Lisp code can easily distinguish CONS cells from non CONS cells and speed up list traversal.
FRAME 0b101 A pointer to an array on the stack.  I’m not sure if I’ll continue to use this going forward – it’s really just for bootstrapping. I may reuse this tag to represent a pointer to C++ objects on the C++ heap.
CHARACTER 0b00111 A character in a 32-bit value (>> 5 to unbox).
SINGLE-FLOAT 0b01111 Represents a single precision C/C++ float, a 32-bit value (>>5 to unbox).
UNUSED1 0b10111 Suggestions?
UNUSED2 0b11111 Suggestions?

I’m especially interested in peoples thoughts on embedding something like a double-precision float within 61 bits. I’m 99.9(repeating)% sure it can’t be done but I’d love something like that. Arithmetic, Arrr! she be a harsh mistress.

All pointer access in Clasp is performed using a C++ template class called smart_ptr and it was relatively easy to modify this class to manage the tagged pointers. The untagging/tagging of OTHER-PTR and CONS-PTR pointers is carried out by overloading the C++ dereferencing operators: operator-> and operator*, of the smart_ptr template class.

It was a bit tricky to get these immediate values to play nice with C++ types and inheritance because the Clasp C++ code does not treat all Common Lisp types as a single undifferentiated pointer type. Clasp uses C++ types like T_sp, List_sp, Cons_sp, Float_sp, Symbol_sp, HashTable_sp etc. to distinguish different Common Lisp pointer types from each other within the C++ code. An assignment like:

Symbol_sp y(...);
T_sp x = y;

is valid and very common within the Clasp C++ code. But:

T_sp a(...);
Symbol_sp s = gc::As<Symbol_sp>(a);

will signal a Common Lisp error if a doesn’t point to a SYMBOL at the time of the assignment to s.

To deal with this I defined a template class: TaggedCast<ToPtr,FromPtr> that provides an “isA” function and a “castOrNULL” function that can be specialized to simulate inheritance between classes that represent immediate values to the regular C++ class hierarchy.

Zach BeaneCLISP supporting libraries need maintainers, too

· 50 days ago

Jörg Höhle: “it seems to me that part of the difficulties with compiling, packaging and obtaining clisp stem from the core libraries that it uses, namely libffi and esp. libsigsegv. Both were created by Bruno Haible decades ago and both are in need of a maintainer.”

You can read the entire article on gmane.

Sam Steingold follows up: “note that libffi is actively maintained[, ] clisp uses libffcall”

Quicklisp newsASDF 3 is coming to Quicklisp soon

· 53 days ago
In the next week or two, I'm going to update Quicklisp so that if ASDF is missing, or not at least version 3.1, it will fetch and load ASDF 3.1.4. This should be a pretty conservative change. Almost all implementations already include ASDF 3.

If you'd like to try the new Quicklisp client that fetches ASDF 3 if needed, see this post to the Quicklisp mailing list.

Zach BeaneCLISP needs maintainers

· 54 days ago

Jörg Höhle writes: "[…] we’re looking for a maintainer, i.e. somebody who has time to donate to assess patches, test and build a new release. It would be nice if somebody younger than us, the old guys, who have been with clisp for over 20 years, would jump in and care."

Quicklisp newsMay 2015 Quicklisp dist update now available

· 56 days ago

This Quicklisp update is supported by my employer, Clozure Associates. If you need commercial support for Quicklisp, or any other Common Lisp programming needs, it's available via Clozure Associates.

New projects:

  • bytecurry.asdf-ext — ASDF extension(s) for generating atdoc documentation. — MIT
  • cl-durian — dynamic html generation from list structures (interpolation friendly) — WTFPL
  • cl-marklogic — Common Lisp library for accessing MarkLogic Server. — LGPL3
  • cl-pslib — A CFFI wrapper for the pslib library, a library for generating PostScript files. — LLGPL
  • cl-pslib-barcode — A barcode generator for the cl-pslib library. — LLGPL
  • cl-simple-concurrent-jobs — A simple API for running concurrent jobs and collecting the results — BSD 2-Clause
  • cl-sophia — High-level API for Sophia key-value storage — WTFPL
  • cl-strftime — Common Lisp compiler for the strftime language. — MIT
  • cl-tga — TGA file loader — MIT
  • erudite — Literate Programming System for Common Lisp — MIT
  • intel-hex — A library to handle Intel HEX format. — MIT
  • snakes — Python style generators for Common Lisp. — Apache 2.0

Updated projects: alexandria, antik, arrow-macros, babel, beirc, birch, blackbird, buffalo, bytecurry.mocks, caveman, chanl, chipz, cl+ssl, cl-ana, cl-async, cl-charms, cl-enumeration, cl-gobject-introspection, cl-grace, cl-graph, cl-i18n, cl-launch, cl-mtgnet, cl-netstring-plus, cl-ply, cl-quickcheck, cl-rabbit, cl-read-macro-tokens, cl-readline, cl-rethinkdb, cl-sdl2, cl-singleton-mixin, cl-slug, cl-voxelize, cl-yaml, clack, clack-errors, clim-widgets, climacs, clinch, clipper, closer-mop, colleen, common-doc, common-html, common-lisp-stat, commonqt, corona, dartsclhashtree, dartsclmessagepack, defclass-std, dyna, eazy-process, exscribe, f2cl, fare-csv, fast-http, function-cache, gbbopen, generic-comparability, gsll, hu.dwim.delico, hu.dwim.stefil, hu.dwim.syntax-sugar, hu.dwim.util, immutable-struct, inferior-shell, jonathan, json-responses, lass, let-over-lambda, lev, lisp-interface-library, lisp-invocation, lisp-matrix, lquery, lucerne, metap, mexpr, mgl-pax, nibbles, ningle, nsort, perlre, plump, proc-parse, qlot, qtools, quadtree, quasiquote-2.0, scalpl, scriba, scribble, serapeum, shellpool, should-test, shuffletron, staple, stmx, stumpwm, thorn, transparent-wrap, trivial-download, usocket, with-c-syntax, wookie.

To get this update, use (ql:update-dist "quicklisp").

Quicklisp newsApril 2015 download stats

· 56 days ago
Here are the top 100 downloads for last month:
 4230  alexandria
3401 trivial-features
3377 babel
3366 cl-ppcre
3102 cffi
2717 bordeaux-threads
2620 closer-mop
2611 trivial-garbage
2605 cl-fad
2575 flexi-streams
2574 trivial-gray-streams
2471 cl+ssl
2366 nibbles
2245 usocket
2228 cl-base64
2210 split-sequence
2191 chunga
2087 slime
2047 iterate
2046 trivial-backtrace
2042 drakma
2002 anaphora
1925 ironclad
1691 puri
1682 chipz
1653 local-time
1520 named-readtables
1455 md5
1399 hunchentoot
1259 metabang-bind
1152 cl-colors
1124 let-plus
1070 optima
1051 cl-unicode
1020 trivial-utf-8
992 cl-syntax
950 rfc2388
946 cl-interpol
943 trivial-types
941 cl-annot
934 cl-ansi-text
846 postmodern
839 prove
803 parse-number
799 asdf-system-connections
791 uuid
780 cl-utilities
765 cl-containers
765 metatilities-base
760 quicklisp-slime-helper
701 fast-io
695 jsown
672 ieee-floats
671 lparallel
664 static-vectors
658 cl-json
640 plump
605 fiveam
601 zpng
594 lquery
590 trivial-indent
587 clss
578 buildapp
566 array-utils
559 xsubseq
557 cl-sqlite
547 salza2
546 quri
542 command-line-arguments
540 osicat
521 garbage-pools
520 fast-http
515 cl-mssql
509 cl-who
508 cl-vectors
506 clx
505 iolib
503 py-configparser
499 dynamic-classes
497 asdf-finalizers
497 cl-log
496 cl-marshal
494 cl-markdown
484 trivial-mimes
483 cl-abnf
477 clack
465 fare-utils
450 ningle
418 zpb-ttf
408 cl-dbi
408 st-json
404 http-body
403 circular-streams
398 closure-common
392 fare-quasiquote
380 cl-csv
380 cxml
377 parenscript
373 myway
352 map-set

Zach BeaneLispWorks 7.0 is released

· 57 days ago

Dave Fox just posted this message to the LispWorks mailing list:

LispWorks Ltd is pleased to announce the release of LispWorks 7.0 on Windows®, Macintosh®, x86/x86_64 Linux®, ARM Linux®, FreeBSD®, AIX®, x86/x64 Solaris™ and SPARC/Solaris™ platforms. Also, new LispWorks for Mobile Runtime products target Android and iOS apps.

LispWorks 7.0 brings these new features:

  • 32-bit implementation for ARM Linux.
  • 32-bit and 64-bit implementations for PowerPC/AIX.
  • Java interface.
  • Characters and strings support all Unicode planes.
  • Editor supports Unicode entire range, including Chinese and Japanese characters.
  • Improved documentation including consolidation of CAPI manuals with more cross references, and more self-contained examples.
  • Code coverage tools.
  • Asynchronous socket I/O and UDP sockets.
  • Editor supports more fonts on Cocoa.
  • Support for multi-touch gestures.
  • Graphic Tools API (beta quality).
  • More CAPI enhancements including efficient transient display on output-panes.
  • Improvements in the IDE including Directory mode and buffers list options in Editor.
  • Various other new features including:
    • Thread-safe operations for ensuring a hash table entry.
    • Efficient access for 8-bit simple vectors.
    • FLI type for holding the address of a foreign symbol (suitable for C to Lisp callbacks).
    • 64-bit integer FLI types in 32-bit LispWorks.
    • Efficient raw 64-bit arithmetic and vector access in 64-bit LispWorks.
    • UTF-16 and KOI8-R encodings.
    • CLOS optimizations for copying objects.
    • On Windows, delivered DLLs can use a private copy of the MSVCRT runtime.
    • On Macintosh, the Cocoa IDE’s event loop has better error handling and a new defense against deadlocks.
  • Bug fixes.

Not all features are supported for every LispWorks product and platform. For details please see the feature table at

LispWorks Professional (64-bit implementation now available) and Enterprise Editions still offer excellent value to commercial users and academic institutions with no runtime license fees. An annual maintenance contract is available.

LispWorks Hobbyist and HobbyistDV Editions now offer an affordable fully functional Common Lisp IDE for non-commercial and non-academic use by individuals.

LispWorks for Android Runtime allows you to deliver (non-GUI) Lisp libraries which you can incorporate into Android apps (“mobile Lisp libraries”). It is available as an add-on to customers with LispWorks 7.0, which you use to develop your code before creating the mobile Lisp library. We do not charge runtime license fees for mobile Lisp libraries generated by LispWorks for Android Runtime.

LispWorks for iOS Runtime allows you to deliver (non-GUI) Lisp libraries which you can incorporate into iOS apps. It is available as an add-on to customers with LispWorks 7.0, which you use to develop your code before creating the mobile Lisp library. We do not charge runtime license fees for mobile Lisp libraries generated by LispWorks for iOS Runtime.

More details of the improvements in this release can be found in the Release Notes at

Pricing, sales, and other information is available in the original lisp-hug article.

Vladimir SedachSymbolics MacIvory auctions

· 57 days ago
For anyone interested in acquiring a Symbolics system, here is an email I just received from David K. Schmidt, director of sales at Symbolics:

We are in the process of consolidating two storage facilities into one new one. To reduce the amount that needs to be moved we are liquidating some of the inventory that is not needed for future operations.

Along with auctions for 36xx boards and other spare parts we are going to sell the following 4 MacIvory systems in serial auctions one after the other:

1. MacIvory model 3 with 8 MWords in a Quadra 700 with 20 MBs and a 9GB 10000 rpm disk

2. MacIvory model 3 with 8 MWords in a Mac IIci with 20 MBs, a 4GB 7200 rpm disk and a Radius Accelerated 24-bit graphics card

3. MacIvory model 2 with 2.6 MWords in a Mac IIfx with 20 MBs, a 4GB 7200 rpm disk, a Radius Accelerated 24-bit graphics card and an ethernet card

4. MacIvory model 2 with 1.3 MWords in a Quadra 650 with 64 MBs and a 9GB 10000 disk

I will send you a link to each auction as they are put on line.

If you are interested in any 36xx part that you don't see an auction for, email me for a quote.

You can contact David at

For older items, see the Planet Lisp Archives.

Last updated: 2015-07-01 15:44