How do you test if an object is a cons that has the desired symbol in the car?

(typep x '(cons (eql :foo)))

Sure,

(and (consp x) (eq :foo (car x)))

is essentially just as short...

I still find cons types neat, even if they're a nightmare when it comes to type derivation, but that's a different matter. Some nightmares aren't all bad.

The classes of the Common Lisp spec, the MOP and big CLOS projects like CLIM use some naming conventions which may not make sense at first. I'm speaking of the STANDARD- and BASIC- prefixes. What is the idea behind them?

Protocol Oriented Programming

In CLOS one defines "protocols" using classes and generic functions. Remember: Methods do belong to generic functions and not to classes. There are no dedicated means to define protocols - it is idiomatic. A common practice is to define "protocol classes" which often do not have any slots at all. They serve as superclasses to all classes which implement the protocol.

Basics and Standards

In the set of protocol implementation classes there may be one, which contains a minimal complete implementation and/or there might be one which is something like the default implementation. Assuming a protocol for FOOBAR defined through the protocol-class FOOBAR the minimal implementation could be called BASIC-FOOBAR and the default one STANDARD-FOOBAR. The BASIC-** one if often called FUNDAMENTAL-** too.

Extensibility

It is often a good idea to implement STANDARD-** by inheriting the BASIC-** implementation, overriding the necessary parts. A protocol user who wants to implement his own implementation can decide on which level he wants to get in: He could inherit from STANDARD-** if his implementations is only a small variant. He may inherit from BASIC-** if the implementation is more complicated and perhaps incompatible to what STANDARD-** does. If even BASIC-** is to strict, he could try to implement the protocol class completely on its own - but this is then more like implementing interfaces in classical object oriented languages, where you cannot inherit behaviour at all. You have to provide methods for any necessary generic function the protocol is made of. This is the most work and implementing a protocol is often more than just following the interfaces.

(set-macro-character #\{ (lambda (s c) (read-delimited-list #\} s t)) nil)
(set-macro-character #\} (get-macro-character #\)) nil)

EDIT: I'm not saying Didier is wrong. He isn't. While readmacros are extremely flexible, the system has some rather obvious shortcomings.

I'm just pointing out that making {foo} read as list is trivial in case someone walked away from his post with the impression that it cannot be done.

If there's something I disagree with, it is his characterization of readmacros as "shaky". Used to do things they can do, in the manner they're designed to be used, they're perfectly robust. Just not as nice or convenient as I would like them to be.

The really interesting bit is his observation that READ-DELIMITED-LIST doesn't support dotted lists, which I hadn't realized before. It isn't specified to, but it would be nice if it did...

Common Lisp in the desire to be as cool as possible includes in its specification the Common Lisp Object System, or CLOS, which itself can be introspected and altered in great detail using the MetaObject Protocol, or MOP, as described in The Art of the Metaobject Protocol. Unfortunately, MOP didn’t make it into the ANSI standard, but most implementations include MOP as it is described in the book, and a compatibility package :closer-mop (available in Quicklisp) makes using the symbols described seamless between implementations.

One of the features of MOP is the ability to make an instance of a CLOS class funcallable, that is allow a class instance to be a valid first argument to funcall. This behavior in a lot of ways can resemble the traditional method model from other languages, but that’s not how I intend use it here.

The Idea

I’m going to show and tell an implementation of a generic finite state machine that uses the MOP concepts of funcallable-standard-object and funcallable-standard-class to marshal the flow of incoming events to the machine and the concept of CLOS generic method dispatch to handle the execution of transition handlers for any given state of the machine.

These features will allow us to build a structure that lets us focus solely on the problem at hand and defer features like event data binding, state-dependent method selection and unexpected state handling entirely to the language without pushing the boundaries of any specific feature.

The Design

The design we’re going for is such that we can define a class with a state slot that will hold :keyword name of a state. We’ll make instances of this class funcallable so that when we make an instance we will be able to simply (funcall fsm-instance fsm-evemt) repeatedly and have the machine dispatch to the correct event, perform any logic, and transition to the next state based on the input.

We would be able to query the state of the machine with (state fsm-instance) and receive a keyword, and we should be able to drive events into the machine until we’re in a desired or unexpected state. Any attempt to feed the machine an event while the machine is in an invalid state should result in an error.

Setup

Dependencies in Common Lisp these days should come from Quicklisp. Install it now before even reading the next paragraph, and during the installation phase ask it to write itself to your lisp configuration’s init file.

The code samples below, as should any code using MOP intending to run more than once, uses the :closer-mop package. Fetch it from Quicklisp before attempting these examples by evaluating:

(ql:quickload :closer-mop)

At the REPL before attempting to evaluate any example code.

Implementation

We’ll first declare the base of our state machine. We’ll call the base standard-state-machine to follow the convention CL seems to have of sticking the standard- prefix to base classes.

(defclass standard-state-machine (c2mop:funcallable-standard-object)
  ((state :initform :initial :initarg :state
          :accessor state)
   (last-event :initform (get-internal-real-time)
               :accessor last-event))

  (:metaclass c2mop:funcallable-standard-class))

This grants us the majority of our desired features from the design. The resulting class holds a :keyword state name in a state slot, and is declared to be funcallable by the inclusion of the funcallable-standard-object base and the funcallable-standard-class metaclass for the class itself.

When events fire through this state machine we will need to invoke a method that we can use to dispatch to various handlers in the machine depending on the state we happen to be in. We can use the CLOS generic method feature called eql specializers to distinguish which method should be invoked based on the name of the state a machine happens to be in. EQL specializers are just like type specializers for method parameters, but instead of the comparison of the parameter to the specialization happening based on the type of the parameter, it happens as if an (eql parameter-value parameter-specialization) test is performed to determine if a given method is applicable to an invocation.

With this in mind we can design a generic method that is aware of three things as parameters that we can choose to specialize: The current state machine, the state of the current state machine, and the event that was received by the machine.

(defgeneric standard-state-machine-event (machine state event))

We won’t define any such methods now, because right now we have no state machine to model, and as such have no state handing methods to define.

To actually invoke a function when the funcallable instance is called as a function a lambda must be bound to be the funcallable instance function of a given instance of a class. Since there is no mandate for an inherent this pointer in CL, the best place to bind this function is often in one of the specializations of initialize-instance and close over the current instance in the process. The attachment of a function to an instance is done with the MOP function set-funcallable-instance-function, which places no special requirements on the args list of the attached function.

We’ll bind our driver to the state machine instance in the :before specializer of its initialize-instance method, and use an args list that expects only the event that is being delivered to the state machine:

(defmethod initialize-instance :before ((machine standard-state-machine) &key)
  (c2mop:set-funcallable-instance-function
   machine
   #'(lambda (event)
       (multiple-value-bind (next-state recur-p)
           (standard-state-machine-event machine (state machine) event)

         (setf (last-event machine) (get-internal-real-time)
               (state machine) (or next-state (state machine)))

         (if recur-p
             (funcall machine event)
             (values machine (state machine)))))))

Now when we have instance of the state machine invoked with (funcall fsm-instance event), the inner lambda expression will get passed the event and compute a return. This function despite having some length does very little.

First, it dispatches the event to our previous standard-state-machine-event method using the instance of the state machine we closed over as the first parameter, the state of the state machine as the second parameter and the event that the machine was called with.

It expects a multi-value return, but if the method only returns one value, no error is raised and the second value, recur-p is left bound to nil. Its meaning will become abundant in a moment.

Once the state handler is called the new state returned by the handler is stored as the current state of the state machine, and if a second value was returned, the machine is invoked again with the same event it had just received before returning from the handler the value of the current machine instance for chaining and the value of the current state.

The second, recur-p value is a hook that can be used from a state handler to force the machine to retry handling the event in a new state. This is useful when the same event should loop through the machine a second time before the transition is complete, and the optional second value return from a state handler allows that without additional external plumbing.

Functionally, the implementation of the state machine is complete. It can now cycle through a graph of events, provided that those events are defined as specialized methods. But, in its current state, the API leaves very much to be desired.

The Polish

As the state machine stands now, to implement one that for all inputs remains in the default :initial state, one would have to be defined explicitly by specializing the standard-state-machine event.

(defmethod standard-state-machine-event
            ((fsm standard-state-machine)
             (state (eql :initial))
             event)
  nil)

The trailing nil makes the return value explicit, due to the length of the specializations. This makes the structure of the dispatch clear, and the nil return value could be traced through the driver above to determine that this machine would do absolutely nothing given any kind of input. Remaining eternally in the :initial state.

To make the task of defining states for a given type of state machine we can write a simple macro to write the above form for us. After all, the majority of it is filler.

(defmacro defstate (machine-type state-name (machine-sym event-sym) &body body)
  `(defmethod standard-state-machine-event
       ((,machine-sym ,machine-type) (state (eql ,state-name)) ,event-sym)
     ,@body))

Using the macro above we can now declare an identical looping :initial state for the standard-state-machine as follows:

(defstate standard-state-machine :initial (fsm event)
  nil)

Which results in identical code to the one we wrote previously. The symbols given in the argument list of the defstate forms are bound to the machine evaluating the event and the event that is being sent through the machine. The remaining forms will be evaluated just as in any other defun/defmethod with the return value treated as either next state for the machine to enter, nil to stay in the same state, or a multivalue return with one of the previous and a non-nil second value to indicate the event should be fired through the machine a second time before the final state is stored.

To create subclasses of the standard state machine, we again run into a similar situation as before with boilerplate requirements. Each subclass of standard-state-machine must not only include the standard-state-machine class in the list of parents, it must also include funcallable-standard-class as its metaclass, as in:

(defclass my-fsm (standard-state-machine)
  ()
  (:metaclass c2mop:standard-funcallable-class))

This peculiarity could be documented a thousand times in a thousand places, but that will make it no less ugly or difficult to remember. It would be much easier to provide a familiar tasting API to consumers that results in the same code. For example, using a structure like this to generate code equivalent to the above

(deffsm my-fsm ()
  ())

This leaves the consumer free to do whatever their heart desires with CLOS without disturbing the requirements of our function. Such a construct can be stated simply as another tiny macro.

(defmacro deffsm (name parents slots &rest options)
  `(defclass ,name ,(append (list 'standard-state-machine) parents)
     ,slots
     (:metaclass c2mop:funcallable-standard-class)
     ,@options))

The Result

If we were to put all of that code together in one place with a bunch of documentation strings for the methods, classes and slots in a slightly more condensed form than this article, it might look something like this. Which is actually a minimal extraction of the state machine that drives the HTTP parser in Hinge. If this article gets enough interest, I will extract standard-state-machine into a QuickLisp compatible package for even easier reuse.

The Demo

Let’s construct an FSM for determining if a sequence of characters contains the string “Hi”, but before the character “!” appears. Once a string contains “Hi” it can contain “!” characters again.

We’ll model this as a machine of three states, with events being single characters of input. The states will be :initial, :want-i and :done.

In the :initial state, we’ll accept any input without transitioning except for “!” which will cause the machine to transition into an error state so that no more input can pass, and “H” will cause the machine to transition to the :want-i state.

Any event other than “i” in the :want-i state will transition back to :initial and the event “i” will cause the machine to enter the state :done, which will be a no-op looping state that allows input to pass through uninspected. The implementation for such a machine would look like this, assuming we have previously defined the :fsm package.

(fsm:deffsm hi-fsm ()
  ())

(fsm:defstate hi-fsm :initial (fsm c)
  (case c
    (#\! :error)
    (#\H :want-i)))

(fsm:defstate hi-fsm :want-i (fsm c)
  (if (char-equal c #\i)
      :done
      (values :initial t)))

(fsm:defstate hi-fsm :done (fsm c)

We can observe it in action by running it through a couple of strings and measuring the terminal state of the machine:

(let ((fsm (make-instance 'hi-fsm))
       (input "Oh? Hello there. Hi. How are you!?"))
  (map 'list 
       (lambda (c) 
         (if (eql :error (fsm:state fsm))
             (format t "Skipping: ~S~%" c)
             (funcall fsm c))) 
       input)
  (fsm:state fsm))

Resulting in :DONE as the final state of hi-fsm.

This will iterate the string character by character through the machine we just defined, and unless the machine is in an error state, it will submit each token, then return the state of the machine when it has completed its run.

If we change the string to one where neither “Hi” nor “!” appear at all, the machine will remain in :initial and if “!” appears before “Hi”, then the machine will leave the loop in an :error state, and any tokens after the “!” won’t even be sent into the machine. Changing the binding of input in the above to “Go away! Now.” results in the value of :ERROR and the printing of the lines

Skipping: #\ 
Skipping: #\N
Skipping: #\o
Skipping: #\w
Skipping: #\.

So there you have it. You now know how to leverage the powers of Common Lisp and MOP to build a pretty cool and useful abstraction for a pretty common algorithm.

The slime repl is used to interact with a Common Lisp process, and while printing textual information is good enough most of the time, being able to display an image can be really useful.

I used to simply call an external process, the image viewer feh, but the following method is more interesting.

You first need to load the slime-media module which is provided with slime, for example by adding it to the list of modules to load with slime:

(slime-setup '(slime-repl
               slime-asdf
               slime-fuzzy
               slime-banner
               slime-indentation
               slime-media))

You then have to authorize Slime to evaluate arbitrary lisp forms coming from the Common Lisp process:

(setq slime-enable-evaluate-in-emacs t)

Let's display an image:

It looks pretty good, but I really wanted to have the return value printed below the image, and not on its right.

The following function is a modification of slime-media-insert-image; an end-of-line character is inserted just after the image. This way, the result value is printed below the image. Furthermore, when more than one image is displayed during the same call, it won't be crammed on the same line.

(defun nm-slime-media-insert-image (image string)
  (with-current-buffer (slime-output-buffer)
    (let ((marker (slime-output-target-marker :repl-result)))
      (goto-char marker)
      (slime-propertize-region `(face slime-repl-result-face
                                      rear-nonsticky (face))
        (insert-image image string)
        (insert ?\n))
      ;; Move the input-start marker after the REPL result.
      (set-marker marker (point)))
    (slime-repl-show-maximum-output)))

Here's a little Monday Troll.

To my greatest disappointment, I discovered today that it is not possible to replace Lisp parenthesis by, say, ... curly braces. What a shame. Hell, it's not even possible to freely mix the two. Very naively, I had expected that:

(set-macro-character #\{ (get-macro-character #\())
(set-macro-character #\} (get-macro-character #\)))

would suffice, but no. All implementations that I have tested seem to agree on this, although the error messages may differ. For instance, trying to evaluate {and} gives you an "unmatched close parenthesis error" except for CMU-CL which chooses to ignore it, but then report an end-of-file error. The unmatched close parenthesis, of course, is the closing curly brace! So what is going on here?

When an opening curly brace is read, the original left paren macro function is called. In SBCL for instance, this is SB-IMPL::READ-LIST, which looks for a hardwired right paren on the stream. Yuck. It doesn't find one, but it finds my closing brace which triggers the "standalone" right paren behavior (spurious paren alert). In passing, it also surprised me that SB-IMPL::READ-LIST is not implemented in terms of READ-DELIMITED-LIST.

EDIT: as mentioned in several comments, we could use read-delimited-list to look for a closing curly brace, but even this won't work completely. The problem is with dotted lists (see Pascal's comment). SBCL hard-wires #\) in its dotted lists parsing procedures.

So it appears that dispatching macro characters are only shaky. What we miss is a true concept of syntactic categories (Common Lisp character syntax types are close, but not quite there yet). In fact, TeX, with its notion of catcodes (category codes), seems to be the only language that gets this right. Ideally, any character with associated status LIST TERMINATOR should do as good as a right paren (the problem is only with closing, not opening).

Instead of hard-wiring the right paren in the Lisp parser, a quick workaround would be to check whether the next character on the stream is a dispatching one, and in such a case, whether its macro function is the one originally associated with the right paren. If so, it should then simply stand as a list terminator. This is actually an interesting idea I think: could the built-in macro functions become equivalent to actual category codes, and could we completely remove hard-wired characters in Lisp parsers?

Anyway, this whole story is a true scandal because it ruined an otherwise cool live demo of mine. So much for syntax extensibility. I will immediately complain to the concerned authorities.

Looking for the concerned authorities to complain to... please wait.

I usually like to update Quicklisp libraries on the first weekend of each month. A few things conspired to delay me this time around, but I hope to have updates published by this Sunday, May 13th.

The coming update includes version 0.10.7.1 of CFFI, which has a handful of backwards-incompatible changes. For the most part, Quicklisp projects have been updated to fix compile-time problems with CFFI compatibility. There remains the possibility that some problems lurk that will only appear at runtime. If you work on a project that relies on CFFI, you may want to run it through its paces with the new version to make sure everything still works properly.

update I was wrong about the CFFI updates. There haven't been any backwards-incompatible changes from 0.10.6 to 0.10.7.1. Sorry for the confusion.

Secure Outcomes is looking for a hardcore Lisp developer with deep experience in Common Lisp/CLOS, image processing and artificial intelligence techniques, advanced cryptography, communications, GUI, and embedded real-time. Direct experience in electronic livescan fingerprint collection, archiving, and transmission, and/or crypto NIST certification is a distinct plus.

Secure Outcomes is a venture funded closely-held Colorado company that designs and manufactures the very best electronic livescan fingerprint collection, lookup, and transmission systems available for the price.

We design all of our systems at our headquarters facility in beautiful Evergreen, Colorado at an elevation of 7,200-feet in the foothills of the Rocky Mountains about 25-miles west of Denver.

Secure Outcomes provides terrific opportunities for fiercely talented and driven people.

Our systems combine advanced FBI-Certified fingerprint scan-head devices and super-compact 21st century low-power control electronics and display touchscreens with custom control and analysis software designed with the user in mind to yield the easiest to use, most capable, and most user-friendly systems on the market – all in Lisp.

Evergreen, Colorado, the home of Secure Outcomes, offers an extremely high-quality of life. World class skiing is just about an hour away at Keystone, Vail, Winter Park, and many others. Lake Evergreen offers excellent night/day ice skating and summer water sports/sailing. Mountain and touring biking and hiking abound during the summer months. In addition, Evergreen supports a thriving cultural community that includes the Evergreen Jazz Festival in the summer and the Evergreen Chorale and Evergreen Players for live music and theater year round. Evergreen has 300+ days of annual sunshine.

Competitive salaries. Fully paid medical.

See our products in action at http://www.youtube.com/watch?v=Ti2q8eK0l58

Resumes to lisp@secureoutcomes.net. No calls please.

http://www.secureoutcomes.net/Employment.html

I am sorry. Really, I am. I cannot follow you all in all your usual communication channels: comp.lang.lisp, the ecl forum, #lisp irc channel... But please consider this:

First of all, ECL is maintained in free time. In this sense, it is more beneficial for you if the maintainers spend more time coding than trolling about lisp.

Second, an IRC channel is not a place to look for support. Sourceforge offers you a bug reporting facility (https://sourceforge.net/tracker/?group_id=30035), a feature request database (https://sourceforge.net/tracker/?group_id=30035&atid=398056), a mailing list (http://news.gmane.org/gmane.lisp.ecl.general/)...

Response times are usually very fast, and only in the most difficult cases (it involves legacy code, requires some nonstandard architecture, etc), or when a release is on queue, does it takes longer.

In particular, in order to ease your life, bug reporting and feature requests are *now* possible for people without an account in Sourceforge. This is an experiment and I will try to keep it that way as long as spammers allow. Please make good use of it.

Finally, if you find something does not work the way you expect it, instead of spending a week on it and then complaining about it anywhere, just leave a message in any of those communication channels I mentioned above. It will be more productive for you and for the free software community in general.

Ah, and before I forget it: a good bug report for ECL needs at least the following information: 1) platform information and operating system, 2) configuration options, 3) if it failed at configuration time, the log, 4) if it failed at build time (i.e. during "make") a log of that, 5) if it fails with a standard (quicklisp library), steps to reproduce it, 6) if it fails with your code, ideally some sample to reproduce it.

Hinge -- v0.0.1:

Hinge, my Node.js inspired API for Common Lisp wrapping libev is finally to a state where I feel okay giving it a non-zero version.

I've decided now would be a good milestone because the HTTP server functions, and has a set of exported APIs that let you reply to the requests it generates.

The quick set of terrible benchmarks I performed on the "Hello World!" example show me a requests/second throughput of 1500-2500 depending on the mood of ab (and the state of the number of sockets available on localhost). I intend to run it through a more serious (and HTTP/1.1 supporting) tool in the future, but for now ab has convinced me that the decisions I've made so far have not been abysmal.

The examples directory contains a bunch of example programs, or in the earlier cases, code snippets demonstrating various levels of functionality.

In the near future, in addition to covering the issues in the README, adding additional polish, and any bugs or blatant performance issues that come up I would like at some point to add support for:

• HTTP Clients
• SSL Sockets and Servers
• SSL HTTP Servers and Clients
• Async wrappers around FS calls
• Websockets (RFC6455) support
• Mongrel2 ZeroMQ handler support (through m2cl)
• Mongrel2 ZeroMQ protocol-compatible server
• SPDY Support (Somewhere down the list, maybe)

Any feedback is welcome however way you want to deliver it to me. This initial version stands to serve as a basic starting point for the concepts I wanted to convey, the take on the core APIs I wanted to offer, and a proof of a series of related concepts and architectural decisions. It may (hopefully) be useful as a stable base for progress, rather than chasing an otherwise potentially unstable head.

One of the great things about Common Lisp is the variety of implementations and the scope of deployment platforms and performance characteristic trade-offs (run-time vs compile-time vs memory size) they encompass. This is also one of the things that complicates library development. Testing on any significant combination of implementations and platforms is not practical for most Free Software library authors.

cl-test-grid is a project that provides automated, distributed testing for libraries available via Quicklisp. You download cl-test-grid on your machine, tell it about which CL implementations you have available, it runs the test suites from many libraries in the current Quicklisp release, and sends the test results to a publicly available report as well as a public bug tracker. The report details test failures by Common Lisp implementations, platform/OS, and library version.

The report is a great resource for library authors and implementation maintainers. If your library is distributed via Quicklisp and isn't tested by cl-test-grid yet, Anton Vodonosov provides some tips for making your test suite cl-test-grid friendly. If you're looking to contribute to Free Lisp software, one of the best ways is to get cl-test-grid and run the tests. See if there are any failures in the test suites, and report the bugs to the library maintainers and help find a fix.

This isn't Madeira proper yet, but something small and useful on it's own, I hope: MADEIRA-PORT.

Main feature is :MADEIRA-PORT ASDF component class:

(defsystem :foo
  :defsystem-depends-on (:madeira-port)
  :serial t
  :components
  ((:file "package")
   (:module "ports"
    :components
    ((:madeira-port "sbcl" :when :sbcl)
     (:madeira-port "ccl" :when :ccl)
     (:madeira-port "ansi" :unless (:or :sbcl :ccl))))
   (:file "foo")))

The :WHEN and :UNLESS options support an extended feature syntax, which allows things such as:

(:find-package :swank)
(:find-function #:exit :sb-ext)

This extended feature syntax is also accessible by calling EXTEND-FEATURE-SYNTAX at the toplevel, after which regular #+ and #- readmacros will also understand it -- but unfortunately they will also lose any implementation specific extensions in the process.

Happy Hacking!

Seems ECL supports more codepages than iconv on certain platforms

;;; Testing ISO-2022-JP-1
iconv: conversion from `ISO-2022-JP-1' is not supported
Try `iconv --help' or `iconv --usage' for more information.
(From regressions in http://ecls.sourceforge.net/logs.html )

This is so in some not too old Linuxes, OpenBSD, NetBSD, etc.

Probably nobody cares about this, but there has been some activity in ECL in various fronts. I periodically report that to the ECL mailing list, but this seems to lead other people to think the project is silent or dead. From now on, expect a little bit more noise through the ECL news RSS.

The biggest change ECL will suffer with the upcoming release is the multithreading library. So far we were relying on the POSIX primitives for that (pthread_mutex_lock, condition variables, etc), but this has proven to be a nightmare to maintain. The primitives are not interruptible and they do not interact well with mp:process-interrupt, a feature unfortunately everybody expects in a Lisp implementation.

The upcoming library will be based on a combination of hardware-coded atomic operations (via libatomic-ops) with a new waiting queue that makes clever use of the operating system facilities. The result should be a library that is fair (different threads have equal opportunities), it recognizes fast paths (when a synchronization object is free it is quickly acquired), and in the worst case it is not CPU consuming (relies on sigsuspend or SleepEx to really sleep a waiting thread).

Everything is right now in git/CVS (http://ecls.sourceforge.net/download.html) and under heavy testing with new dedicated tests. Help is welcome from anybody who is interesting in ensuring stability.

Another side effect is that "make check" now works in ECL. I have abandoned the development of "ecl-tests" and instead the tests are now integral part of ECL. In order to save space, the ANSI test suite is not shipped with ECL but rather automatically downloaded thanks to ECL's new components: ecl-curl and ecl-minitar. The test suite also allows downloading of quicklisp and automated testing of selected libraries.

In the end this was the last missing ingredient to resucitate our compiler farm, which is now up and running again: http://ecls.sourceforge.net/logs.html

Expect a new release soon, and also a few more posts on other upcoming features.

Cheers,

Juanjo

P.S.: I would like to thank the ELS organizers and steering committee for a wonderful meeting in Zadar. Great location, amazing food, and even better discussions.

Another job opportunity at EyeCarePro.net

EyeCarePro.net (www.eyecarepro.net) is looking for another full time SBCL programmer to help manage and improve our main website builder application. No relocation required - North American working hours, 5+ years of SBCL experience (Hunchentoot / Debian) with strong communication skills. Contact DanielR@eyecarepro.net for more details.

In the latest CVS M-. became slightly more convenient.
It’s now able to find definitions of symbols whose names are prefixed with punctuation, for example, some lisps produces warnings

In FOO: Undeclared free variable X

And pressing M-. on FOO tried to find “FOO:” before, now it sees that there’s no symbol named “FOO:”, it tries to find “FOO”.

This is controlled by two new variables, which can be customized in ~/.swank.lisp

*find-definitions-right-trim*, default value “,:.”
*find-definitions-left-trim*, default value “#:”.

All those are arguments to cl:string-x-trim.

With the default value for left-trim it’s now possible to use M-. on
#: symbols in package definitions.

A most excellent day. I mistimed my morning, and had to miss breakfast in order to arrive in time to catch the ferry to Preko (a village on the island opposite Zadar) with a small but perfectly-formed band of intrepid explorers: Didier Verna, David Johnson-Davies, Alessio Stalla and Nils Bertschinger. Given the attractive description of the castle of St Michael as a "telecommunications station", with my employment, how could I not attempt the walk. And so we walked through the sunshine to the top of the mountain, to see what we could see (the other side of the mountain, it turns out, but also more islands, more sea, more sun...). Some fun discussions, about the more esoteric parts of various lisp dialects, machine learning, method-combination-based hacks, the nature of objects, and so on (quick riddle: what are the possible fixed points of type-of?) Then, as we were sitting down to yet another Southern European lunch, what should happen but an extremely localized rain shower?! How odd.

Then it was back to reality; it was time to face the fact that I was going to have to leave the sunny, sleepy Croatian coast to return to rainy, cold, difficult real life. (It has its compensations...) A very good conference; a big thank you to Franjo Pehar and Damir Kero, and the University of Zadar, for being such excellent hosts; it's been a great three days.

Nice article by Kent Pitman, mostly about his experiences with the Common Lisp standardization process: Common Lisp, the Untold Story.

A short article about an apparently successful digital livescan fingerprinting system.

Happy Protest Day.

Day two of ELS 2012 began with war stories from Ernst van Waning on his work as a consultant for SRI (his talk should of course not be confused with his employer's views; Pascal Costanza's opinions from yesterday were similarly disclaimed) on their AURA project. We had a confusing video-in-video demonstration, which is perhaps taking the conservative approach to the Demo Effect a little bit too far. A point that might be worth mentioning again is that Lisp is not immune to memory leaks; in the KM (warning: 1990s-compatible website) knowledge management software, many interned symbols get generated during the course of a query and do not get removed later. We've encountered this in SBCL in the past; for example, in the SBCL build itself, towards the end of the process, we compile and load PCL, then in a fresh bootstrap image just load the fasls before dumping the final image; this is so that many internal symbols created by reading the code don't end up in the final image. (Though something that has been suggested a few times is that packages should intern their symbols in a hash table which is weak: dear cleverweb, is there any way in which this can be detected? Think like pfdietz).

After coffee, and surprisingly packaged filled croissants, Marco Benelli gave us his experiences of using Scheme (specifically Gambit-C) in industrial automation. There were some interesting constraints – the approach had to support uClinux as well as more full-fat distributions, and... exotic architectures like sh. After that came Gunnar Völkel, talking about Algorithm Engineering with Clojure ("Algorithm Engineering" here seemed to mean the cycle of algorithm design, analysis, implementation and evaluation); the start of the talk was about implementing tracing and profiling, using a name-based registry before function definition to specify interceptions which wrap function implementation, which worried me because it seemed like a description of what should already exist (the lack of documented support for advice/fwrappers in SBCL notwithstanding). After that, we had a description of their team's Experiment Definition Language, used to generate code in an Experiment Setup Language, which then performs a whole experimental run (of the order of weeks) for various different parameters. I'm not convinced about the composability of the interception design; one issue is that since it overrides the defining forms, it is automatically incompatible with any similar extension (just as SERIES is incompatible with SCREAMER and Lexicons: each of them wants to own defun) – another is that, because the interceptions modify the source code, there's no sensible ordering: one of the example functions was both traced and profiled, which means that either the profiling code becomes traced (where the user is probably not interested in the execution of the otherwise-invisible profiling code) or the tracing code becomes profiled (which detracts from the utility of the profiling data).

One more long lunch break later, and we're into the afternoon iteration: Irène Durand talking about enumerators, and Alessio Stalla about do+, both dealing with ways of structuring iteration. Irène's taxonomy of enumerators might bear closer attention, while Alessio's "I don't hate loop" polemic against the use of a code walker in an iteration construct (iterate) seemed to be a totally reasonable point. There was some interesting probing of the limitations as well as the extensibility of the design – Pascal Costanza brought up the fact that loop allows e.g. if foo collect bar into quux else nconc baz into quux (forgive the attack of metasyntosis), while this variety of accumulation function into the same accumulator appears not possible in the do+ design.

Lightning talks: a virtual filesystem based on queries; return-with-dynamic-extent; HIV as recursive immune system process; pathnames (gah! pathnames!); high-performance network appliances; interoperability choices; and homoiconicity (Didier gives highly-engaging lightning talks: "musical notation is Tuning-complete"). Then announcements (a big thanks to Marco Antoniotti, Franjo Penar and his local team), dinner, bed.

SB-UNIX is an internal implementation package. If you use functionality provided by it, sooner or later your code will break, because SBCL changed its internals: It is subject to change without notice. When we stop using a function that used to live there, that function gets deleted. Things may also change names and interfaces.

CL-USER> (documentation (find-package :sb-unix) t)

"private: a wrapper layer for SBCL itself to use when talking
with an underlying Unix-y operating system.
This was a public package in CMU CL, but that was different.
CMU CL's UNIX package tried to provide a comprehensive,
stable Unix interface suitable for the end user.
This package only tries to implement what happens to be
needed by the current implementation of SBCL, and makes
no guarantees of interface stability."

Instead, use either SB-POSIX (which is the supported external API), or call the foreign functions directly. Alternatively, if you're using something from SB-UNIX that doesn't have a counterpart in SB-POSIX or elsewhere, put a feature request / bug report on Launchpad explaining what you need. Just saying "wanted: a supported equivalent of SB-UNIX:UNIX-FOO" is enough.

(The same holds more or less for all internal packages, of course, but SB-UNIX is the most common offender.)

I realize this is an imperfect world, and sometimes using an unsupported API is the best thing you can do, but please try to avoid this especially in libraries used by other people as well.

ELS 2012 Liveblogging! Well, I'm constrained to one diary entry a day, so maybe it's a bit of a stretch to claim that I've joined the socially network world, but baby steps...

First impression: Zadar is really quite pretty. Shiny white stone, clean, old buildings, seafront. My first impression is probably coloured by the fact that when I left from London, the weather was 4°C and pouring with rain – and I emerged from the plane into 25°C heat and a blue sky. I confess, I even had a little nap near the Sea Organ while waiting for the evening meeting and welcome reception; at that reception, let us just say that a good amount of Maraschino and another good amount of the local beer was consumed, both in good company. Also, it's asparagus season; yum.

This morning, after a generous welcome from Marco Antoniotti, this year's programme chair, Juanjo García Ripoll gave a very interesting overview of ECL and its history, and made some good points about its design philosophy. The key argument is probably that designing ECL for embeddability adds options, rather than being a limitation; he made a plausible case for those things which are currently lost compared with more traditional implementations – particularly, image saving – are reimplementable, at least up to a point. Juanjo also listed a number of good improvements in ECL since the last time: Unicode support, multithreading, improved CLOS and MOP support, and plenty of other things.

After a good long coffee break, we had the first paper sessions: first, a presentation of Climb (no website yet, apparently), an image processing toolkit developed by Laurent Senta with Didier Verna; some interesting stuff in there, even if the dreaded Demo Effect came along. A particularly neat-looking demo of a (prototype) visual environment for chaining processing tasks; performance is a bit more of a hot topic (read: not yet implemented), both in terms of parallelizing individual operations and (I think) in terms of compiling networks of processing tasks to minimize redundant computation. After that, Giovanni Anzani gave an autocad-based talk on calculating and visualizing optimal (for some value of "optimal"; sufficient for architecture, anyway) points of intersection of incommensurate measurements. Again, a pretty nifty demo, this time within AutoCAD using AutoLisp; somewhat surprisingly, it seems that there is no matrix manipulation library support within AutoLisp. (I think I need to read the paper for this work, to understand exactly what the problem the method presented is aiming to solve).

One lesson in Southern European lunchtimes later (even longer than academic lunchtimes!) we were into the second session, starting with Alessio Stalla talking about ABCL and its interoperability with Java. I got a shout-out, because in amongst the various integrations of ABCL with its JVM host was a note that the sequences in ABCL support the extensible sequence protocol that I proposed in 2007; the example given was of using instances of Java java.util.List class instances as Lisp sequences, directly. The demo effect struck again; instead of launching slime, a button in the modified Java web framework made the compiler enter an infinite compiling loop. Bad luck. (Demoing things is a particular nightmare, I know; the trick as far as I have managed to formalize it is to leave as little as possible to chance: this includes even typing, unless you're very confident: use short file or variable names, define key bindings or keyboard macros, or write scripts to do things for you.) Nils Bertschinger talked about probabilistic programming in Clojure: implementing Metropolis-Hastings sampling of program paths with given probabilities, and consequently allowing conditioning on some program choice points and Bayesian inference on the hidden parameters. It looks interesting, but the killer feature of Clojure (immutable data structures, for cheap undo) might also be the cause of a performance problem. Still, looks interesting. The demo worked.

Pascal Costanza rounded out the day's schedule with his discussion on reflection in Lisp and elsewhere, talking about fexprs, 3-lisp and macros through to metaobject protocols. Unfortunately, as a regular attendee at Lisp events, I've seen much of it before; it's still interesting, but maybe I need to get out more. To dinner!

A Boston Lisp Meeting will take place on Thursday, May 17th 2012 at 1800 at MIT 32-D463 (Star conference room). Zach Beane will speak about Quicklisp.

Additionally, we will have two Lightning Talks. James Knight will discuss his ideas on how Common Lisp hackers could better collaborate. François-René Rideau will present recent Common Lisp hacks including ASDF 2.21, λ-reader, inferior-shell, and more.

1 Zach Beane on Quicklisp

Quicklisp is a library manager for Common Lisp. It works with your existing Common Lisp implementation to download, install, and load any of over 700 libraries with a few simple commands. http://www.quicklisp.org/beta/

Zach Beane, also known as Xach, is a Common Lisp hacker known for the cool toys he builds and his "just do it" attitude.

2 Lightning Talks

At every meeting, before the main talk, there are two slots for strictly timed 5-minute "Lightning Talks" each followed by 2 minutes for questions and answers.

James Knight will discuss his ideas on what Common Lisp hackers could do to improve how they share software.

François-René Rideau will present recent Common Lisp hacks: xcvb, asdf, package-renaming, asdf-encodings, asdf-bundle, λ-reader, reader-interception, inferior-shell, rpm, fare-memoization, fare-utils.

3 Time and Location

The Lisp Meeting will take place on Thursday, May 17th 2012 at 1800 (6pm) at MIT 32-D463 (Star conference room).

The Star conference room, MIT 32-D463 on the fourth floor of the Ray and Maria Stata Center, 32 Vassar St, Cambridge MA 02139. NB: there are two sets of elevators, you want to use those on the south-western side, further away from Main St.

MIT map: http://whereis.mit.edu/?go=32

Google map: http://maps.google.com/maps?q=Stata+Center,+Vassar+Street,+Cambridge,+MA

Many thanks go to Professor Gerald J. Sussman for arranging for the room, and to MIT for welcoming us.

4 Dinner

We don't have any sponsors to offer us dinner, but we're big boys and can provide for ourselves. Before we start the conference, we'll organize a big pizza order, where those who want can chip in; after the conference is when we'll eat it.

5 More about the Meeting

We are resuming the Boston Lisp Meeting after a hyatus of over a year and a half.

We're always looking for more speakers. The call for speakers and all the other details are at: http://fare.livejournal.com/120393.html Volunteers to give Lightning Talks are also sought. http://fare.livejournal.com/143723.html

For more information, see our web site http://boston-lisp.org/ For posts related to the Boston Lisp meetings in general, follow this link: http://fare.livejournal.com/tag/boston-lisp-meeting or subscribe to our RSS feed: http://fare.livejournal.com/data/rss?tag=boston-lisp-meeting

Please forward this information to people you think would be interested. Please accept my apologies for your receiving this message multiple times. My apologies if this announce gets posted to a list where it shouldn't, or fails to get posted to a list where it should. Feedback welcome by private email reply to fare at tunes.org.

The master branch of CFFI has the cffi-libffi system, which permits passing and returning structures by value. As part of that addition, some of the syntax for structures has changed, even without loading that system. Before, "bare structure" references were permitted, and they were interpreted to mean pointers. Now, the syntax to specify a structure type is (:struct name) for the structure by value, and :pointer or (:pointer (:struct name)) for the structure by reference. The bare structure form is still accepted, with a style warning.

The function mem-aref returns an element of the array, an object; since we now can translate foreign structures, that includes structures. If the type is designated as (:struct name), that means a CL representation of the structure will be returned. By default (with no translate-from-foreign method defined), that is a plist. If you want a pointer instead, use the new function mem-aptr. As a special case, using the bare structure type specification, mem-aref still returns a pointer, but if you use the new form, you need to call mem-aptr to get the pointer instead of the object translated into CL.

(Rationale for the new function is here.)

If this isn't working as described, please post to cffi-devel.

I got interviewed a couple of weeks ago. (If you're reading this on Planet Lisp, you probably already know this). I had a quick update to one of the points I made, but failed to write it down anywhere and have since forgotten. So, instead, a somewhat delayed and probably more dull update. (What, more dull than the delayed response of someone to their own interview? Well, be ready to be amazed. But I can't help but feel that the additional information was that I had forgotten some whole class of Lisp programming that I actually do, which is a bit embarrassing in an interview in a series titled "Lisp Hackers"...).

Maybe the first thing was to dip my toes back into SBCL maintenance: some nice if minor fixes from me this cycle: one was a simultaneous bug fix and optimization to modular arithmetic, motivated by pfdietz's resurfacing and running of his random form tester, which inevitably revealed that we have been slack in the last five years or so (where does the time go?); the other was a fix to the powerpc implementation of ldb, which broke the build after the previous fix. All sorted out now, phew. (And there's lots of other stuff that's gone in this month, unlike the previous "month" which sort of rolled on for the best part of three months, so it's probably worth testing).

But onward, to my desire to learn a bit more about Emacs Lisp. I've used emacs for many a year – indeed, the interview reminded me that I learnt Lisp by being given a difficult problem to work on, instructions on how to start XEmacs, and time to read USENET – but have never considered myself a real Emacs User; my ~/.emacs is so tiny, I daren't show it to the world for fear that I will lose all my remaining Lisp Hacker credibility. Acting with the view that the best way to learn is to do, shortly after starting to use EMMS as a media player, I've implemented support for DISCNUMBER metadata (this matters if, like me, you have a large number of multiple-disc sets). I've also revived (again) SWANKR, putting it up on github also, since I have started getting patches; I look forward to exploring this "social coding" idea. And I have also written a hacky but just-about-usable interface to the BBC iPlayer (using the excellent but fairly user-hostile get_iplayer utility) – particularly pleasing to my family now that digital switchover has reached London and I no longer possess equipment capable of receiving the UK television signal.

I now return to the Teclo vortex. But I am going to Zadar for the European Lisp Symposium; I hope to see some old and new faces there!

zsort is a library that I started working on as a simple hobby project. More or less around the same time I decided to check which algorithms the different Common Lisp implementations use. It is now part of Quicklisp so it can be easily used (thanks Zack!).

The main goal of zsort is to be a collection of portable sorting algorithms. If they can be fast, even better. Common lisp provides the sort and stable-sort functions but these can have different algorithms implemented according to each implementation, which can make an application unportable if you rely on a specific type of sorting. Also, the standard functions might not be the best for a certain situation and as such you might need a specialized sort. Even if for most situations the standard functions are more than enough, the zsort library could be a useful complement.

Right now the implemented algorithms are: insertion sort, quicksort, randomized quicksort, merge sort, heapsort and counting sort. The plan is to add more algorithms, for example, bucket sort and timsort. However, the main thing on the todo list is adding the possibility of external sorting (to handle large amounts of data) and parallel versions of some sorting algorithms. I am considering using lparallel for this but I am still undecided.

There is still a lot of work to be done, but I think the library as it is can already be a little useful. And of course, all kind of suggestions and improvements are welcome!

Filed under: Programming Tagged: Common Lisp, library, Lisp, Sorting Algorithms, zsort

European Lisp Symposium 2012, Zadar, Croatia, April 29th - May 1st, 2012

Call for Participation.

Web site: european-lisp-symposium.org

The purpose of the European Lisp Symposium is to provide a forum for the discussion and dissemination of all aspects of design, implementation and application of any of the Lisp and Lisp-inspired dialects, including Common Lisp, Scheme, Emacs Lisp, AutoLisp, ISLISP, Dylan, Clojure, ACL2, ECMAScript, Racket, SKILL, and so on. We encourage everyone interested in Lisp to participate.

The main theme of the 2012 European Lisp Conference is "Interoperability: Systems, Libraries, Workflows". Lisp based and functional-languages based systems have grown a variety of solutions to become more and more integrated with the wider world of Information and Communication Technologies in current use. There are several dimensions to the scope of the solutions proposed, ranging from "embedding" of interpreters in C-based systems, to the development of abstractions levels that facilitate the expression of complex context dependent tasks, to the construction of exchange formats handling libraries, to the construction of theorem-provers for the "Semantic Web". The European Lisp Symposium 2012 solicits the submission of papers with this specific theme in mind, alongside the more traditional tracks which have appeared in the past editions.

Registration:

Until April 15th the registration fee will be reduced. Use the form on european-lisp-symposium.org to register.

Preliminary Program schedule:

Sunday April 29.

• 11:00 Sea & Sun & Coffee (meeting place: The Sea organ & Greeting to the Sun)
• 19:30 - 21:30 Welcome Cocktail at the Garden Bar

Monday April 30.

• 8:30 - 09:00 Registration
• 09:30 - 10:00 Welcome
• 10:00 - 11:00 Juan Jose Garcia-Ripoll, Instituto de Física Fundamental, CSIC, Spain
• 11:00 - 11:30 Coffee Break
• 11:30 - 13:00 Session 1
  • 11:30 - 12:15 Laurent Senta, Christopher Chedeau and Didier Verna. Generic Image Processing with Climb
  • 12:15 - 13:00 Giovanni Anzani, An iterative method to solve overdetermined systems of nonlinear equations applied to the restitution of planimetric measurements
• 13:00 - 15:00 Lunch Break
• 15:00 - 16:30 Session 2
  • 15:00 - 15:45 Alessio Stalla. Java interop with ABCL, a practical example
  • 15:45 - 16:30 Nils Bertschinger. Embedded probabilistic programming in Clojure
• 16:30 - 17:30 Pascal Costanza, ExaScience Lab, Intel, Belgium
• 17:30 - 18:00 Lightening Talk (1)
• 18:00 - 20:00 Buffet at the University of Zadar

Tuesday May 1.

• 10:00 - 11:00 Ernst van Waning, Infometrics, Netherlands
• 11:00 - 11:30 Coffee Break
• 11:30 - 13:00 Session 3
  • 11:30 - 12:15 Marco Benelli. Scheme in Industrial Automation
  • 12:15 - 13:00 Gunnar Völkel, Johann M. Kraus and Hans A. Kestler Algorithm Engineering with Clojure
• 13:00 - 15:00 Lunch Break
• 15:00 - 16:30 Session 4
  • 15:00 - 15:45 Irène Anne Durand. Object enumeration
  • 15:45 - 16:30 Alessio Stalla. Doplus, the high-level, Lispy, extensible iteration construct
• 16:30 - 17:00 Lightening Talk (2)
• 17:00 - 18:00 Announcements & Wrap-up

Venue:

Zadar is one of the enchanting cities on the Adriatic coast rich in history. It still preserves a very old network of narrow and charming city streets, as well as a Roman forum dating back to the first century AD. In addition, Zadar region encompasses many natural beauties, most prominent among them is the Kornati National Park, the most unusual and indented set of small islands in the Mediterranean (89 islands, islets and reefs) located only 15 nautical miles south of Zadar - a visit to Kornati is planned as a part of the conference.

Accommodation:

Participants are responsible for their own accommodation arrangements. Reservations should be made directly to the hotels.

When making reservations in hotel Hotel Kolovare, please indicate that you are with ELS 2012 / University of Zadar and you will get a discount. Due to the popularity of the location in spring, an early reservations is recommended.

• Hotel Kolovare * * * *: www.hotel-kolovare.com, +385 23 211 017, +385 203 200

Other accommodation options:

• Hotel Bastion * * * *: www.hotel-bastion.hr, +385 23 / 494 950, info [at] hotel-bastion.hr
• Falkenstiner Hotels & resorts - (Funimation)* * *: www.falkensteiner.com, +385 (0)23 206 400, +385 (0)23 332 065, sales.borik [at] falkensteiner.com
• Hotel Mediteran * * *: www.hotelmediteran-zd.hr, info [at] hotelmediteran-zd.hr
• Hotel Porto * * *: www.hotel-porto.hr, +385-(0)23-292300, hotel.porto [at] zd.t-com.hr
• Hotel Villa Nico * * *: www.hotel-villanico.com, +385 23 331198, +385 23 331960, nico.moric [at] villanico.t-com.hr
• Hotel President * * * *: www.hotel-president.hr, +385 23 337 500+385 (23) 333 696 , info [at] hotel-president.hr
• Hostel Zadar: +385 331-145

Pascal Costanza is a researcher, and an active Common Lisp programmer and community enthusiast:

• he's the maintainer of Closer to Mop library, that provides a common facade to the MOP implementation in different Lisps, and is the basis of some of his more advanced libraries like: ContextL and FilteredFunctions;


• the originator of Common Lisp Document Repository (CDR) project, that collects proposals for improving the language (a la JCP for Java or PEP for Python);


• and the author of a Highly Opinionated Guide to Lisp, which can serve as introductory text for those, who come from other languages. (It was quite a useful text for me, when I started studying Lisp.)

In the interview Pascal shares a lot if insight into his main topic of interest — programming language design — grounded in his experience with Lisp, Java, C++ and other languages.

Tell us something interesting about yourself.

I share a birthday with Sylvester Stallone and George W. Bush. I have been a DJ for goth and industrial music in the Bonn/Cologne area in Germany in the past. I once played a gay Roman emperor in comedic theatre play. I played a few live shows with a band called "Donner über Bonn" ("Thunder over Bonn"). My first rock concert I ever attended was Propaganda in Cologne in 1985. The first programming language I really liked was Oberon. I often try to hide pop culture references in my scientific work, and I wonder if anybody ever notices. My first 7" single was "Major Tom" by Peter Schilling, my first 12" single was "IOU" by Freeez, my first vinyl album was "Die Mensch-Maschine" by Kraftwerk, and my first CD album was "Slave to the Rhythm" by Grace Jones. I don't remember what my first CD single was.

What's your job? Tell us about your company.

I currently work for Intel, a company whose primary focus is on producing CPUs, but that also does business in a lot of other hardware and software areas. (Unfortunately, Intel's legal department requires me to mention that the views expressed in this interview are my own, and not those of my employer.)

I work in a project that focuses on exascale computing, that is, high-performance computers with millions of cores that will be on the market by the end of the decade, if everything goes well. I am particularly involved in developing a scheduler for parallel programs that can survive hardware failures, which due to the enormous scale of such machines cannot be solved by hardware alone anymore, but also need to be dealt with at the software level. The scheduler is based on Charlotte Herzeel's PhD thesis, and you can find more information about it in a paper about her work and at http://www.exascience.com/cobra/.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

At Intel, I do all software prototyping in Lisp. The scheduler I mentioned above is completely developed and tested in Lisp, before we port it to C++, so that other people in the same project and outside can use it as well. It didn't require a major effort to convince anybody to do this in Lisp. It is actually quite common in the high-performance computing world that solutions are first prototyped in a more dynamic and flexible language, before they are ported to what is considered a "production" language. Other languages that are used in our project are, for example, MATLAB, Python and Lua. (Convincing people to use Lisp beyond prototyping would probably be much harder, though.)

The implementation we use for prototyping is LispWorks, which is really excellent. It provides a really complete, well-designed and efficient API for parallel programming, which turns LispWorks into one of the best systems for parallel programming of any language, not just in the Lisp world. The only other system that is more complete that I am aware of is Intel's Threading Building Blocks for C++.

What brought you to Lisp? What holds you?

I have participated in one of the first Feyerabend workshops, organized by Richard Gabriel, one of the main drivers behind the original Common Lisp effort. I have also read his book Patterns of Software around that time. Later we had a small discussion in the patterns discussion mailing list. He tried to promote Lisp as a language that has the "quality without a name", and I made some cursory remarks about Lisp's unnecessarily complicated syntax, just like anybody else who doesn't get it yet.

To me, the most important comment he made in that discussion was: "True, only the creatively intelligent can prosper in the Lisp world." The arrogance I perceived in that comment annoyed me so much that it made me want to learn Lisp seriously, just to prove him wrong and show him that Lisp is not as great as he thought it is. As they say, the rest is history.

I actually dabbled a little bit in Lisp much earlier, trying out a dialect called XLisp on an Atari XL computer at the end of the 80's. Unfortunately, it took too long to start up XLisp, and there was not enough RAM left to do anything interesting beyond toy examples, plus I was probably not smart enough yet to really get it. I was just generally curious about programming languages. For example, I also remember trying out some Prolog dialect on my Atari XL.

In the end, Lisp won me over because it turns out that it is the mother of all languages. You can bend it and turn it into whatever language you want, from the most flexible and reflective interpreted scripting language to the most efficient and static compiled production system. For example, the scheduler mentioned above easily gets in the range of equivalent C/C++-based schedulers (like Cilk+, TBB, or OpenMP, for example), typically only a factor of 1.5 away for typical benchmarks, sometimes even better. On the other hand, ContextL uses the reflective features of the CLOS Metaobject Protocol to bend the generic function dispatch in really extreme ways. I am not aware of any other programming language that covers such a broad spectrum of potential uses.

What's the most exciting use of Lisp you had?

When I decided to make a serious attempt at learning Common Lisp, I was looking for a project that would be large enough to prove to myself that it is actually possible to use it for serious projects, but that would also be manageable in a reasonable amount of time. At that time, I was intimately familiar with the Java Virtual Machine architecture, because I had developed compilers for Java language extensions as part of my Diploma and PhD theses. So I decided to implement a Java Virtual Machine in Common Lisp - under normal circumstances, I wouldn't have dared to do this, because this is quite a complex undertaking, but I had read in several places that Lisp would be suitable for projects that you would normally not dare to do otherwise, so I thought I would give it a try. Over the course of 8 weeks, with something like 2 hours per day, or so (because I was still doing other stuff during the day), I was able to get a first prototype that would execute a simple "Hello, World!" program. On top of that, it was a portable (!) just-in-time compiler: It loaded the bytecode from a classfile, translated it into s-expressions that resemble the bytecodes, and then just called Common Lisp's compile function to compile those s-expressions, relying on macro and function definitions for realizing these "bytecodes as s-expressions." I was really impressed that this was all so easy to do.

The real moment of revelation was this: to make sure to reuse as many of the built-in Common Lisp features as possible, I actually translated Java classes into CLOS classes, and Java methods into CLOS methods. Java's super calls posed a problem, because it was not straightforward to handle super calls with plain call-next-method calls. Then I discovered user-defined method combinations, which seemed like the right way to solve this issue, but I was still stuck for a while. Until I discovered that moving a backquote and a corresponding unquote around actually finally fixed everything. That was a true Eureka moment: In every other programming language that I am aware of, all the problems I encountered until that stage would have required a dozen redesigns, and several attempts to start the implementation completely from scratch, until I would have found the right way to get everything in the right places. But Common Lisp is so flexible that at every stage in your development, you can tweak and twist things left and right, but in the end you still get a convincing, clean, and efficient design. As far as I can tell, this is not possible in any other language (not even Scheme).

What you dislike the most about Lisp?

There is not much to dislike about Lisp itself. There are some technical details here and there, some minor inconsistencies, but nothing that cannot be fixed in easy and straightforward ways. From a purely conceptual point of view, Common Lisp is one, if not the most complete and best integrated programming language that covers a lot of ground. Some rough edges are just to be expected, because nothing is ever perfect.

What concerns me a lot more is that there is too much unwarranted arrogance in the Lisp community. I don't know exactly where this comes from, but some Lispers seem to believe, just because they understand some Lisp concepts, that they are much smarter than anybody else in the universe. What they are forgetting is that computer science as a discipline is very, very young. In one or two hundred years from now, Lisp concepts will be common knowledge, just like elementary algebra. There is no reason to be arrogant just because you know that the earth is round, even if most other people still believe that it is flat.

Describe your workflow, give some productivity tips to fellow programmers.

I strongly believe that the one thing that made me most productive as a programmer is my interest in doing some form of art. I used to spend a lot of time making my own music, both by myself with synthesizers and computers, as well as in bands. I also was an actor in an amateur theater group. Art gives you a sense of making parts (notes, chords, melodies, rhythms, or acts, characters, plot lines) relate to each other and form a coherent whole. It also makes you aware that there is an inner view on a piece of music or a play, as seen by the artist, but also an outer view, as seen or heard by an audience, and if you want to make good art, you need to be able to build a bridge between those two parts.

Programming is exactly the same: you need to make parts (functions, data structures, algorithms) relate to each other, and you need to bridge the inner view (as seen by the designer and implementer) and the outer view (as seen by the user of a library or the end user of the final software).

The important aspect here is that you need to be able to change perspectives a lot, and shift between the local, detailed view, the global, architectural view, and the many different levels of a layered design. This is especially important when it comes to designs that incorporate meta-programming techniques and reflective approaches, but also already for simpler designs.

Like in art, the concrete workflow and the concrete tools that work best vary a lot for different people. It's also a good idea to just play around with ideas, expecting that most of them will turn out useless and need to be thrown away. Artists do this all the time. Artificial, seemingly nonsensical rules and restrictions can be especially enlightening (use only effect-free functions; use only functions that receive exactly one argument, not more, not less; make every function pass around an additional environment; use only classes with exactly two slots; etc., etc.), and then try to build larger programs strictly following such rules - this will make your mind a lot more flexible and train you to see new potential solutions that you wouldn't see otherwise.

(I actually believe that this is what makes fans of static typing so excited: Static type systems always impose some artificial restrictions on your programs, and enforce them to the extent that programs that violate these rules are rejected. If you then program in such a statically typed programming language, you will indeed have some interesting insights and see new solutions that you would otherwise miss. However, the category error that fans of static typing often seem to make is that they ascribe the results to the static type system, and therefore usually get stuck with one particular set or kinds of rules.)

Apart from that, I believe that LispWorks is a really good development environment.

Among software projects you've participated in what's your favorite?

I don't know how to answer that. At any point in time, I'm always most excited by the one I'm currently working on. So far, I am quite proud of ContextL and the ClassFilters project for Java, because they are or were both used in one way or the other in "real" applications. Closer to MOP is a favorite project of mine, because it makes me feel like I can give something back to the Lisp community from which I otherwise benefit so much. But this doesn't mean I dislike anything else I have done in the past.

One of your papers, "Reflection for the Masses", researches the ideas behind 3-Lisp, "a procedurally reflective dialect of LISP which uses an infinite tower of interpreters". Can you summarize them here?

That paper was actually mostly the work of Charlotte Herzeel, and I was only her sounding board for detailing the ideas in that paper. Reflection is one of the essential concepts that was born out of Lisp. Every program is about something, for example a financial application is about bank accounts, money and interest rates, and a shopping application is about shopping items, shopping carts and payment methods. A program can also be about programs, which turns it into a meta-program. Lisp macros are meta-programs, because they transform pieces of code in the form of s-expressions into other pieces of code. C++ templates are also meta-programs in the same sense. Some meta-programs are about "themselves," and thus become reflective programs.

3-Lisp is "procedurally reflective" in two senses: On the one hand, it allows you to inspect and change the body of procedures (functions). Common Lisp, for example, also gives you that, in the form of function-lambda-expression and compile/eval, among others. On the other hand, 3-Lisp also allows you to inspect and alter the control flow. Scheme, for example, gives you that in the form of call/cc, but in 3-Lisp this is actually part of the eval interface. 3-Lisp goes further than Common Lisp and Scheme combined, in that it provides not only first-class access to function bodies and continuations, but also to lexical environments, always both with facilities to inspect and modify them, and provides a clean and integrated interface to all of these features. Unfortunately, because 3-Lisp goes that far, it cannot be fully compiled but always needs to be able to resort to interpretation, if necessary.

The reason for the requirement to always have an interpreter around is because the eval interface in 3-Lisp is so flexible that you can run programs inside an eval invocation that in turn can inspect and change (!) the environment in which eval runs, and can then invoke further evals in those changed environments, to arbitrary levels of recursive invocations of eval. These recursive invocations of eval build what is called a reflective tower, where every level in the tower is conceptually an interpreter being executed by an interpreter one level up in the tower of interpreters. The amazing thing about 3-Lisp is that an implementation of 3-Lisp can actually collapse the tower into one level of interpretation, and arrange that one interpreter in such a way that the several different levels of interpretations are only simulated, so the tower is actually just an "illusion" created by the 3-Lisp implementation.

This may all sound quite esoteric, but is actually practically relevant, because there is strong evidence that all meta-programming approaches eventually need such towers, and some ad-hoc way to collapse the towers. For example, you can find the tower in Racket's and R6RS's macro systems, where they are explicitly mentioned; in Common Lisp's macros, where eval-when is used to control which part of the tower sees which definitions; in the CLOS Metaobject Protocol, where the generic function dispatch can be influenced by other generic functions, which can in turn be modified by some meta-classes at a higher level; in the template metaprogramming system of C++, where "concepts" were devised for C++11 (and rejected) to introduce a type system for the template interpreter; and so on, and so on. If you understand the concept of a reflective tower better, you can also better understand what is behind these other meta-programming approaches, and how some of their sometimes confusing semantic difficulties can be resolved.

If you had all the time in the world for a Lisp project, what would it be?

I have some ideas how to design reflection differently for a Lisp dialect, which I believe has not been tried before. If I had all the time in the world, I would try to do that. I also have many other ideas, so I'm not sure if I would be able to stick to a single one.

Anything else I forgot to ask?

I think one of the most underrated and most underused Lisp dialects is ISLISP. I think people should take a much closer look at it, and should consider to use it more often. Especially, it would be an excellent basis for a good teaching language.

Discussion on HackerNews

CFFI now has (on the master branch, but not yet in quicklisp) the system cffi-libffi, which allows calling and returning structures by value. For example, in the GSL library, complex numbers and functions are defined passing and returning the complex structures by value, so we can define

    (cffi:defcstruct (complex-double-c :class complex-double-type)
      (dat :double :count 2))

    (defmethod cffi:translate-into-foreign-memory ((value complex) (type complex-double-type) p)
      (cffi:with-foreign-slots ((dat) p (:struct complex-double-c))
        (setf (cffi:mem-aref dat :double 0) (realpart value)
    	  (cffi:mem-aref dat :double 1) (imagpart value))))

    (defmethod cffi:translate-from-foreign (p (type complex-double-type))
      (cffi:with-foreign-slots ((dat) p (:struct complex-double-c))
        (complex (cffi:mem-aref dat :double 0)
    	     (cffi:mem-aref dat :double 1))))

    (cffi:define-foreign-library libgslcblas
      (:unix (:or "libgslcblas.so.0" "libgslcblas.so"))
      (t (:default "libgslcblas")))

    (cffi:define-foreign-library libgsl
      (:unix (:or "libgsl.so.0" "libgsl.so"))
      (t (:default "libgsl")))

    (cffi:use-foreign-library libgslcblas)
    (cffi:use-foreign-library libgsl)

    (defun complex-log (c)
       (cffi:foreign-funcall "gsl_complex_log"
        (:struct complex-double-c) c (:struct complex-double-c)))

and then find

   (complex-log #C(1.0d0 2.0d0))
   #C(0.8047189562170501d0 1.1071487177940904d0)

Note that there are changes that affect CFFI users even if they don't load cffi-libffi; using bare structure names foo insead of (:struct foo) or :pointer or (:pointer (:struct foo)) will trigger a style warning, and there is a new function mem-aptr that does for structure arrays what mem-aref does when the bare structure names are used. Please try it and report issues on cffi-devel. Thank you.

Note: the old system FSBV, which did what cffi-libffi does but in a clumsier way, is now obsolete and will not work with this version of CFFI.

When *print-escape* is true, symbols are normally printed with package prefixes only if the current package *package* is not the symbol’s home package. It’s easy to make sure that a symbol is always printed with a package prefix, e.g. for debugging.

When the current package is the keyword package, non-keyword symbols are printed with package prefixes, and keywords are printed with their normal colon prefix. For example, in SBCL:

(in-package #:cl-user)
(let ((*package* (find-package "KEYWORD")))
  (prin1-to-string '(car stream :car quit)))

=> "(COMMON-LISP:CAR COMMON-LISP:STREAM :CAR SB-EXT:QUIT)"

For full details, see 22.1.3.3.1, Package Prefixes for Symbols.