Planet Lisp

Nikodemus Siivola: Neat TYPEP Trick

Tue, 15 May 2012 17:19:49 GMT

How do you test if an object is a cons that has the desired symbol in the car?

(typep x '(cons (eql :foo)))

Sure,

(and (consp x) (eq :foo (car x)))

is essentially just as short...

I still find cons types neat, even if they're a nightmare when it comes to type derivation, but that's a different matter. Some nightmares aren't all bad.

Jochen Schmidt: CLOS Class Naming

Tue, 15 May 2012 09:59:00 GMT

The classes of the Common Lisp spec, the MOP and big CLOS projects like CLIM use some naming conventions which may not make sense at first. I'm speaking of the STANDARD- and BASIC- prefixes. What is the idea behind them?

Protocol Oriented Programming

In CLOS one defines "protocols" using classes and generic functions. Remember: Methods do belong to generic functions and not to classes. There are no dedicated means to define protocols - it is idiomatic. A common practice is to define "protocol classes" which often do not have any slots at all. They serve as superclasses to all classes which implement the protocol.

Basics and Standards

In the set of protocol implementation classes there may be one, which contains a minimal complete implementation and/or there might be one which is something like the default implementation. Assuming a protocol for FOOBAR defined through the protocol-class FOOBAR the minimal implementation could be called BASIC-FOOBAR and the default one STANDARD-FOOBAR. The BASIC-** one if often called FUNDAMENTAL-** too.

Extensibility

It is often a good idea to implement STANDARD-** by inheriting the BASIC-** implementation, overriding the necessary parts. A protocol user who wants to implement his own implementation can decide on which level he wants to get in: He could inherit from STANDARD-** if his implementations is only a small variant. He may inherit from BASIC-** if the implementation is more complicated and perhaps incompatible to what STANDARD-** does. If even BASIC-** is to strict, he could try to implement the protocol class completely on its own - but this is then more like implementing interfaces in classical object oriented languages, where you cannot inherit behaviour at all. You have to provide methods for any necessary generic function the protocol is made of. This is the most work and implementing a protocol is often more than just following the interfaces.

Nikodemus Siivola: Trolled

Tue, 15 May 2012 05:36:46 GMT

(set-macro-character #\{ (lambda (s c) (read-delimited-list #\} s t)) nil)
(set-macro-character #\} (get-macro-character #\)) nil)

EDIT: I'm not saying Didier is wrong. He isn't. While readmacros are extremely flexible, the system has some rather obvious shortcomings.

I'm just pointing out that making {foo} read as list is trivial in case someone walked away from his post with the impression that it cannot be done.

If there's something I disagree with, it is his characterization of readmacros as "shaky". Used to do things they can do, in the manner they're designed to be used, they're perfectly robust. Just not as nice or convenient as I would like them to be.

The really interesting bit is his observation that READ-DELIMITED-LIST doesn't support dotted lists, which I hadn't realized before. It isn't specified to, but it would be nice if it did...

Slava Shirokov: CLOS FSM with MOP Sauce

Tue, 15 May 2012 05:33:00 GMT

Common Lisp in the desire to be as cool as possible includes in its specification the Common Lisp Object System, or CLOS, which itself can be introspected and altered in great detail using the MetaObject Protocol, or MOP, as described in The Art of the Metaobject Protocol. Unfortunately, MOP didn’t make it into the ANSI standard, but most implementations include MOP as it is described in the book, and a compatibility package :closer-mop (available in Quicklisp) makes using the symbols described seamless between implementations.

One of the features of MOP is the ability to make an instance of a CLOS class funcallable, that is allow a class instance to be a valid first argument to funcall. This behavior in a lot of ways can resemble the traditional method model from other languages, but that’s not how I intend use it here.

The Idea

I’m going to show and tell an implementation of a generic finite state machine that uses the MOP concepts of funcallable-standard-object and funcallable-standard-class to marshal the flow of incoming events to the machine and the concept of CLOS generic method dispatch to handle the execution of transition handlers for any given state of the machine.

These features will allow us to build a structure that lets us focus solely on the problem at hand and defer features like event data binding, state-dependent method selection and unexpected state handling entirely to the language without pushing the boundaries of any specific feature.

The Design

The design we’re going for is such that we can define a class with a state slot that will hold :keyword name of a state. We’ll make instances of this class funcallable so that when we make an instance we will be able to simply (funcall fsm-instance fsm-evemt) repeatedly and have the machine dispatch to the correct event, perform any logic, and transition to the next state based on the input.

We would be able to query the state of the machine with (state fsm-instance) and receive a keyword, and we should be able to drive events into the machine until we’re in a desired or unexpected state. Any attempt to feed the machine an event while the machine is in an invalid state should result in an error.

Setup

Dependencies in Common Lisp these days should come from Quicklisp. Install it now before even reading the next paragraph, and during the installation phase ask it to write itself to your lisp configuration’s init file.

The code samples below, as should any code using MOP intending to run more than once, uses the :closer-mop package. Fetch it from Quicklisp before attempting these examples by evaluating:

(ql:quickload :closer-mop)

At the REPL before attempting to evaluate any example code.

Implementation

We’ll first declare the base of our state machine. We’ll call the base standard-state-machine to follow the convention CL seems to have of sticking the standard- prefix to base classes.

(defclass standard-state-machine (c2mop:funcallable-standard-object)
  ((state :initform :initial :initarg :state
          :accessor state)
   (last-event :initform (get-internal-real-time)
               :accessor last-event))

  (:metaclass c2mop:funcallable-standard-class))

This grants us the majority of our desired features from the design. The resulting class holds a :keyword state name in a state slot, and is declared to be funcallable by the inclusion of the funcallable-standard-object base and the funcallable-standard-class metaclass for the class itself.

When events fire through this state machine we will need to invoke a method that we can use to dispatch to various handlers in the machine depending on the state we happen to be in. We can use the CLOS generic method feature called eql specializers to distinguish which method should be invoked based on the name of the state a machine happens to be in. EQL specializers are just like type specializers for method parameters, but instead of the comparison of the parameter to the specialization happening based on the type of the parameter, it happens as if an (eql parameter-value parameter-specialization) test is performed to determine if a given method is applicable to an invocation.

With this in mind we can design a generic method that is aware of three things as parameters that we can choose to specialize: The current state machine, the state of the current state machine, and the event that was received by the machine.

(defgeneric standard-state-machine-event (machine state event))

We won’t define any such methods now, because right now we have no state machine to model, and as such have no state handing methods to define.

To actually invoke a function when the funcallable instance is called as a function a lambda must be bound to be the funcallable instance function of a given instance of a class. Since there is no mandate for an inherent this pointer in CL, the best place to bind this function is often in one of the specializations of initialize-instance and close over the current instance in the process. The attachment of a function to an instance is done with the MOP function set-funcallable-instance-function, which places no special requirements on the args list of the attached function.

We’ll bind our driver to the state machine instance in the :before specializer of its initialize-instance method, and use an args list that expects only the event that is being delivered to the state machine:

(defmethod initialize-instance :before ((machine standard-state-machine) &key)
  (c2mop:set-funcallable-instance-function
   machine
   #'(lambda (event)
       (multiple-value-bind (next-state recur-p)
           (standard-state-machine-event machine (state machine) event)

         (setf (last-event machine) (get-internal-real-time)
               (state machine) (or next-state (state machine)))

         (if recur-p
             (funcall machine event)
             (values machine (state machine)))))))

Now when we have instance of the state machine invoked with (funcall fsm-instance event), the inner lambda expression will get passed the event and compute a return. This function despite having some length does very little.

First, it dispatches the event to our previous standard-state-machine-event method using the instance of the state machine we closed over as the first parameter, the state of the state machine as the second parameter and the event that the machine was called with.

It expects a multi-value return, but if the method only returns one value, no error is raised and the second value, recur-p is left bound to nil. Its meaning will become abundant in a moment.

Once the state handler is called the new state returned by the handler is stored as the current state of the state machine, and if a second value was returned, the machine is invoked again with the same event it had just received before returning from the handler the value of the current machine instance for chaining and the value of the current state.

The second, recur-p value is a hook that can be used from a state handler to force the machine to retry handling the event in a new state. This is useful when the same event should loop through the machine a second time before the transition is complete, and the optional second value return from a state handler allows that without additional external plumbing.

Functionally, the implementation of the state machine is complete. It can now cycle through a graph of events, provided that those events are defined as specialized methods. But, in its current state, the API leaves very much to be desired.

The Polish

As the state machine stands now, to implement one that for all inputs remains in the default :initial state, one would have to be defined explicitly by specializing the standard-state-machine event.

(defmethod standard-state-machine-event
            ((fsm standard-state-machine)
             (state (eql :initial))
             event)
  nil)

The trailing nil makes the return value explicit, due to the length of the specializations. This makes the structure of the dispatch clear, and the nil return value could be traced through the driver above to determine that this machine would do absolutely nothing given any kind of input. Remaining eternally in the :initial state.

To make the task of defining states for a given type of state machine we can write a simple macro to write the above form for us. After all, the majority of it is filler.

(defmacro defstate (machine-type state-name (machine-sym event-sym) &body body)
  `(defmethod standard-state-machine-event
       ((,machine-sym ,machine-type) (state (eql ,state-name)) ,event-sym)
     ,@body))

Using the macro above we can now declare an identical looping :initial state for the standard-state-machine as follows:

(defstate standard-state-machine :initial (fsm event)
  nil)

Which results in identical code to the one we wrote previously. The symbols given in the argument list of the defstate forms are bound to the machine evaluating the event and the event that is being sent through the machine. The remaining forms will be evaluated just as in any other defun/defmethod with the return value treated as either next state for the machine to enter, nil to stay in the same state, or a multivalue return with one of the previous and a non-nil second value to indicate the event should be fired through the machine a second time before the final state is stored.

To create subclasses of the standard state machine, we again run into a similar situation as before with boilerplate requirements. Each subclass of standard-state-machine must not only include the standard-state-machine class in the list of parents, it must also include funcallable-standard-class as its metaclass, as in:

(defclass my-fsm (standard-state-machine)
  ()
  (:metaclass c2mop:standard-funcallable-class))

This peculiarity could be documented a thousand times in a thousand places, but that will make it no less ugly or difficult to remember. It would be much easier to provide a familiar tasting API to consumers that results in the same code. For example, using a structure like this to generate code equivalent to the above

(deffsm my-fsm ()
  ())

This leaves the consumer free to do whatever their heart desires with CLOS without disturbing the requirements of our function. Such a construct can be stated simply as another tiny macro.

(defmacro deffsm (name parents slots &rest options)
  `(defclass ,name ,(append (list 'standard-state-machine) parents)
     ,slots
     (:metaclass c2mop:funcallable-standard-class)
     ,@options))

The Result

If we were to put all of that code together in one place with a bunch of documentation strings for the methods, classes and slots in a slightly more condensed form than this article, it might look something like this. Which is actually a minimal extraction of the state machine that drives the HTTP parser in Hinge. If this article gets enough interest, I will extract standard-state-machine into a QuickLisp compatible package for even easier reuse.

The Demo

Let’s construct an FSM for determining if a sequence of characters contains the string “Hi”, but before the character “!” appears. Once a string contains “Hi” it can contain “!” characters again.

We’ll model this as a machine of three states, with events being single characters of input. The states will be :initial, :want-i and :done.

In the :initial state, we’ll accept any input without transitioning except for “!” which will cause the machine to transition into an error state so that no more input can pass, and “H” will cause the machine to transition to the :want-i state.

Any event other than “i” in the :want-i state will transition back to :initial and the event “i” will cause the machine to enter the state :done, which will be a no-op looping state that allows input to pass through uninspected. The implementation for such a machine would look like this, assuming we have previously defined the :fsm package.

(fsm:deffsm hi-fsm ()
  ())

(fsm:defstate hi-fsm :initial (fsm c)
  (case c
    (#\! :error)
    (#\H :want-i)))

(fsm:defstate hi-fsm :want-i (fsm c)
  (if (char-equal c #\i)
      :done
      (values :initial t)))

(fsm:defstate hi-fsm :done (fsm c)

We can observe it in action by running it through a couple of strings and measuring the terminal state of the machine:

(let ((fsm (make-instance 'hi-fsm))
       (input "Oh? Hello there. Hi. How are you!?"))
  (map 'list 
       (lambda (c) 
         (if (eql :error (fsm:state fsm))
             (format t "Skipping: ~S~%" c)
             (funcall fsm c))) 
       input)
  (fsm:state fsm))

Resulting in :DONE as the final state of hi-fsm.

This will iterate the string character by character through the machine we just defined, and unless the machine is in an error state, it will submit each token, then return the state of the machine when it has completed its run.

If we change the string to one where neither “Hi” nor “!” appear at all, the machine will remain in :initial and if “!” appears before “Hi”, then the machine will leave the loop in an :error state, and any tokens after the “!” won’t even be sent into the machine. Changing the binding of input in the above to “Go away! Now.” results in the value of :ERROR and the printing of the lines

Skipping: #\ 
Skipping: #\N
Skipping: #\o
Skipping: #\w
Skipping: #\.

So there you have it. You now know how to leverage the powers of Common Lisp and MOP to build a pretty cool and useful abstraction for a pretty common algorithm.

Nicolas Martyanoff: Display images in Emacs from Common Lisp

Mon, 14 May 2012 19:10:00 GMT

The slime repl is used to interact with a Common Lisp process, and while printing textual information is good enough most of the time, being able to display an image can be really useful.

I used to simply call an external process, the image viewer feh, but the following method is more interesting.

You first need to load the slime-media module which is provided with slime, for example by adding it to the list of modules to load with slime:

(slime-setup '(slime-repl
               slime-asdf
               slime-fuzzy
               slime-banner
               slime-indentation
               slime-media))

You then have to authorize Slime to evaluate arbitrary lisp forms coming from the Common Lisp process:

(setq slime-enable-evaluate-in-emacs t)

Let's display an image:

It looks pretty good, but I really wanted to have the return value printed below the image, and not on its right.

The following function is a modification of slime-media-insert-image; an end-of-line character is inserted just after the image. This way, the result value is printed below the image. Furthermore, when more than one image is displayed during the same call, it won't be crammed on the same line.

(defun nm-slime-media-insert-image (image string)
  (with-current-buffer (slime-output-buffer)
    (let ((marker (slime-output-target-marker :repl-result)))
      (goto-char marker)
      (slime-propertize-region `(face slime-repl-result-face
                                      rear-nonsticky (face))
        (insert-image image string)
        (insert ?\n))
      ;; Move the input-start marker after the REPL result.
      (set-marker marker (point)))
    (slime-repl-show-maximum-output)))

Didier Verna: Monday Troll: the syntax extension myth

Mon, 14 May 2012 00:00:00 GMT

Here's a little Monday Troll.

To my greatest disappointment, I discovered today that it is not possible to replace Lisp parenthesis by, say, ... curly braces. What a shame. Hell, it's not even possible to freely mix the two. Very naively, I had expected that:

(set-macro-character #\{ (get-macro-character #\())
(set-macro-character #\} (get-macro-character #\)))

would suffice, but no. All implementations that I have tested seem to agree on this, although the error messages may differ. For instance, trying to evaluate {and} gives you an "unmatched close parenthesis error" except for CMU-CL which chooses to ignore it, but then report an end-of-file error. The unmatched close parenthesis, of course, is the closing curly brace! So what is going on here?

When an opening curly brace is read, the original left paren macro function is called. In SBCL for instance, this is SB-IMPL::READ-LIST, which looks for a hardwired right paren on the stream. Yuck. It doesn't find one, but it finds my closing brace which triggers the "standalone" right paren behavior (spurious paren alert). In passing, it also surprised me that SB-IMPL::READ-LIST is not implemented in terms of READ-DELIMITED-LIST.

EDIT: as mentioned in several comments, we could use read-delimited-list to look for a closing curly brace, but even this won't work completely. The problem is with dotted lists (see Pascal's comment). SBCL hard-wires #\) in its dotted lists parsing procedures.

So it appears that dispatching macro characters are only shaky. What we miss is a true concept of syntactic categories (Common Lisp character syntax types are close, but not quite there yet). In fact, TeX, with its notion of catcodes (category codes), seems to be the only language that gets this right. Ideally, any character with associated status LIST TERMINATOR should do as good as a right paren (the problem is only with closing, not opening).

Instead of hard-wiring the right paren in the Lisp parser, a quick workaround would be to check whether the next character on the stream is a dispatching one, and in such a case, whether its macro function is the one originally associated with the right paren. If so, it should then simply stand as a list terminator. This is actually an interesting idea I think: could the built-in macro functions become equivalent to actual category codes, and could we completely remove hard-wired characters in Lisp parsers?

Anyway, this whole story is a true scandal because it ruined an otherwise cool live demo of mine. So much for syntax extensibility. I will immediately complain to the concerned authorities.

Looking for the concerned authorities to complain to... please wait.

Quicklisp news: Updates coming soon

Thu, 10 May 2012 15:40:00 GMT

I usually like to update Quicklisp libraries on the first weekend of each month. A few things conspired to delay me this time around, but I hope to have updates published by this Sunday, May 13th.

The coming update includes version 0.10.7.1 of CFFI, which has a handful of backwards-incompatible changes. For the most part, Quicklisp projects have been updated to fix compile-time problems with CFFI compatibility. There remains the possibility that some problems lurk that will only appear at runtime. If you work on a project that relies on CFFI, you may want to run it through its paces with the new version to make sure everything still works properly.

update I was wrong about the CFFI updates. There haven't been any backwards-incompatible changes from 0.10.6 to 0.10.7.1. Sorry for the confusion.

Lispjobs: COMMON LISP/CLOS - Evergreen, Colorado USA

Thu, 10 May 2012 06:53:25 GMT

Secure Outcomes is looking for a hardcore Lisp developer with deep experience in Common Lisp/CLOS, image processing and artificial intelligence techniques, advanced cryptography, communications, GUI, and embedded real-time. Direct experience in electronic livescan fingerprint collection, archiving, and transmission, and/or crypto NIST certification is a distinct plus.

Secure Outcomes is a venture funded closely-held Colorado company that designs and manufactures the very best electronic livescan fingerprint collection, lookup, and transmission systems available for the price.

We design all of our systems at our headquarters facility in beautiful Evergreen, Colorado at an elevation of 7,200-feet in the foothills of the Rocky Mountains about 25-miles west of Denver.

Secure Outcomes provides terrific opportunities for fiercely talented and driven people.

Our systems combine advanced FBI-Certified fingerprint scan-head devices and super-compact 21st century low-power control electronics and display touchscreens with custom control and analysis software designed with the user in mind to yield the easiest to use, most capable, and most user-friendly systems on the market – all in Lisp.

Evergreen, Colorado, the home of Secure Outcomes, offers an extremely high-quality of life. World class skiing is just about an hour away at Keystone, Vail, Winter Park, and many others. Lake Evergreen offers excellent night/day ice skating and summer water sports/sailing. Mountain and touring biking and hiking abound during the summer months. In addition, Evergreen supports a thriving cultural community that includes the Evergreen Jazz Festival in the summer and the Evergreen Chorale and Evergreen Players for live music and theater year round. Evergreen has 300+ days of annual sunshine.

Competitive salaries. Fully paid medical.

See our products in action at http://www.youtube.com/watch?v=Ti2q8eK0l58

Resumes to lisp@secureoutcomes.net. No calls please.

http://www.secureoutcomes.net/Employment.html

ECL News: Bug reports: please do them!

Wed, 09 May 2012 21:50:29 GMT

I am sorry. Really, I am. I cannot follow you all in all your usual communication channels: comp.lang.lisp, the ecl forum, #lisp irc channel... But please consider this:

First of all, ECL is maintained in free time. In this sense, it is more beneficial for you if the maintainers spend more time coding than trolling about lisp.

Second, an IRC channel is not a place to look for support. Sourceforge offers you a bug reporting facility (https://sourceforge.net/tracker/?group_id=30035), a feature request database (https://sourceforge.net/tracker/?group_id=30035&atid=398056), a mailing list (http://news.gmane.org/gmane.lisp.ecl.general/)...

Response times are usually very fast, and only in the most difficult cases (it involves legacy code, requires some nonstandard architecture, etc), or when a release is on queue, does it takes longer.

In particular, in order to ease your life, bug reporting and feature requests are *now* possible for people without an account in Sourceforge. This is an experiment and I will try to keep it that way as long as spammers allow. Please make good use of it.

Finally, if you find something does not work the way you expect it, instead of spending a week on it and then complaining about it anywhere, just leave a message in any of those communication channels I mentioned above. It will be more productive for you and for the free software community in general.

Ah, and before I forget it: a good bug report for ECL needs at least the following information: 1) platform information and operating system, 2) configuration options, 3) if it failed at configuration time, the log, 4) if it failed at build time (i.e. during "make") a log of that, 5) if it fails with a standard (quicklisp library), steps to reproduce it, 6) if it fails with your code, ideally some sample to reproduce it.

Slava Shirokov: Hinge -- v0.0.1

Wed, 09 May 2012 06:10:16 GMT

Hinge -- v0.0.1:

Hinge, my Node.js inspired API for Common Lisp wrapping libev is finally to a state where I feel okay giving it a non-zero version.

I've decided now would be a good milestone because the HTTP server functions, and has a set of exported APIs that let you reply to the requests it generates.

The quick set of terrible benchmarks I performed on the "Hello World!" example show me a requests/second throughput of 1500-2500 depending on the mood of ab (and the state of the number of sockets available on localhost). I intend to run it through a more serious (and HTTP/1.1 supporting) tool in the future, but for now ab has convinced me that the decisions I've made so far have not been abysmal.

The examples directory contains a bunch of example programs, or in the earlier cases, code snippets demonstrating various levels of functionality.

In the near future, in addition to covering the issues in the README, adding additional polish, and any bugs or blatant performance issues that come up I would like at some point to add support for:

HTTP Clients
SSL Sockets and Servers
SSL HTTP Servers and Clients
Async wrappers around FS calls
Websockets (RFC6455) support
Mongrel2 ZeroMQ handler support (through m2cl)
Mongrel2 ZeroMQ protocol-compatible server
SPDY Support (Somewhere down the list, maybe)

Any feedback is welcome however way you want to deliver it to me. This initial version stands to serve as a basic starting point for the concepts I wanted to convey, the take on the core APIs I wanted to offer, and a proof of a series of related concepts and architectural decisions. It may (hopefully) be useful as a stable base for progress, rather than chasing an otherwise potentially unstable head.