Planet Lisp

Lispjobs: Common Lisp and Ruby on Rails Programmers, MNCA Dental, Ft Lauderdale, Florida

Thu, 02 Feb 2012 23:04:52 GMT

MCNA Dental of Ft Lauderdale has landed every contract it went for and is quickly becoming a standout in the field of dental insurance. We are developing cutting-edge enterprise software to help manage that growth. Management has asked me to let the world know MCNA is looking for solid, self-starting Common Lisp and/or Ruby/Rails developers who can self-manage on significant sub-projects and execute them efficiently, handling everything from design to coding on their own.

“Candidates should be willing to relocate to sunny Ft Lauderdale, FL for permanent or consulting positions, permanent preferred.

“We are doing very cool things in Common Lisp and Postgres but using Ruby/Rails3 as a bridge to manage the new business now. Ruby/Rails developers can expect to be doing Lisp in the future. The future also includes the IT department spinning off as a standalone enterprise involved in more than just dental insurance. We have a strong team of five developers but given the ambitious software goals and rapid growth MCNA has room for several strong candidates. Note that a strong developer in any Lispy language will be considered for a permanent position.

“Resumes and letters of interest can be sent to ktilton at mcna dot net

Jorge Tavares: Sorting algorithms used in the CL implementations

Thu, 02 Feb 2012 09:45:29 GMT

Which sorting algorithm should one implement when developing a program? The best answer is probably none. Use the sort provided by your system/library/etc. Unless you know your input data has some special properties that you can take advantage of, the provided sort should be enough for your needs and probably is more efficiently implemented.

However, I think it is important to know what sorting algorithm is implemented. If one knows the properties of the data, it is possible to understand if the provided sort can or will pose a problem. In the same way a programmer shouldn’t implement a sorting algorithm every time it needs to sort something, the programmer should also be aware of the limitations/advantages of the system sort. That way one can decide if a special sort is needed or not.

Common Lisp provides the functions sort and stable-sort. The HyperSpec describes their operation well but it does not define the sorting algorithm. That decision is left free to the implementations. In addition, both functions don’t necessarily share the same algorithm. The difference between the two is that the second function sorts in a way that guarantees stability, i.e., two elements that are equal remain in the same position after sorting is completed. The use of sort and stable-sort requires some care (see the section sort pitfalls) but lets focus on the algorithms and not on its usage.

What sorting algorithms do the major open source CL implementations actually implement? I was curious about it and went to check the source for ABCL, CCL, CLISP, CMUCL, ECL and SBCL. Not surprising, we find some differences between the implementations. What it was more unexpected to discover is that some implementations also use different sorting algorithms according to the sequence type. A quick survey of the findings is summarized in the following table (if anythings is incorrect, please tell me). The links for the source code are in the implementation name (careful, in CCL and SBCL there are two links).

Implementation	sort	stable-sort
ABCL	merge sort (lists) / quicksort	merge sort
C CL	merge sort (lists) / quicksort	merge sort
CLISP	tree sort	tree sort
CMUCL	heapsort	merge sort
ECL	merge sort (lists) / quicksort	quicksort (strings + bit vectors) / merge sort
SB CL	merge sort (lists) / heapsort	merge sort

In terms of the implementation of sort, quicksort is the most used algorithm, followed by heapsort. The choice for these algorithms is expected. Both have an average-case performance of O(nlgn) and heapsort guarantees a worst-case performace of O(nlgn) too. Quicksort has a worst-case performance of O(n²) but it can be optimized in several ways so that it also gives an expected worst-case performance of O(nlgn). However, it seems that the quicksort implementations are not completely optimized. In ECL (and ABCL) quicksort implements a partition scheme which deals better with duplicate elements (although is not the three-way partitioning) but it always picks as pivot the first element. CCL chooses the pivot with a median-of-3 method and always sorts the smaller partition to ensure a worst-case stack depth of O(lgn).

As for CLISP, I think it uses a tree sort but I am not entirely sure. The only source file I could find with a sort implementation was sort.d and it looks like it contains an implementation of tree sort with a self-balanced binary tree, which also gives this algorithm an average and worst-case performance of O(nlgn).

As expected, most of the implementations use merge sort to implement stable-sort since it is a stable sort with average and worst-case performance of O(nlgn). Apparently, all implementations are bottom-up merge sorts with the exception of CCL and ECL. Another interesting thing is that merge sort is also used for lists in sort, in most of the implementations. However, I found it surprising to find quicksort in the stable-sort column because it is not a stable algorithm. Since it is only used for strings and bit vectors, it is not really an issue. While reading the source code of the implementations, I realized that ABCL was using quicksort in stable-sort for all non-list sequences. This is a problem that exists in the current 1.0.1 release but I’ve sent a bug report with a quick fix to the maintainers. The next release should have stable-sort fixed.

This exploration of the sorting algorithms used in the open source implementations was very educational and interesting to me. I’ve learned what algorithms are actually used and enjoyed seing how they were implemented. Just spotting the issue in ABCL stable-sort made this review worthwhile. I think there is still room for improvement in some implementations but knowing now the strengths and weaknesses of the sorts in CL is already good enough. On a final note, I just wonder what are the algorithms used in ACL and LW.

Filed under: Programming Tagged: Common Lisp, Lisp, Sorting Algorithms, Survey

John Q. Splittist: A year of living vi-rously

Tue, 31 Jan 2012 14:24:58 GMT

For reasons which are not apparent* even to myself (although I suspect it had to do with general contrariness mixed with regret for some arrogant comments from the past) I spent 2011 emacs-free. More than that, I spent it using only vi (or, rather, vim) to do my text – including program text – editing. Since I couldn’t get SLIMV to work at the beginning of 2011 – it is in much better shape now – I even gave up the wonder that is Slime/Swank.

I can report that, even for a duffer like me, it is possible to develop very efficiently with vi(m) and a repl running in another terminal. So there, old-splittist!

And just to make sure I’m not making any forward progress, I’ve switched back to emacs for 2012. Now my buffers are full of “jjjj” and “kkkk”…

* I do this a lot, don’t I?

Russ Tyndall: I Published My Common Lisp Docstring Search Engine

Mon, 30 Jan 2012 21:13:38 GMT

My Common Lisp documentation search engine has been published to http://lisp-search.acceleration.net. In a previous post I wrote about using the montezuma full-text search engine to build an index of documentation available from within my common lisp runtime. I ended up going the extra mile on this one and indexing all of the documentation available for all of the packages in quicklisp (as well as readme files and other packages that sbcl had already loaded). The result is a 90M search index (4M tar.gz) that can be used search through all of the doc strings of all of the easily loadable packages.

The user interface is a bit clunky, searches don't always return the most relevant results first, but it is live, fast, and seems already useful. Perhaps with some help from the internet, this search engine can reach its full potential. I named the software package that does this manifest-search-web, because it was inspired by gigamonkey's manifest project. I still have not come up with a reasonable name for the published search engine (lisp-search seems a touch blasé and under-descriptive).

Hopefully, I will never again spend time writing a library only to find the already written, open source alternative after I publish mine. Also, perhaps this will inspire better doc-strings, now that doc-strings might be what leads to someone finding your project.

Other things todo:

Integrate manifest-search with slime
Have the documentation index be distributable in quicklisp (not sure how to do that efficiently)
Find a way to unify CLIKI, l1sp.org, lisp-search and other lisp documentation resources into a more cohesive single website / search
Improve the query language to ensure that it behaves according to user expectations as opposed to lucene expectations

As always, please report bugs and make suggestions for improvements. Cheers and happy lisping.

Tamas K Papp: cl-cairo2: new maintainer, new license

Thu, 26 Jan 2012 09:12:00 GMT

cl-cairo2 was one of the first Common Lisp libraries I wrote, but I haven't been using it much for the last year or so (currently I am experimenting with cl-pdf as a backend for my new plotting library, which will be released soon). I have been pretty busy with research, so I didn't have time to merge (and test) patches, also, I didn't even contemplate updating the library to make use of the latest version of cairo. So when Ryan Pavlik contacted me about adding compatibility with cl-freetype2, I asked him whether he wants to take over as a maintaner. He kindly agreed, so I have transferred the repository to him, and he already merged a lot of patches.

One last thing that I wanted to fix before handing the repository over is the license. Originally, the library was licensed under the GPL — in retrospect, I think that

I was very naive about software licenses, and didn't really understand what GPL means in the context if Common Lisp (I still don't claim that I do :-P), and
I didn't realize that there are a lot of commercial applications in the Common Lisp world, whose authors are wary of depending on GPL'ed libraries.

Consequently, I received many complaints about the license of the library, and decided to change it. I picked the Boost Software License, and contacted all who contributed to the library for permission. All contributors approved the change, so now the library has a simple, permissive non-copyleft free software license. However, it is always possible that I missed someone, so if you contributed to cl-cairo2 in the past but didn't hear from me regarding the license change, please get in touch (eg via the Github issue tracker).

I would like to thank (in alphabetical order) Ala'a Mohammad Alawi, Jay Bromley, Pau Fernández, Johann Korndoerfer, Peter Mikula, and especially Kei Suzuki (who did the last major reorganization) for their contributions to the library (again, I apologize if I missed anyone). I would also like to thank Ryan for taking over — I am convinced that the library is in good hands.

ABCL Dev: Closing in on CLOSER-MOP in abcl-1.1.0-dev

Tue, 24 Jan 2012 22:32:00 GMT

In response to inquiries recently on #abcl (from @loke among others), we'd like to point out that recent work on the Armed Bear trunk--what will be abcl-1.1--by Rudi has started to converge on a plausible AMOP strategy. Stay tuned.

Andy Hefner: Piddling Plugins

Mon, 23 Jan 2012 12:15:37 GMT

The Shuffletron music player, in various branches, has accumulated some neat features (particularly last.fm scrobbling in Brit Butler's branch) that deserve merging, and ought to be cleanly separated from the core of the program. Leslie Polzer sent me a novel implementation early on which used generic functions for extensibility, adding/removing methods via the MOP as plugins load/unload. Clever as that was (and I'm impressed how little code is required, rereading the patch now), I wasn't comfortable with it, and the lack of a pressing need for a plugin interface let me put it off for a good long while.

Building extensibility around generic functions seemed the right thing to do though, and a slightly different idea, of writing plugins in the style of mixins and calling CHANGE-CLASS to enable them at runtime, stuck in the back of my head until (with some prodding) I was motivated to try it out. It's hardly a new idea (both Gsharp and McCLIM contain implementations of similar ideas, as does the AMOP book, just to name a few examples), and a minimal implementation doesn't take much code at all:

(defvar *configurations* (make-hash-table :test 'equal))

(defun configuration (plugins)
  (or (gethash plugins *configurations*)
      (setf (gethash plugins *configurations*)
            (make-instance 'standard-class
                           :name (format nil "MY-APPLICATION~{/~A~}" plugins)
                           :direct-superclasses (cons (find-class 'my-application)
                                                      (mapcar #'find-class plugins))))))

(defun reconfigure (application plugins &rest; initargs)
  (apply #'change-class application (configuration plugins) initargs))

(defun active-plugins (instance)
  (mapcar #'class-name (rest (sb-mop:class-direct-superclasses (class-of instance)))))

(defun enable-plugin (application plugin &rest; initargs)
  (apply #'reconfigure application (adjoin plugin (active-plugins application)) initargs))

(defun disable-plugin (application plugin)
  (reconfigure application (remove plugin (active-plugins application))))

(defun make-application (&rest; initargs)
  (apply 'make-instance (configuration '()) initargs))

This isn't an ideal implementation, and there's a limit to how good it's going to get when CLOS doesn't fully support anonymous classes. However, a more serious attempt should work on arbitrary classes, provide a place to hang init/shutdown code for plugins, and the ability to list which plugins are enabled within an instance.

Piddling Plugins

Piddling-plugins adds these features using only slightly more code than above, along with some superfluous macro magic for writing defun-style definitions that are extensible by plugins. I've made light use of it in a branch of Shuffletron, confirming to myself that it's a good fit.

The code is tiny and self-explanatory, so I'll just post the examples from the README file for fun.

Examples

Imagine we have written a music player, looking something like this deliberately simplified code:

(defclass music-player () ())

(defun run-music-player ()
  ;; You need to set or bind *application* to your application instance
  ;; if you use defun-extensible. It's a good idea even if you don't.
  (let ((*application* (make-instance 'music-player)))
    (init-audio)
    (init-library *application*)
    (loop (execute-command (read-line)))))

Functions extensible by plugins can be defined using DEFUN-EXTENSIBLE. This is just syntactic sugar for defining a generic function specialized on the application object, with a wrapper that passes in the value of *APPLICATION*.

(defun-extensible execute-command (command)
  ...)

(defun-extensible play-song (song)
  ...)

(defun-extensible song-finished (song)
  ...)

Plugins extend the behavior of the application by defining methods on the extensible functions (or rather the generic functions defined behind the scenes, which are prefixed by "EXTENDING-"):

(defclass scrobbler ()
  ((auth-token :accessor auth-token)))

(defmethod plugin-enabled (app (plugin-name (eql 'scrobbler)) &key &allow-other-keys)
  (setf (auth-token app) (get-auth-token))
  (format t "~&Scrobbler enabled.~%"))

(defmethod plugin-disabled (app (plugin-name (eql 'scrobbler)))
  (format t "~&Scrobbler disabled.~%"))

(defmethod extending-song-finished :after ((plugin scrobbler) song)
  (scrobble song (auth-token plugin)))

(defclass status-bar () ())

(defmethod extending-play-song :after ((plugin status-bar) song)
  (redraw-status-bar))

(defmethod extending-execute-command :after ((plugin status-bar) command-line)
  (declare (ignore command-line))
  (redraw-status-bar))

To enable a plugin:

(enable-plugin *application* NAME [INITARGS...])

To disable a plugin:

(disable-plugin *application* NAME)

To set precisely which plugins are enabled:

(reconfigure *application* LIST-OF-PLUGINS [INITARGS...])

References

Strandh R., Hamer J., Baumann G. "Using Stealth Mixins to Achieve Modularity" (2007)
- Implentation in Gsharp: http://common-lisp.net/cgi-bin/gitweb.cgi?p=projects/gsharp/gsharp.git;a=blob;f=utilities.lisp
"Modes" implementation in McCLIM's ESA framework (formerly part of Climacs)
- http://git.boinkor.net/gitweb/mcclim.git/blob/HEAD:/ESA/utils.lisp#l446
Throwaway classes (comp.lang.lisp): http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/78ef4a5fcd6a1661?pli=1
The Art of the Metaobject Protocol, Section 2.4
- http://www.foldr.org/~michaelw/lisp/amop-programmatic-class.lisp
ContextL - http://common-lisp.net/project/closer/contextl.html
Dynamic Classes - http://common-lisp.net/project/dynamic-classes/

Zach Beane: ZS3 updates

Sat, 21 Jan 2012 15:35:57 GMT

I updated ZS3, my CL library for interacting with Amazon S3, with a few new features.

First, there's support for S3's new multi-object deletion system. In the past, S3 required one API call per object to delete stuff. Now you can delete up to 1000 objects with a single call, and ZS3's existing delete-objects function has been updated to use the new interface and will automatically split up the objects to be deleted into groups of 1000 as needed.

Multi-object deletion can be a big deal, since each API call costs money.

Second, there's support for the "reduced redundancy" storage class. Reduced redundancy storage is less durable than standard storage, and it comes with a corresponding reduction in cost. You can choose reduced redundancy when using put-object or related functions, or set the storage class after the fact with set-storage-class.

Third, there's support for automatic object expiration, aka bucket lifecycle configuration. With bucket lifecycle rules you can specify that objects with names that match a certain prefix expire after a certain period of time. You can change a bucket's lifecycle configuration with bucket-lifecycle and related functions.

Automatic expiration of objects is another way to save money on API calls. If objects are deleted automatically, you don't need to use any API calls at all to get rid of them.

Please let me know if there's an S3 feature you really want to see in ZS3. I feel like I'm on a roll and would love to add some more stuff that people need.

Vladimir Sedach: Upcoming presentation about Parenscript

Thu, 19 Jan 2012 00:06:00 GMT

I'm going to be giving a talk about Parenscript to the Montreal Scheme/Lisp Users Group on Thursday, January 19 (meeting details here).

The slides I'm going to be using are here, and a list of links referenced in the talk is below. The last time I gave a presentation on Parenscript was to LispNYC in 2007. Parenscript has received a huge number of changes and improvements since then, and continues to be the best language/compiler to JavaScript and one of the best tools available for web application development. What's also new since 2007 are libraries and tools that extend Parenscript: Red Daly has added CLOS and the Common Lisp condition system to JavaScript as a Parenscript library, and there are now several options for interactive development with SLIME in your browser.

Links:

Paul Khuong: Migration and Synopsis

Wed, 18 Jan 2012 23:56:00 GMT

This blog has been going for five years. Back then, it seemed like the only widely-used static blog generators were Blosxom or pyBlosxom. They weren’t that hard to set up, but getting everything right rather than good enough is a lot of work. Latex and MathML support was also very weak, so I wound up using a (insane) one-off hack with tex4ht. I feel like Octopress and MathJax now do everything I need out of the box, better than anything I could design by myself.

The permalinks from the old blog are still around, but not the rss feeds or the date-based links.

I figure this is a good opportunity to make sure the (marginally useful) permalinks are available somewhere else than via google.

Lisp-related posts

Another way to accumulate data in vectors describes a copying-free extendable vector. The advantage over the usual geometric growth with copy is that the performance with respect to the number of elements added is much smoother. Runtimes are then more easily predictable, and sometimes improved (e.g. right when a copy would be needed). It’s also more amenable to a lock-free adaptation, while preserving O(1) operation complexity (assuming that integer-length on machine integers is constant time), as shown in Dechev et al’s “Lock-free Dynamically Resizable Arrays”.

Common Cold is a really old hack to get serialisable closures in SBCL, with serialisable continuations built on top of that. Nowadays, I’d do the closure part differently, without any macro or change to the source.

Concurrency with MVars has short and simple(istic) code for mvars, and uses it to implement same-fringe with threads.

Constraint sets in SBCL: preliminary exploration summarises some statistics on how constraint sets (internal SBCL data structures) are used by SBCL’s compiler.

SBCL’s flow sensitive analysis pass explores what operations on constraint sets actually mean. This, along with the stats from the previous post, guided a rewrite, not of constraint sets, but of the analysis pass that uses them. The frequency of slow operations or bad usage patterns is reduced enough to take care of most (all?) performance regression associated with the original switch to bit-vector-based constraint sets, without penalising the common case.

Finalizing foreign pointers just late enough is a short reminder that attaching finalizers to system area pointers isn’t a good idea: SAPs are randomly unboxed and consed back, like numbers.

Hacking SSE Intrinsics in SBCL (part 1) walks through an SBCL branch that adds support for SSE operations. Alexander Gavrilov has kept a fork on life support on github. There’s still no part 2, in which the branch is polished enough to merge it in the mainline.

In the meantime, Complex float improvements for sbcl 1.0.30/x86-64 built upon the original work on SSE intrinsics to implement operations on (complex single-float) and (complex double-float) with SIMD code on x86-64. That sped up most complex arithmetic operations by 100%. That work also came with support for references to unboxed constants on x86oids; this significantly improved floating point performance as well, for both real and complex values.

Initialising structure objects modularly is a solution to a problem that I hit, trying to implement non-trivial initialisation for structures, while allowing inheritance. Tobias Rittweiler points out that the protocol is very similar to a common CLOS pattern where, instead of functions that allocate objects, class designators are passed. It also looks a bit like the way Factor libraries seem to do struct initialisation, but with actual initialisation instead of assignment (which matters for read-only slots).

An Impure Persistent Dictionary is an example of a technique I find really useful to implement persistent versions of side-effectful data structures. Henry Baker has a paper that shows how shallow binding can be used to implement persistent arrays on top of functional arrays, with constant-time overhead for operations on the latest version. It’s a really nice generalisation of trailing in backtracking searches. Here, I use it to get persistent hash tables in only a couple dozen lines of code.

Pipes is an early attempt to develop a DSL for stream processing, like an 80% SERIES. I’ve refocused my efforts on Xecto, which only handles vectors, rather than potentially unbounded streams. The advantage is that Xecto looks like it has the potential to be simpler while achieving near-peak performance to me; the main downside is that vectors don’t allow us to represent control flow as data via lazy evaluation… and I’m not sure that’s such a bad thing.

The post on string-case is an overview of how I structured a CL macro to dispatch that compares with string= instead of eql. If I were to do this again, I’d probably try and improve string=; I later tested an SSE comparison routine, and it ended up being, in a lot of cases, faster and simpler (with a linear search) than the search tree generated by string-case.

The type-lower-bound branch describes early work on a branch that provides a way to shut the compiler up about certain failed type-directed optimisations. A lot of the output from SBCL’s compiler amounts to reports of optimisations that couldn’t be performed (e.g. converting multiplication by a constant power of two to a shift), and why (e.g. the variant argument isn’t known to be small enough). Sometimes, there’s nothing we can do about it: we can’t show the compiler that the argument is small enough because we know that it will sometimes be too large! Yet, CL’s type system (like most) does not let us express that information. Programmers are expected to provide upper bounds on the best static type of values (e.g. we can specify that a value is always a fixnum, although it may really only be integers between 0 and 1023). We would like a way to specify lower bounds as well: “I know that this will take arbitrary fixnum values.” Once we have that, the compiler can skip reporting optimisations that we know can’t be performed (as opposed to those we don’t know whether they can be performed).

Finally, Yet another way to fake continuations sketches a simple but somewhat inefficient way to implement continuations for pure programs. It may be useful for IO-heavy applications (web programming), or in certain cases similar to backtracking search, but in which most of the work is performed outside of backtracking (e.g. during constraint propagation).

General low-level programming issues

SWAR implementation of (some #’zerop …) sketches how we can use SIMD-within-a-register techniques to have fast search for patterns of sub-word size. A degenerate case is when we look for 0 or 1 in bit vectors; in these case, it’s clear how we can test whole words at a time. The idea can be extended to testing vectors of 2, 4, 8 (or any size) -bit elements. I haven’t found time to move this in SBCL’s runtime library (yet), but it would probably be a neat and feasible first project.

Revisiting VM tricks for safepoints explores the performance impact of switching from instrumented pseudo-atomic code sequences to safepoints. The bottom line is that it’s noise. However, some members of the russian Lisp mafia have used it as inspiration, and have managed to implement seemingly solid threaded SBCL on Windows! It’s still a third-party fork for now, but some committers are working on merging it with the mainline.

Fast Constant Integer Division has some stuff on integer division by constants. It’s mostly superseded by Lutz Euler’s work to implement the same algorithm as GCC. There are some interesting identities that can be used to improve on that algorithm a tiny bit and, more interestingly, to implement truncated multiplication by arbitrary fractions. I only stumbled upon those a long time after I wrote the post; I’ll try and come back to this topic in the coming months.

There’s more to locality than caches tracks my attempts to understand why a data structure designed to be cache-efficient did not perform as well as expected. It turns out that cache lines aren’t exactly read atomically (so reading two adjacent addresses may be significantly slower than only one), and that sometimes L2 matters less than TLB. The latter point was an important lesson for me. TLBs are used to accelerate the translation of virtual addresses to physical; every memory access must be translated. TLBs are usually fully associative (behave like content-addressed memory or hash tables, basically), but with a small fixed size, on the order of 512 pages for the slower level. With normal (on x86oids) 4KB pages, that’s only enough for 2 MB of data! Even worse: a cache miss results in a single access to main memory, which is equivalent to ~60-100 cycles at most; a TLB miss, however, results in a lookup in a 4 level page table on x86-64, which often takes on the order of 2-300 cycles. Luckily, there are workarounds, like using 2 MB pages.

Napa-FFT(2) implementation notes is where I try to make the code I wrote for a Fast Fourier transform understandable, especially why it does what it does. Napa-FFT and Napa-FFT2 are vastly faster than Bordeaux-FFT (and than all other CL FFT codes I know, on SBCL), but it’s still around 20-50% slower than the usual benchmark, FFTW. Napa-FFT3 is coming, and it’s a completely different approach which manages to be within a couple percent points of FFTW, and is faster on some operations.

0x7FDE623822FC16E6 : a magic constant for double float reciprocal is a surprisingly popular post. I was trying to approximate reciprocals as fast as possible for a mathematical optimization method. The usual way to do that is to use a hardware-provided approximation and then improve it with a couple iterations of Newton’s method. The post shows how we can instead use the way floats are laid out in memory to provide a surprisingly accurate guess with an integer subtraction. I actually think the interesting part was that it made for a practical use case for the golden section search…

Some notes on Warren has a couple notes about stuff in Warren’s book Hacker’s Delight. The sign extension bit probably deserves more attention; it seems like someone on #lisp asks how they can sign-extend unsigned integers at least once a month.

Two variations on old themes has some stuff on Linux’s ticket spinaphores, and is the beginning of my looking into Robin Hood hashing with linear probing for cache-friendly hash tables.

Interlude: Numerical experiments in hashing covers a first stab at designing a hash table that exploits cache memory. 2-left hashing looks interesting, but its performance was worse than expected, for various reasons, mostly related to the fact that caches can be surprisingly complicated. Two years later, More numerical experiments in hashing: a conclusion revisits the question, and settles on Robin Hood hashing with linear probing. It’s a tiny tweak to normal open addressing (insertions can bump previously-inserted items farther from their insertion point), but it suffices to greatly improve the worst and average probing length, while preserving the nice practical characteristics of linear probing. I’ve also started some work on implementing SBCL’s hash table this way, but there are practical issues with weak hash functions, GC and mutations.

Miscellaneous stuff

In Specify absolute deadlines, not relative timeouts and the sequel, I argue that we should have interfaces that allow users to specify an absolute timeout, with respect to a monotonic clock. Timeouts are convenient, but don’t compose well: how do we implement a timeout version of an operation that sequences two calls to functions that only offer timeouts as well? Any solution will be full of race conditions. PHK disagrees; I’m not sure if all of his complaints can be addressed by using a monotonic clock.

Finally, Space-complexity of SSA in practices has some early thoughts on how Static single assignment scales for typical functional programs. It’s fairly clear that many compilers for functional languages have inefficient (wrt to compiler performance) internal representations; however, it’s not as clear that the industry standard, SSA, would fare much better.