Planet Lisp

May 8, 2008

Sean RossRenaming files in Common Lisp.

So I ran across this while going through my morning catchup routine and thought it could do with some clearing up.

The part of the post I'm actually interested is this.


When Haskell can compete on those types of problems, it'll be easier to induce people to learn it. (Same with CL, my fav language....)

Now, bonus points for proclaiming CL as being his favorite language, but minus 10 billion for continuing the meme that CL is somehow unsuited for writing[1] these kinds of simple software (for whatever definition of simple you have).

One of the examples given is "I have a bunch of files, and I want to rename them all according to some pattern." and as it so happens translate-pathname[2] makes this wonderfully simple.


(defun rename-files (from to)
(dolist (file (directory from))
(rename-file file (translate-pathname file from to))))

And thats it, 3 lines of code, or for our simple testing purposes

(defun show-rename-files (from to)
(dolist (file (directory from))
(format t "Renaming ~A to ~A~%" file
(translate-pathname file from to))))

(show-rename-files "/usr/share/pixmaps/*.xpm" "/usr/share/pixmaps/backup-*.xpm")

The funny thing is that this isn't secret knowledge but is pulled straight from the Hyperspec (see the examples).

----
[1]: Please note, I said 'writing' them, not 'creating a 2k binary' of them, please people CL /= Unix.
[2]: Granted, the behavior of translate-pathname isn't specified in detail by the spec but that doesn't mean we cannot use it.

This post brought to you by Lispworks 5.1, clisp 2.41 and SBCL 1.0.12  

Gary KingSoftware Engineering Radio interviews Richard Gabriel

This may be old news for everyone else, but Dick Gabriel was interviewed on Software Engineering Radio about a million years ago (back in 2007).

I finally got around to listening to it yesterday. The interview is very nice and covers the early history of Lisp, the AI winter, some of what Lisp special, CLOS, poetry, and creativity. Good stuff. Dick is a good talker and explainer who has pushed himself in many different and admirable ways.

There also a good song at the end.

 

Nick LevineJFLI and the memory leaks of doom

Finally, I've got the OK to open source some of the work I was doing last year. First step is the fixes I made to jfli (Java Foreign Language Interface), now available for CVS download or as a tarball. It's the first time anyone has posted changes to this library in some years, so I took the bull by the horns and gave it version number 0.2.

For the record, this is what I have changed. It was an interesting object lesson in trying to get two GCs to be nice to each other.

Memory problems:

  1. add-special-free-action takes a symbol, not a function. If you give it a function it doesn't do anything. In this case, that meant that all global-refs were leaking on the Java side. That's a hell of a pile-up - run anything for long enough and Java will run out of memory.

  2. Lisp processes just accumulated in *process-envs*, which meant that the associated stacks etc end up leaking on _both_ sides of the fence, i.e. into both Java and lisp. My first attempt at solving this involved a call to mp:ensure-process-cleanup but...

  3. Suppose a new thread allocates before mp:*current-process* has been set. Then delete-global-ref might be called, which invokes current-env for the first time on this thread, which goes ensure-process-cleanup with null mp:process. SEGV. Farfetched? Well, it happened.

  4. The "access functions" (calls to anything seen in defvtable, i.e. all the JNI's calls into the JVM) failed to memoize a dereferenced foreign-slot-value which was the same every time, and so burned 56 unnecessary bytes per shot (in lisp). This was reclaimed by the GC but it messed up the allocation figures when I was out hunting for real leaks.

  5. Untimely Finalization

    Consider this little problem which exhibited pathological behaviour from time to time:

    • The special free actions are not run when a GC occurs inside mp:without-interrupts, because that could cause a deadlock if the action function claims any locks (e.g. uses hash-tables). Instead of freeing them, the GC just keeps them alive with their special free actions intact.
    • The system maintains a table of all of the objects marked for special free actions (so the GC can find them all easily). Unfortunately, flag-special-free-action takes O(n2) time for n objects in the table. ["Ouch", says Nick.]
    • This table is enlarged by flag-special-free-action, inside without-interrupts to make it atomic with respect to other calls to flag-special-free-action and finalization.
    • If you're unlucky, all of the GC operations are triggered by the enlargement of this table.

    This final aspect completed a vicious cycle: none of the special objects were ever freed, because all of the GC operations occurred inside without-interrupts and hence their special free actions could not be run at that time. Excessive allocation occured, caused by the enlargement of the table which was always filled again quickly for the same reason. (By "excessive", I mean images bloating to over 1GB in very short order.)

    The recommended solution from lispworks-support was to manually mark-and-sweep generation 0 every 1000 or so allocations. Without it, you'd occasionally get generation 0 trying to climb over 1GB while you're sat there wondering why your emacs was running so slow.

Non-memory problems:

  1. Exports from JFLI package of box-integer and unbox-integer instead of the documented box-int and unbox-int - I restored the documented behaviour.

  2. I needed more configurable exception handling.

  3. No support for system building - you needed a live JVM connection in order to macroexpand source and so couldn't save the image (well, you could, but when you restarted it you wouldn't be able to connect).

Fixed by upgrading to LispWorks 5.1:

  1. Occurances of java.lang.NullPointerException, java.lang.ArrayIndexOutOfBoundsException, etc which had no explanation even after reading Sun Java sources.

    This turned out to be caused by a bug in the FLI, which could leave the CPU's direction flag set incorrectly in some cases. When the direction flag was set incorrectly, some optimized memory copying routines would corrupt adjacent objects. The bug, fixed in LispWorks 5.1, affected 32-bit x86 platforms running Linux, FreeBSD or Mac OS X (not Windows).

 

Ingvar Mattsson8 May 2008

I told a lie, unintentionally. There is anon CVS available. There's also a web-CVS interface..

So far, I've had a bunch of emailed queries. I haven't responded to all, hopefully I will.  

LispjobsCommon Lisp Scientific Programmer, SRI, Menlo Park, CA


SRI Common Lisp job. 

link

 

Leslie PolzerIronclad hashes in SBCL

Getting a SHA1 digest from Ironclad turned out to be a quite bumpy ride; its "convenience functions" are not convenient enough to take a string, they only operate on octets.

There’s a helper function that converts an ASCII string to octets, but I needed to be able to supply Unicode strings as well. I ended up finding STRING-TO-OCTETS:

(defun sha1 (str)
  (ironclad:byte-array-to-hex-string
    (ironclad::digest-sequence :sha1 (SB-EXT:STRING-TO-OCTETS str))))

If someone knows an easier or more portable way, I’m all for it. I’m also interested in other free implementations’ functions to convert their strings to octets.

 

May 7, 2008

Ingvar Mattsson7 May 2008

I don't think I posted about it, at the time. However, NOCtool now has a common-lisp.net project thingie. No public CVS at the moment. NOt even tarballs ready to snag. However, it's there and tehre's mailing lists and stuff. If you want to help, give me a shout.  

May 6, 2008

Ingvar Mattsson6 May 2008

I found a way of testiung Hunchentoot handlers from the REPL. Unfortunately, it only works with a non-mod-lisp instance. It also relies of having the server instance to be tested in *hunchentoot* (not a problem for me, but you may want to check that). I'll have a rummage through the Hunchentoot source at some point and see if I can get it to work when mod_lisp is involved.

(defun hunchentest (uri)
  (let* ((hunchentoot:*server* *hunchentoot*)
         (hunchentoot:*reply* (make-instance 
'hunchentoot::reply))
         (hunchentoot:*request* (make-instance 
'hunchentoot::request
                                               :uri uri)))
    (funcall (essay-dispatcher hunchentoot:*request*)
             hunchentoot:*request*)))
 

May 5, 2008

Ingvar Mattsson5 May 2008

Found the problem. Old, stale, flexi-streams installation. Snagging a new one and recompiling meant that (setf flexi- stream-bound) suddently was in existence and my faffing around now works. Thanks for the clues, Robert!  

John WiegleyReady Lisp version 20080428 now available

There is a new version of Ready Lisp for Mac OS X available. This version is based on SBCL 1.0.16, and requires OS X Leopard 10.5. The most notable change from the previous version is that 64-bit mode and experimental threading are no longer supported, since both have been known to have issues on OS X, while the purpose of Ready Lisp is to smoothly introduce Common Lisp to new users.

What is Ready Lisp? It’s a binding together of several popular Lisp packages for OS X, including: Aquamacs, SBCL and SLIME. Once downloaded, you’ll have a single application bundle which you can double-click – and find yourself in a fully configured Common Lisp REPL. It’s ideal for OS X users who want to try out Lisp with a minimum of hassle. The download is approximately 76 megabytes.

There is a GnuPG signature for this file in the same directory; append .asc to the above filename to download it. To install my public key onto your keyring, use this command:

$ gpg --keyserver pgp.mit.edu --recv 0x824715A0

Once installed, you can verify the download using the following command:

$ gpg --verify ReadyLisp.dmg.asc

For more information, see the Ready Lisp project page.

 

May 4, 2008

Gary KingAnnouncing: LIFT to 1.4.3

Version 1.4.3 of LIFT adds some declare ignorable in the with-test-slots macro to keep the LispWorks compiler from complaining. It’s a good thing.

 

Ingvar Mattsson4 May 2008

I had planned on doing a quick web-related project, to get started with Hunchentoot, since it seems to be the lisp- based web server of preference. Unfortunately, I seem to be made from fail. I can get it to listen to a port, but I cannot, however much I try, make it send anything back.

The docs indicate taht if I start the server and then point a browser at it, I should get a short info page back. Unfortunately, all I get is "nothing". I've even whipped out a Wireshark to confirm. Ah, well, maybe it'll be obvious, at some point.

Of course, trouble-shooting this is a bit more difficult than it could be, since it's a threaded server and it's harder than "trivial" to get tracing and the like to work.

As and when I get a bit more round tuits, I'll probably go check the hunchentoot mailing list archive, I doubt I'm the first person to run into this.

Hmm, looking at this again, the server replies and immediately closes the connection.  

Leslie PolzerFunction encapsulation in SBCL

The advice functionality allows the programmer to replace or encapsulate an existing function binding (akin to :AROUND methods in the CLOS). See Gary King’s "What is advice" for some useful links and explanations on this.

CLISP and SBCL, my favourite Common Lisp implementations, do not seem to supply this functionality. I haven’t checked whether CLISP has advice under its hood, but SBCL does, and it’s termed "Function encapsulation" there. Once you know how to do it, it’s surprisingly simple to use.

This is the definition of SB-INT:ENCAPSULATE in SBCL’s src/code/fdefinition.lisp:

;;; Replace the definition of NAME with a function that binds NAME's
;;; arguments to a variable named ARG-LIST, binds name's definition
;;; to a variable named BASIC-DEFINITION, and evaluates BODY in that
;;; context. TYPE is whatever you would like to associate with this
;;; encapsulation for identification in case you need multiple
;;; encapsulations of the same name.
(defun encapsulate (name type body)
  (let ((fdefn (fdefinition-object name nil)))
    (unless (and fdefn (fdefn-fun fdefn))
      (error 'undefined-function :name name))
    ;; We must bind and close over INFO. Consider the case where we
    ;; encapsulate (the second) an encapsulated (the first)
    ;; definition, and later someone unencapsulates the encapsulated
    ;; (first) definition. We don't want our encapsulation (second) to
    ;; bind basic-definition to the encapsulated (first) definition
    ;; when it no longer exists. When unencapsulating, we make sure to
    ;; clobber the appropriate INFO structure to allow
    ;; basic-definition to be bound to the next definition instead of
    ;; an encapsulation that no longer exists.
    (let ((info (make-encapsulation-info type (fdefn-fun fdefn))))
      (setf (fdefn-fun fdefn)
            (named-lambda encapsulation (&rest arg-list)
              (declare (special arg-list))
              (let ((basic-definition (encapsulation-info-definition info)))
                (declare (special basic-definition))
                (eval body)))))))

From this we can figure out that basic advice can be gotten from SBCL like this:

CL-USER(19): (defun gorm (x) (1+ x))
GORM
 
CL-USER(20): (SB-INT:ENCAPSULATE 'gorm 'identity '(apply sb-int:basic-definition sb-int:arg-list))
 
#<closure (SB-C::&OPTIONAL-DISPATCH SB-IMPL::ENCAPSULATION) {ACE428D}>
CL-USER(21): (gorm 10)
11
 
CL-USER(22): (SB-INT:ENCAPSULATE 'gorm 'add-five '(+ 5 (apply sb-int:basic-definition sb-int:arg-list)))
#</closure><closure (SB-C::&OPTIONAL-DISPATCH SB-IMPL::ENCAPSULATION) {ACEE31D}>
 
CL-USER(23): (gorm 10)
16
 
CL-USER(24): (SB-INT:ENCAPSULATE 'gorm 'add-seven '(+ 7 (apply sb-int:basic-definition sb-int:arg-list)))
 
#</closure><closure (SB-C::&OPTIONAL-DISPATCH SB-IMPL::ENCAPSULATION) {ACF84ED}>
CL-USER(25): (gorm)
23
</closure>

We have built a pretty onion, each layer of which calls the next inner layer via BASIC-DEFINITION (the analog to CALL-NEXT-METHOD). Let’s peel specific layers from it:

CL-USER(26): (SB-INT:UNENCAPSULATE 'gorm 'add-five)
T
 
CL-USER(27): (gorm 10)
18
 
CL-USER(28): (SB-INT:UNENCAPSULATE 'gorm 'add-seven)
T
 
CL-USER(29): (gorm 10)
11
 

May 3, 2008

Liam HealySBCL in Debian testing (lenny)

Recently I noticed that SBCL in Debian testing is quite old; it is currently at 0.9.16 and evidently blocked there:
=== sbcl:
= Missing build(s) on alpha,sparc
 This might need manual action from your side.
 See http://buildd.debian.org/pkg.cgi?pkg=sbcl
= No migration to testing for 585 days.
 See <http://release.debian.org/migration/testing.pl?package=sbcl>

I have tried to stir the pot, and there has been some progress. This version of testing will release as lenny in the fall, and it would be a shame if it released with such an old version of SBCL.  

May 2, 2008

Nick LevineIt's so neat

The Customer wanted to an option to simplify (occasionally) the "run" by dropping one of the tests through which the data has to pass before going on to the next stage. Hmm, I asked, given this is in an innermost loop for almost all the data are you willing to take a mild performance hit so I can check each time around whether to apply this test? We'd rather not, said the Customer.

This innermost test is a hash-table lookup. The solution is so neat:
(make-hash-table :test 'true :hash-function (constantly 0))
and setf that (at the beginning of the run, outside all those loops) on top of the real table.

Thankyou lisp. (Hey, I get paid to do this stuff).  

Gary KingParallel computing and DSLs

(finally, something I can claim is about Lisp!)

Peter Bright over at Ars Technica:

Parallel computing is hard.

and goes on to talk about industry/acedemic ventures happening that hope to address the growing mismatch between our (collective) programming skillz and our (clustered) computation engines. Standford’s project (along with Sun, AMD, NVIDIA, IBM, Intel, and HP) is the Pervasive Parallelism Lab and one of their aims is to develop a “set of ‘domain-specific languages’ (DSLs).” Hmmm, I think I’ve heard about those being a central feature of own of my favorite languages. :)

There is a lot of energy around Erlang/Lisp right now; I hope that Lisp (both in general and Common Lisp in particular) can win big in the years to come. let’s hope that Standford’s DSL’s are our kind of PPL. (this pun might work better if said our loud: people, ppl, get it. sigh.)

 

May 1, 2008

Zach BeaneLinks and meetings

Here are a few Lisp-related links I liked in April:

And here are the coming Lisp-related meetings in May I know about:

If you're having a meeting in May and I didn't list it, let me know.  

Nick LevineWalter Sawyer died

For a couple of years after we moved into Cambridge, our neighbour was W. W. Sawyer, mathematician, Professor Emeritus, author of several popular books, and general thorn (for his continued criticism of "modern maths") in the educational establishment's side.

I heard a couple of days ago that Walter died in February. He was 96. He had been living with his daughter in Toronto and them more recently moved to a nursing home there. I got reports from time to time. He was still sharp, still playing chess.

Searching for him on the web today, there's less than I would have expected about the man himself, given the influence he had on people I've mentioned him to. There's a good biography here along with a number of his papers; one of his books (Prelude to Mathematics) is currently available online.

 

Leslie PolzerA kind of magic

One often has to checking file uploads for correctness, for example with respect to size, file type or file name.

Here’s a sketch for checking the type of image files:

(defun matches-magic (file magic &optional (offset 0))
    (with-open-file (s file :element-type '(unsigned-byte 8))
      (file-position s offset)
      (loop for c across magic
            unless (eql c (code-char (read-byte s)))
            do (return-from matches-magic))
      t))
 
(defun jpeg-p (file) ; won't catch Exif files with JPEG inside
  (matches-magic file "JFIF" 6))
 
(defun png-p (file)
  (matches-magic file "PNG" 1))
 
(defun gif-p (file)
  (matches-magic file "GIF89a"))
 
(defun canonical-image-extension (file)
  (cond
    ((png-p file) "png")
    ((gif-p file) "gif")
    ((jpeg-p file) "jpeg")))
 
(defmacro any-predicate (preds &rest args)
    `(or ,@(loop for p in preds
                 collect `(,p ,@args))))
 
(defun valid-image-p (file)
  (any-predicate (jpeg-p png-p gif-p) file))

ANY-PREDICATE could also be written as a function (with a slightly different form of arguments), here’s another quick draft:

(defun any-predicate (preds &rest args) ; largely untested
  (some #'identity (mapcar (lambda (x) (apply x args))  preds)))
 
(defun valid-image-p (file)
  (any-predicate (list #'jpeg-p #'png-p #'gif-p #'exif-p) file))

Of course, you could also chain SYMBOL-FUNCTION to get rid of the sharp-signs in the call. Whatever suits you.
I like the macro better, though, since it’s clearer and probably more efficient. Update: see below for a comment by Zach Beane on this.

Homework would be writing a simple DSL to jot down file type data:

(define-file-type "jpeg" JFIF 6)

Alternatives from the outer world would be calling file(1) or parsing magic(4).

 

April 30, 2008

Holger SchauerUCW leadership change

In case you're interested in continuation-based web development with Common Lisp, it might be of interest to learn that Drew Campsie has voluntered to take over UCW development. Marco Baringer, initiator and maintainer of UncommonWeb (aka UCW) seemed somwhat relieved that finally somebody with more time on his hand steps up to take over.

That Marco hadn't had the time to keep pushing UCW is quite obvious from the mailing list archive. Lately Attila Lendvai seemed to me (an innocent external observer) to had pushed UCW foreward, however, as he Attila announced, he wants to go into a different direction than Drew.

I've been using ucw-dev during the last two years and the current situation really called for a change, in my opinion. UCWs main problem, as I observe it, is lacking documentation and clearly stated goals which way UCW will go. The automatically extracted documentation doesn't contain information how the bits fit together and all tutorials are a) incomplete, b) outdated as far as they are targetting ucw-dev, whereas development has mainly happened in ucw-ajax. However, it was not all clear that ucw-ajax really would become the main line of UCW.

I like UCWs component architecture a lot and will eagerly follow what is going to happen. With two prominent developers such as Marco and Attila more or less leaving the project, it will be interesting to see with how much activity development will happen from now on.
 

April 29, 2008

Zach BeaneYou're doing it wrong

Last year, while creating AutoMotivator, I threw together a picture of John McCarthy with fake inspirational text. To my surprise, a lot of people really liked it.

I recently extended the software to create bigger AutoMotivator images for printing. Here are a few of the test images (click for full size):


Suitable for printing at US letter size


Suitable for printing at A4 size

Print one out on your company laser printer and hang it on your cubicle wall. Extra motivation guaranteed or your money back.  

François-René RideauNext Boston Lisp Meeting: Tuesday May 27th 2008, 6pm at MIT 34-401B

ITA Software, a fine employer of Lisp hackers (full disclosure: I work there), has kindly offered to sponsor a dinner for our Monthly Boston Lisp Meeting. Please send mail to boston-lisp-meeting-register at common-lisp.net with a list of attendees so we may order the correct amount of food.

Ivan Krstić will give a 25' talk about Security and Programming Languages. Ivan Krstić http://radian.org/ is notably the prized author of Bitfrost, the security architecture for the OLPC XO laptop.

Greg Cooper will give a 50' talk about FrTime: A Dataflow Extension of DrScheme. Dataflow programming extends functional programming with time-varying values called signals. Signals provide a simple, declarative mechanism for expressing event-driven programs without callbacks or explicit side-effects. This talk will present FrTime, an extension of PLT Scheme with dataflow evaluation. The language's distinguishing features include an event-driven evaluation model, transparent reuse of Scheme code, support for reactive data structures, and integration with the DrScheme programming environment. The talk will include a demonstration of the language and programming environment, along with a discussion of the key design decisions and main ideas underlying the implementation strategy. Greg Cooper developed FrTime while he was a graduate student at Brown University, working with Shriram Krishnamurthi. He now works for ITA Software.

Please note that the meeting is taking place at an unusual date, to accommodate for the availability of our main speaker.

The Lisp Meeting with take place at MIT, room 34-401B. As the numbers indicate, this is in Building 34, on the 4th floor.

MIT map: http://whereis.mit.edu/bin/map?selection=34

Google map: http://maps.google.com/maps?q=50+Vassar+St,+Cambridge,+MA+02139,+USA

PS: The previous Boston Lisp Meeting on April 22nd was a success with 40 participants, despite a few organizational glitches for which I apologize. Thanks a lot to all those who came. I hope we'll meet again and have more of those interesting conversations.

PPS: We're still looking for speakers. We have a lot of potential speakers, but not enough confirmed speakers at scheduled dates. The call for speakers and all the other details are at http://fare.livejournal.com/120393.html

PPPS: Please forward this information to people who would be interested. Please accept my apologies for your receiving this message multiple times.

For more information, see our new web site boston-lisp.org. For posts related to the Boston Lisp meetings in general, follow this link: http://fare.livejournal.com/tag/boston-lisp-meeting or subscribe to our RSS feed: http://fare.livejournal.com/data/rss?tag=boston-lisp-meeting

 

Zach BeaneI broke slime

If you use a pre-1.0.8 SBCL with CVS SLIME, it won't work.

When I'm trying to debug functions, I often add (declare (optimize (debug 3))) to the top and then use "v" in sldb to jump directly to the location of the error in the source. Then I take out the declare bit after I'm done troubleshooting.

To automate this, I added an option to compilation that uses the new experimental SB-EXT:RESTRICT-COMPILER-POLICY function to compile with maximum debug. That is, C-u C-c C-c will recompile individual forms at debug 3. You don't have to manually add and remove it, and you don't have to run your whole system with elevated debug.

It only works on SBCL 1.0.8 and newer, though. I think it'd be handy for all backends, but I don't know how to add it.  

April 28, 2008

Zach BeaneSBCL leadership change

In 2000, William Harold Newman announced his new Lisp project, SBCL, to comp.lang.lisp:

I'm working on SBCL,
        http://sbcl.sourceforge.net/
a variant of CMU CL which should be easier to
maintain than CMU CL. Some of the changes involved -- especially
making the system bootstrap itself cleanly -- involved major surgery
on the CMU CL code base, not just local patches, and the result is a
distinct version of the system.

I consider the new release of SBCL (sbcl-0.6.0) to be alpha quality,
unstable but fairly usable. 

For eight years he's been working on SBCL and making new releases every month. But a few weeks ago, he stepped down:

I'd like to resign as project administrator. This decision is not affected by sbcl-1.1 release issues: I've been absently thinking about it for months, and actively for the past month or so. The decision is hardly affected by anything about the SBCL project at all, in fact. Other things in my life have chewed up a lot of my energy and motivation, and it is far harder for me to deal with any extra responsibilities than it used to to be, and SBCL is just one of the casualties.

Christophe Rhodes, Juho Snellman, and Nikodemus Siivola will now manage SBCL releases.

I started using CL and SBCL after Bill's busiest period of SBCL hacking, but he paved the way and made it easy for new people to become vital developers of the system. The CREDITS file lists nearly 60 contributors of great and small features and fixes.

Thanks for creating SBCL, Bill!  

April 27, 2008

Geoff WozniakGetting structure from behaviour

Suppose you wrote the following (contrived) code snippet and tried to evaluate it.

(let ((point (make-instance 'point :xval 1 :yval 2)))
  (print (xval point))
  (print (yval point)))

Without any definition for the class point or function values associated with xval or yval, it will fail. But can you learn something useful from trying to run it? (Maybe even get it to print 1 2?)

Looking at the code, it's reasonable (but not necessarily correct!) to think that is says something like, "I want to make an instance of a class called point. Assume it has two slots called xval and yval because I tend to name the slots in accordance with the initialization arguments. And since I tend to associate accessor functions with symbols bearing the same name as the slots, the calls to xval and yval can be considered accessors."

There are an awful lot of assumptions being made here based on conventions, but in my experience, it's not really that far from the intent. So, what if the system could take advantage of these kinds of conventions to help build structural definitions based on the use of the code? That is something I find really interesting and is what I've been working on for the last little while (in addition to that whole teaching thing).

The goal I set out for myself was to build class definitions dynamically based on the use of the code. Essentially, I want to watch the execution of the code to make some reasonable guesses as to what the classes look like without having to define them ahead of time. It's a bit like programming by example, except I'm not interested in generalizing behaviour; instead, I want to derive structure from behaviour.

This presents a bit of a chicken-and-egg problem: How can you run code that relies on structure without having that structure? One answer is to make the structure as you go along. This means harvesting information from errors while taking advantage of convention and adjusting the definition of correctness.

The last point may be cringe-inducing, but is pragmatically necessary. Getting such code to be correct means defining what "correct" even means in this context. Usually, correct is taken to mean "produces the desired output". Here, the desired output is not what the program would produce given a suitable structure, but a suitable structure itself that may lead to the desired output.

Thus, we trust that what is being said as what is intended. Nothing is to be verified because we have nothing to verify against. All that can be done is to present a structure that in turn is verified by some external agent, such as the developer.

This is really a search through a very large space given some information to go on. The output produced can be put into the program text to help refine the program, but the idea of deriving structure in this manner should not be seen as a kind of operational semantics in which production code can be run on. There are just too many unknowns.

A reasonable question at this point is, "Why bother?" First and foremost, I think it's an interesting challenge and I'm curious to see how far it can be taken. More practically, if you could get reasonable structural definitions from somewhat complicated behavioural descriptions, it might be interesting to incorporate it with testing in some fashion, such as writing tests to produce programs (or significant parts of programs) that pass those tests.

This stems from my annoyance with having to write and amend structural definitions all the time during program development. (This is basically why I stopped using OCaml and its ilk in my day-to-day work.) When writing programs I tend to specify the behaviour first, then fill in the structure because the act of figuring out what I want the program to do helps me figure out what the data should look like.

Over the next few days, I'll be posting more about this, including how I've gotten it to work in a somewhat limited (although mildly interesting) context. It's just too much to put in one post. :)

 

April 26, 2008

Geoff WozniakDone with teaching!

I spent the last three and a half months teaching a programming languages course and working on my dissertation, which is why I haven't posted much here of late. The worst part about the teaching job was that it was in a different city than my home and had an awkward schedule, so I had to commute and stay at someone else's house for the majority of the week. It also meant I couldn't make any of the Toronto Lisp Meetings.

I used Shriram Krishnamurthi's text Programming Languages: Application and Interpretation for the course, which seemed to go over well, despite difficulties. The students coming into the course are mostly exposed only to Java and C/C++, so Scheme tends to break a few brains at the beginning. I'd like to say the whole class had a firm grasp on Scheme when it was over, but I can't.

For the most part, students could write Scheme code, but it was mostly by translating (not very good) Java to Scheme. I'm not sure why. For example, there was an awful lot of this in assignment submissions:

(if (predicate? foo)
    (begin
      (set! ret-val (compute-something ...))
      ret-val)
    ...)

To say that I stressed the idea of expressions evaluating to a value and avoiding mutation in my lectures, labs and through assignment feedback would be an understatement. I'm not sure why this mindset persisted, but I hypothesize it has something to do with wanting to make the simple complex. Or maybe it has something to do with not seeing the forest for the trees. I really don't know.

I had a difficult time getting across the idea of trusting an abstraction, even when comparing to something like the APIs used frequenty in the Java libraries. For example, many students just couldn't deal with continuations without knowing what happened inside them. One even remarked he wished it had a toString method. It took a lot of convincing that seeing #<continuation> in DrScheme was basically the same thing.

Student: "But I don't know what's happening inside it!"

Me: "That's the point: it doesn't really matter."

The same symptom could be seen in trying to understand functions as values, except that I flogged the notion of closures so much that some students' brains couldn't help but give up and understand it. :)

Another thing that struck me was how, after explaining macros in Scheme, most tried to write macros as you would in Common Lisp. I expect its because CL macros let you manipulate the code directly, which fits with the theme of getting at the guts of something. Maybe there's something deeper at work here, but I'm not going to go wild in my conjectures based on this small sample.

After this class, I'm more convinced that starting with something like Scheme followed by Java would be an easier transition than the other way around. Whether starting with Scheme is easier than Java is another question entirely.

Oddly enough, they seemed to grasp Prolog pretty well...

 

April 24, 2008

Zach BeaneApril Boston Lisp meeting

After two failed attempts, I finally made it to the Boston Lisp meeting on Tuesday. I enjoyed the two talks.

ACL2 is a theorem-proving system, and people have used Emacs to drive it for many years. To help people learn ACL2 without learning Emacs at the same time, Peter Dillinger worked on ACL2s, an Eclipse-based interface to ACL2. In his slides he compared the original Emacs interface to a racecar: fast, highly effective in the hands of experienced users, and unforgiving of errors. ACL2s is designed to be easier to learn and more forgiving, like a family sedan.

This reminds me of Cusp, though the original Emacs interface for ACL2 seems more primitive than SLIME. Most of the questions were about with the capabilities of ACL2 rather than the ACL2s interface. I was more interested in how easy it is to extend ACL2s than what was involved in proving theorems. Does it involve writing more Java, or more Lisp? Or something else? I didn't ask Peter about it, though.

Part of the Q&A drew a laugh:

Q: Why not add these advanced features to the Emacs ACL2 interface?

A: Well...um...Eclipse is designed to be extensible, for programming stuff...

Q: ...

Wag: Just say Eclipse can do multiple threads!

A: Multiple threads! ACL2s is using four threads in this demo!

Hans's talk about the BKNR data store was interesting. It's a simple, tested, documented system for object persistence. It was created to avoid the need to install, maintain, and tune a relational database system, and figure out some way to map relational tables to Lisp objects; instead, it's standard Common Lisp all the way down.

The design decisions have worked out well for Hans's applications:

Hans was clear that this is not the perfect solution for all purposes. This setup can fit the needs of many applications, though. Hans demonstrated a website with over 400,000 objects; it took up about 50 megabytes on disk and about triple that in memory. Initializing the objects at startup took a minute or two, but, as Hans noted, "Lisp hackers are used to not restarting very often."

The timing for his talk was good for me. A few weeks ago I extended AutoMotivator to have a user and poster management system, and I wrote a simple persistence system from scratch. Although it was fun to make and was an educational first use of the MOP, challenges of storage, querying, and schema evolution have already cropped up. BKNR has the advantage of being small, mature, documented, and tested by real-world projects with hundreds of thousands of objects. I'd like to switch from my home-grown system to BKNR's storage system soon.

After the talks ITA provided pizza and Indian food, and I got to meet several Lisp hackers in person that I had only spoken with online. We wound up chatting for hours about various topics.

Though several people I know online were there, there wasn't much in the way of introductions and nobody wore much identifying material, e.g, name tags. A few people identified me by my Save Lisp and Die shirt, but I was a little too shy to go around asking people who they were.

Even though I didn't get home until 2AM Wednesday, I'd like to go to the next meeting on May 27th. You should go too!  

Mike AjemianPeculiar Circularities in Common Lisp

I just realized a peculiar circularity of Lisp. I'm building an application that modifies itself while it's running. It'll have a sister app that'll run the apps while making it easy to connect to the editor and modify the component that's running (at least that's how I'm looking at it while it's being designed. It might look different when it's done.)

A defined interface can be up and running and I'll be able to say, "Hey! This widget needs to be modified." And, it'll be a snap to open the editor for that widget and start making changes while the app's running. I don't know if the app is going to be klunky or smooth, useful or useless, but I get that part (editing a live app) and can make it work.

I know it's been done before. And I know that Lisp is the "programmable programming language". But I guess today, it's a little more awe-inspiring than it was yesterday.

Just wanted to share.  

April 23, 2008

Ties Stuijside-tracked

Yet another year went by without me having a chance to go to the European Common Lisp Meeting. And I had so much hope for this year. Finally the stars seemed to have aligned themselves in a perfect constellation: Got that Lisp job? Check! Employer willing to pay your trip? Check! Got the money to pay for your own ticket if all goes awry? Check!

But then all went haywire: I got infatuated with the OLPC XO; the thing and the concept. Ordered the thing, the process of which was so strenuous and frustrating, I don't want to write it down, because just recalling the memories is to painful. But the physical device was not enough. To make a boring story short, my obsession brought me here, typing away on an XO in Kathmandu, volunteering for OLE Nepal. Yes, once again shamelessly copying Luke Gorrie, while still working for my ultracool CL job at Stix.to.

And while the whole of OLPC seems to fret and spirals into a depression over leaving team-members and rumours over XP (both concerns being quite unfounded as far as I can judge), we're totally psyched and upbeat about our pilot-launch on Friday in two rural schools.

Programmatically I've strayed of the one true path. Which is primarily the reason for this post. I have sinned my brothers. At OLE we're developing curriculum complimentary activities in Squeak, the Sugar stuff is in Python and the firmware is in Forth. And actually it's great fun learning a few new languages which you can put into direct use in a useful way. Forth being the most useless for our purposes, but which is by far the coolest of the bunch. And the Luke copying continues...

Here we're coming to the practical conclusion of this post. I won't bog you lispers down with posts about Nepal and OLPC and unworthy programming languages. I made another blog to harbour those kinds of things. The posts there tend to be a bit more structured and as of yet half of them are recycled from other places around the net. As of yet only one of them is about developing, and it's quite high level at that. But then again, what do you programming gods care about mortal wiles.  

Mike AjemianOne Step Up and a Tangent

While building the interface for the web site code, I saw duplication in the layout and the functional core. So, I stepped back, squinted and saw a pattern emerge.

It was a familiar pattern, one that I'd made an attempt to address a while ago. It has to do with being able to build, test and release interfaces easily and quickly.

But the first few attempts I made at building an interface to build interfaces ran into walls. Either I was too immersed in some other problem to fully comprehend the scope or didn't give it the time it required to spec it out. More likely, I just didn't get it yet.

Whichever it was, this time I'm pretty sure I've simplified everything just enough to build a good interface builder for my way of working with Hemlock. I can't say graphically what it is or how it will work yet. Sat down and worked out the core on paper and will be putting it together starting today. Will post an update this weekend.

In most of these posts, I've been pushing projects onto a stack. What I've been looking for is a way to work faster so the projects don't stack up, but flow out. My hope and (pretty firm) belief is that if the interface builder works, then I'll be able to start popping projects off the stack. And that would be quite a relief.

I'm not sure if anybody else would even consider working the way I do - it's odd. But, if it makes it possible for me to release code, I guess that's a good thing in it's own right.  


For older items, see the Planet Lisp Archives.


Last updated: May 8, 2008 03:33 PM