vindarelOh no, I started a Magit-like plugin for the Lem editor

· 4 days ago

Lem is an awesome project. It’s an editor buit in Common Lisp, ready to use out of the box for Common Lisp, that supports more languages and modes (Python, Rust, Elixir, Go, JavaScript, TypeScript, Haskell, Java, Nim, Dart, OCaml, Scala, Swift, shell, asm, but also markdown, ascii, JSON, HTML and CSS, SQL...) thanks to, in part, its built-in LSP support.

I took the challenge to add an interactive interface for Git, à la Magit, because you know, despite all its features (good vim mode, project-aware commands, grep, file tree view and directory mode, multiple cursors, tabs...), there’s so much an editor should do to be useful all day long.

Now, for a Git project (and to a lower extent, Fossil and Mercurial ones) we can see its status, stage changes, commit, push & pull, start an interactive rebase...

I like the shape it is taking, and frankly, what I have been able to assemble in a continuously successful hack session is a tribute to what @cxxxr and the early contributors built. Lem’s codebase is easily explorable (more so in Lem itself of course, think Emacs in steroïds with greater Common Lisp power), clear, and fun. Come to the Discord or watch the repository and see how new contributors easily add new features.

I didn’t even have to build an UI interface, fortunately. I started with the built-in interactive grep mode, and built from there.

Enough talk, what can we do with Lem/legit as of today? After that, we’ll discuss some implementation details.

Disclaimer: there’s room for collaboration ;)

Table of Contents

Lem/legit - manual

NOTE: you'd better read the latest manual on Lem's repository: https://github.com/lem-project/lem/blob/main/extensions/legit/README.md

legit’s main focus is to support Git operations, but it also has preliminary support for other VCSs (Fossil, Mercurial).

We can currently open a status window, stage and unstage files or diff hunks, commit our changes or again start an interactive rebase.

Its main source of inspiration is, obviously, Magit.

Status

legit is in development. It isn’t finished nor complete nor at feature parity with Magit nor suitable for mission-critical work. Use at your own risk.

However it should run a few operations smoothly.

Load

legit is built into Lem but it isn’t loaded by default. To load it, open a Lisp REPL (M-x start-lisp-repl) or evaluate Lisp code (M-:) and type:

(ql:quickload "lem/legit")

Now you can start it with C-x g or M-x legit-status.

Help

Press ? or C-x ? to call legit-help.

M-x legit-status

The status windows show us, on the left:

  • the current branch
  • the untracked files.
  • the unstaged changes, staged changes,
  • the latest commits.

It also warns us if a rebase is in process.

and the window on the right shows us the file diffs or the commits’ content.

Refresh the status content with g.

We can navigate inside legit windows with n, p, M-n and M-p (go to next/previous section).

To change windows, use the usual M-o key from Lem.

Quit with q or C-x 0 (zero).

Stage or unstage files, diff hunks (s, u)

Stage changes with “s”.

When your cursor is on an Unstaged change file, you can see the file changes on the right, and you can stage the whole file with s.

You can also go to the diff window on the right, navigate the diff hunks with n and p and stage a hunk with s.

Unstage a change with u.

Discard changes to a file

Use k. Be careful, you can loose your changes.

Commit

Pressing c opens a new buffer where you can write your commit message.

Validate with C-c C-c and quit with M-q (or C-c C-k).

Branches, push, pull

Checkout a branch with b b (“b” followed by another “b”).

Create a new branch with b c.

You can push to the current remote branch with P p and pull changes (fetch) with F p.

NOTE: after pressing "P" or "F", you will not see an intermediate window giving you choices. Just press "P p" one after the other.

Interactive rebase

You can start a Git interactive rebase. Place the cursor on a commit you want to rebase from, and press r i.

You will be dropped into the classical Git rebase file, that presents you commits and an action to apply on them: pick the commit, drop it, fixup, edit, reword, squash...

For example:

pick 26b3990f the following commit
pick 499ba39d some commit

# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

legit binds keys to the rebase actions:

  • use p to “pick” the commit (the default)
  • f to fixup

and so on.

Validate anytime with C-c C-c and abort with C-c C-k.

NOTE: at the time of writing, "reword" and "edit" are not supported.
NOTE: the interactive rebase is currently Unix only. This is due to the short shell script we use to control the Git process. Come join us if you know how to "trap some-fn SIGTERM" for Windows plateforms.

Abort, continue, skip

In any legit window, type r a to abort a rebase process (if it was started by you inside Lem or by another process), r c to call git rebase --continue and r s to call git rebase --skip.

Fossil

We have basic Fossil support: see current branch, add change, commit.

Mercurial

We have basic Mercurial support.

Customization

In the lem/porcelain package:

  • *git-base-arglist*: the Git program, to be appended command-line options. Defaults to (list "git").

=> you can change the default call to the git binary.

Same with *fossil-base-args* and *hg-base-arglist* (oops, a name mismatch).

  • *nb-latest-commits*: defaults to 10
  • *branch-sort-by*: when listing branches, sort by this field name. Prefix with “-” to sort in descending order. Defaults to “-creatordate”, to list the latest used branches first.
  • *file-diff-args*: defaults to (list "diff" "--no-color"). Arguments to display the file diffs. Will be surrounded by the git binary and the file path. For staged files, –cached is added by the command.

If a project is managed by more than one VCS, legit takes the first VCS defined in *vcs-existence-order*:

(defvar *vcs-existence-order*
  '(
    git-project-p
    fossil-project-p
    hg-project-p
    ))

where these symbols are functions with no arguments that return two values: a truthy value if the current project is considered a Git/Fossil/Mercurial project, and a keyword representing the VCS: :git, :fossil, :hg.

For example:

(defun hg-project-p ()
  "Return t if we find a .hg/ directory in the current directory (which should be the project root. Use `lem/legit::with-current-project`)."
  (values (uiop:directory-exists-p ".hg")
          :hg))

Variables and parameters in the lem/legit package. They might not be exported.

  • *legit-verbose*: If non nil, print some logs on standard output (terminal) and create the hunk patch file on disk at (lem-home)/lem-hunk-latest.patch.

=> to help debugging

see sources in /extensions/legit/

Implementation details

Calls

Repository data is retrieved with calls to the VCS binary. We have a POC to read some data directly from the Git objects (proactively looking for best efficiency) using cl-git.

Basically, we get Git status data with git status --porcelain=v1. This outputs something like:

 A project/settings.lisp
 M project/api.lisp
?? project/search/datasources

we output this a to a string and we parse it.

Interactive rebase

The interactive rebase currently uses a Unix-only shell script.

When you run git rebase --interactive, the Git program creates a special file in .git/rebase-merge/git-rebase-merge-todo, opens it with your $EDITOR in the terminal, lets you edit it (change a “pick” to “fixup”, “reword” etc), and on exit it interprets the file and runs the required Git operations. What we want is to not use Git’s default program, edit the file with Lem and our special Legit mode that binds keys for quick actions (press “f” for “fixup” etc). So we bind the shell’s $EDITOR to a dummy editor, this shell script:

function ok {
    exit 0
}

trap ok SIGTERM
echo "dumbrebaseeditor_pid:$$"

while :
do
        sleep 0.1
done

This script doesn’t simulate an editor, it waits, so we can edit the rebase file with Lem, but this script catches a SIGTERM signal and exits successfully, so git-rebase is happy and terminates the rebase and all is well.

But that’s Unix only.

On that matter Magit seems to be doing black magic.

The basic function to write content to a buffer is

(insert-string point s :read-only t)

And this is how you make actionable links:

(put-text-property start end :visit-file-function function))

where :visit-file-function is any keyword you want, and function is any lambda function you want. So, how to make any link useful? Create a lambda, make it close over any variables you want, “store” it in a link, and later on read the attribute at point with

(text-property-at point :visit-file-function)

where point can be (buffer-point (window-buffer *my-window*)) for instance.

Now create a mode, add keybindings and you’re ready to go.

;; Legit diff mode: view diffs.
;; We use the existing patch-mode and supercharge it with our keys.
(define-major-mode legit-diff-mode lem-patch-mode:patch-mode
    (:name "legit-diff"
     :syntax-table lem-patch-mode::*patch-syntax-table*
     :keymap *legit-diff-mode-keymap*)
  (setf (variable-value 'enable-syntax-highlight) t))

;; git commands.
;; Some are defined on peek-legit too.
(define-key *global-keymap* "C-x g" 'legit-status)

or a minor mode:

(define-minor-mode peek-legit-mode
    (:name "Peek"
     :keymap *peek-legit-keymap*)
  (setf (not-switchable-buffer-p (current-buffer)) t))

;; Git commands
;; Some are defined on legit.lisp for this keymap too.
(define-key *peek-legit-keymap* "s" 'peek-legit-stage-file)
(define-key *peek-legit-keymap* "u" 'peek-legit-unstage-file)
(define-key *peek-legit-keymap* "k" 'peek-legit-discard-file)

TODOs

Much needs to be done, if only to have a better discoverable UX.

First:

  • interactive rebase: support reword, edit.
  • show renamed files

and then:

  • visual submenu to pick subcommands
  • view log
  • stage only selected region (more precise than hunks)
  • unstage/stage/discard multiple files
  • many, many more commands, settings and switches
  • mouse context menus

Closing words

You’ll be surprised by all Lem’s features and how easy it is to add features.

I believe it doesn’t make much sense to “port Magit to Lem”. The UIs are different, the text displaying mechanism is different, etc. It’s faster to re-implement the required functionality, without the cruft. And look, I started, it’s possible.

But, sad me, I didn’t plan to be involved in yet another side project, as cool and motivating as it might be :S


Marco AntoniottiA Rant about R and Python (and anything with a-lists, tuples and "dictionaries").

· 15 days ago

R and Python come from Lisp (R for sure, Python will deny it). Early Lisp. Even before the first edition of "AI Programming" by Charniak, Riesbek and McDermott.

At that time, there were a-lists and p-lists. Then Charniak, Riesbeck and McDermott taught us (not only them of course) to create records in Lisp. You know... those things like:

    DECLARE
      1 STUDENT,
        2 NAME     CHAR (30),
        2 SURNAME  CHAR (50),
        2 ID FIXED DECIMAL (8),
        2 ADDRESS,
          3 STREET  CHAR (80),
          3 NUMBER  FIXED DECIMAL (5),
          3 CITY    CHAR (80),
          3 PRST    CHAR (20),
          3 ZIP     CHAR (10),
          3 COUNTRY CHAR (80);

Using a-lists the above may become:

    (defvar student '((name . "John")
                      (surname . "Blutarski")
                      (id . 42)
                      (address . ((street . "United State Senate")
                                  (number . 0)
                                  (city . "Washington")
                                  (prst . "DC")
                                  (zip . "20510")
                                  (country . "U.S.A.")))
                      ))

In Python you can do the following.

    student = {}    # a 'Dict'; i.e., a hash table.
    student['name'] = "John"
    student['surname'] = "Blutarski"
    student['id'] = 42
    student['address'] = {}
    student['address']['street'] = "United State Senate"
    student['address']['number'] = 0
    student['address']['city'] = "Washington"
    student['address']['prst'] = "DC"
    student['address']['zip'] = "20510"
    student['address']['country'] = "U.S.A."

Not that you must, but surely you can; and this, as in the case of R below, is the root of my rant.

In R you use lists; a misnomer for something that is essentially a dictionary like in Python, patterned, ça va sans dire, after a-lists.

    student = list()
    student$name = "John"
    student$surname = "Blutarsky"
    student$id = 42
    student$address = list()
    student$address$street = "United State Senate"
    student$address$number = 0
    student$address$city = "Washington"
    student$address$prst = "DC"
    student$address$zip = "20150"
    student$address$country = "U.S.A."

This of course gives you a lot of flexibility; e.g., if - in the middle of your code - you need to deal with the student's nickname, you just write the following.

In (Common) Lisp:

    (defun tracking-student ()
        ...
        ...
        (setf student (acons 'nickname "Bluto" student))
        ...
        )

In Python:

    def tracking_student():
        ...
        ...
        student['nickname'] = "Bluto"
        ...

In R:

    tracking_student <- function() {
        ...
        ...
        student$nickname = "Bluto"
        student <<- student     # Yes, R scoping rules are ... interesting.
        ...
    }

The example is relatively simple and innocuous, but it has some consequences when you have to actually read some code.

The problem I have (it may be just me) is that this programming style does not give me an overview of what a data structure (a record) actually is. The flexibility that this style allows for is usually not accompanied by the necessary documentation effort telling the reader what is going into an "object". The reader is therefore left wondering, while taking copious notes about what is what and where.

Bottom line: don't do that. Use defstruct and defclass in CL, classes in Python, struct in Julia, etc. etc. etc. In R, please document your stuff.

You may feel that your code is less malleable, but, in the end it becomes easier to read. At least for me. Sorry.


(cheers)

Eugene ZaikonnikovDeannoyifying SLIME

· 21 days ago

Over all these years I don't think I ever wanted to close all existing SLIME connections when attaching to a remote host. Similarly, the version mismatch between SWANK and SLIME frontend has never stopped me following through with connection. I did however fumbled with y/n confirmations plenty of times. The snippet below removes these interactive checks.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
(defun slime-check-version (version conn)
  (or (equal version slime-protocol-version)
      (equal slime-protocol-version 'ignore)
      (message
       (format "Versions differ: %s (slime) vs. %s (swank)."
               slime-protocol-version version))
      (slime-net-close conn)
      (top-level)))

(defun slime-connect (host port &optional _coding-system interactive-p &rest parameters)
  "Connect to a running Swank server. Return the connection."
  (interactive (list (read-from-minibuffer
                      "Host: " (cl-first slime-connect-host-history)
                      nil nil '(slime-connect-host-history . 1))
                     (string-to-number
                      (read-from-minibuffer
                       "Port: " (cl-first slime-connect-port-history)
                       nil nil '(slime-connect-port-history . 1)))
                     nil t))
  (slime-setup)
  (message "Connecting to Swank on port %S.." port)
  (slime-setup-connection (apply 'slime-net-connect host port parameters)))

A useful feature for working on *nix backends is an ability to parse integers as epoch time in SLIME inspector. For this we need to minimally extend emacs-inspect found in swank-fancy-inspector.lisp:

1
2
3
4
5
6
7
8
9
10
11
12
13
(defmethod emacs-inspect ((i integer))
  (append
   `(,(format nil "Value: ~D = #x~8,'0X = #o~O = #b~,,' ,8:B~@[ = ~E~]"
	      i i i i (ignore-errors (coerce i 'float)))
     (:newline))
   (when (< -1 i char-code-limit)
     (label-value-line "Code-char" (code-char i)))
   (label-value-line "Integer-length" (integer-length i))
   (ignore-errors
     (label-value-line "Universal-time" (format-iso8601-time i t)))
   (ignore-errors
     (label-value-line "Epoch-time"
		       (format-iso8601-time (+ i 2208988800) t)))))

Paolo AmorosoReading A Programmer's Guide to COMMON LISP

· 25 days ago

I got a cheap used copy of the book A Programmer's Guide to COMMON LISP by Deborah G. Tatar, Digital Press, 1987.

The book A Programmer's Guide to COMMON LISP by Deborah G.

Why did I read such an old book, published a few years after CLtL1 and well before ANSI finalized the Common Lisp standard?

I'm always looking for good Lisp books. Since Medley is my primary Lisp environment, I'm particularly interested in books published when the system was originally developed and used. These works are relevant because they cover a set of features close to the state of the Common Lisp implementation of Medley, and present a programming style typical of Lisp development in those years.

Two old reviews got me curious about A Programmer's Guide to COMMON LISP, one by Daniel Weinreb and the other by Richard Caruana.

Both reviews point out the book is different from most contemporary introductory Lisp books which focus on AI. Although Tatar's does contain some AI code, such as an interesting and complete toy expert system, the sample code spans a wider range of domains like a text formatter similar to nroff.

What sets A Programmer's Guide to COMMON LISP apart from other Lisp books is its environment independent discussion of the interactive Lisp programming process. Writing code in the editor, evaluating expressions from the editor, interacting with the REPL for testing expressions and exploring, and so on.

I've never seen the process expressed so clearly in any book, past of present. I'm familiar with it but the material is particulary helpful for complete beginners.

Although the short chapter on macros presents some interesting examples like a simplified version of defstruct, it doesn't discuss gensym and variable capture. This is unusual. But it's only one of a few issues and the book is a valuable addition to my Lisp library.

#CommonLisp #books #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

Paolo AmorosoImporting Common Lisp Files in Medley with TextModules

· 29 days ago

Medley is a residential environment for Interlisp and Common Lisp development.

With some effort it's possible to use Medley as a traditional file based Common Lisp environment. But in specific cases a better approach is to bring in Medley's residential environment Common Lisp sources created in file based environments.

In this post I explain the latter, i.e. how to use TextModules to import Common Lisp files into the residential environment. I go over the steps for converting an example program, the database of CD music tracks in Chapter 3 Practical: A Simple Database of Peter Seibel's book Practical Common Lisp.

Motivation

Using the tools and facilities of the residential environment, such as the File Manager, is the normal way of developing new Lisp programs. To run existing Common Lisp code you don't plan to change often, you can also use Medley as a traditional file based environment.

For existing code written in file based environments you want to use and further develop in Medley, a better option is to import the code into the residential environment and continue working from there. This is what TextModules helps to do.

What is TextModules

TextModules is a Medley tool for bringing Common Lisp sources into the residential environment and place them under the control of the File Manager. It can also do the reverse, i.e. export the File Manager descriptions and metadata to Common Lisp sources accessible from file based environments.

Using TextModules is a one time process. You run the tool once to import the code, then use and modify it with the tools and facilities of the residential environment.

The documentation of textModules starts from page 305 (page 335 of the PDF) of the Lisp Library Modules manual.

Preparing the Common Lisp files

This example involves two of the source files of Seibel's book, packages.lisp and simple-database.lisp in directory practicals-1.0.3/Chapter03 of the code archive.

Unlike CL:LOAD on Medley, TextModules doesn't require any special formatting of Common Lisp source files. For example, they don't need to begin with a semicolon character.

However, the Common Lisp implementation of Medley is incomplete and not ANSI compliant, so be sure to remove or adapt any unsupported forms. This is the case of the database example: packages.lisp makes current the package CL-USER which is missing from Medley. The fix is to substitute xcl-user for cl-user in the file, as XCL-USER is the Medley equivalent of CL-USER.

Another source of incompatibility is the LOOP macro. In Medley it's only a stub that runs an infinite loop no matter what clauses a call specifies.

To import with TextModules code that contains LOOP it would normally be necessary to replace any calls with equivalent expressions. Since this post focuses on TextModules I just use the LOOP calls intended to run an infinite loop, and ignore the others.

Running TextModules

As noted, importing with TextModules is the one time process of running the tool for every source file. Once in the residential environment, you save and manipulate the code as any other code under the File Manager.

First off, load TextModules by evaluating (FILESLOAD TEXTMODULES) at an Interlisp Exec. All its exported symbols are in package TM. Next, call the function TM:LOAD-TEXTMODULE for every Common Lisp file, which is similar to CL:LOAD with some additional processing.

Most Common Lisp programs comprise a file packages.lisp with package definitions, and a number of additional .lisp files that contain the bulk of the code. This dependency requires passing the files to TM:LOAD-TEXTMODULE in the proper order.

The file packages.lisp of the database defines the package for simple-database.lisp, so start with the former. At a Xerox Common Lisp (XCL) Exec with prompt > evaluate:

> (tm:load-textmodule "packages.lisp" :module "SIMPLEDB" :package (find-package "XCL-USER") :install t)
IL:SIMPLEDB

The only required argument is the input file packages.lisp. However, by default TM:LOAD-TEXTMODULE uses the same input file name as the name of the program for the File Manager. It wouldn't make much sense to call a database PACKAGES.LISP. A better choice is to pass the :module parameter with the more descriptive name SIMPLEDB.

The code in packages.lisp begins with an in-package form. To make sure the in-package symbol is accessible without qualifier, it should be read in a package such as XCL-USER that imports the standard Common Lisp symbols. Hence the argument :package (find-package "XCL-USER") in the call.

The argument :install t installs the definitions in the running system. Although not strictly necessary, it's useful for diagnostic purposes and because you likely want to continue working on the imported code.

Next, process simple-database.lisp by evaluating at a XCL Exec:

> (tm:load-textmodule "simple-database.lisp" :module "SIMPLEDB" :package (find-package "XCL-USER") :install t)
IL:SIMPLEDB

Again, the file begins with in-package and the reason for passing the :package argument is the same as for packages.lisp.

Saving the imported code

At this point the imported code is in the running Lisp image and the File Manager is ready to manipulate it. You can check the File Manager noticed the imported definitions by calling FILES? at an Interlisp Exec with prompt :

← (FILES?)
To be dumped:
SIMPLEDB ...changes to VARS: SIMPLEDBCOMS
                       VARIABLES: COM.GIGAMONKEYS.SIMPLE-DB::*DB*
                       FUNCTIONS: COM.GIGAMONKEYS.SIMPLE-DB::MAKE-CD, 
         COM.GIGAMONKEYS.SIMPLE-DB::ADD-RECORD, 
         COM.GIGAMONKEYS.SIMPLE-DB::DUMP-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::PROMPT-READ, 
         COM.GIGAMONKEYS.SIMPLE-DB::PROMPT-FOR-CD, 
         COM.GIGAMONKEYS.SIMPLE-DB::ADD-CDS, 
         COM.GIGAMONKEYS.SIMPLE-DB::SAVE-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::LOAD-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::CLEAR-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::SELECT, 
         COM.GIGAMONKEYS.SIMPLE-DB::WHERE, 
         COM.GIGAMONKEYS.SIMPLE-DB::MAKE-COMPARISONS-LIST, 
         COM.GIGAMONKEYS.SIMPLE-DB::MAKE-COMPARISON-EXPR, 
         COM.GIGAMONKEYS.SIMPLE-DB::UPDATE, 
         COM.GIGAMONKEYS.SIMPLE-DB::DELETE-ROWS

To save the definitions to the symbolic file SIMPLEDB call MAKEFILE from an Interlisp Exec:

← (MAKEFILE 'SIMPLEDB)
{DSK}<home>medley>il>SIMPLEDB.;1

Calling TM:LOAD-TEXTMODULE for every source file, and saving the result to a symbolic file with MAKEFILE, completes the import process.

You may terminate the session and resume later. When you're ready to proceed you can load, run, and modify the imported program as any other code under File Manager control.

Loading and running the imported code

In a new Medley session evaluate (FILESLOAD TEXTMODULES) at an Interlisp Exec. The tool must be in memory whenever you work with imported code, as TextModules sets up a special file environment and readtable the code needs to be read in.

Next, load the symbolic file of the database program by evaluating at a XCL Exec:

> (load "SIMPLEDB")

; Loading {DSK}<home>medley>il>SIMPLEDB.;1
; File created 14-Feb-2024 02:44:46
; IL:SIMPLEDBCOMS
IL:|{DSK}<home>medley>il>SIMPLEDB.;1|

One of the main entry points of the program is the function add-cds to add new records to the database, one record for each music track of a CD. A typical run from a XCL Exec looks like this:

> (setf *package* (find-package "COM.GIGAMONKEYS.SIMPLE-DB"))
#<Package COM.GIGAMONKEYS.SIMPLE-DB>
> (add-cds)
Title: Punch My Cards
Artist: The Fortrans
Rating: 6
Ripped [y/n]: n
Another? [y/n]: y
Title: Lisp n Roll
Artist: The Garbage Collectors
Rating: 8
Ripped [y/n]: y
Another? [y/n]: y
Title: Cdr Care Less
Artist: The Garbage Collectors
Rating: 7
Ripped [y/n]: y
Another? [y/n]: n
NIL

Since all the symbols of the program are in the package COM.GIGAMONKEYS.SIMPLE-DB, and none are exported, for convenience make the package current by setfing *package* as above.

I intentionally didn't rename the package or create nicknames. This is to show that Common Lisp code may be imported and used with minimal or no changes.

The program provides select and where to query the database. But where uses CL:LOOP features Medley doesn't support. To stay close to the original code, instead of modifying where you can use another function in Seibel's book. Define and call the specialized query function select-by-artist to search the database by artist:

> (defun select-by-artist (artist)
    (remove-if-not
     #'(lambda (cd) (equal (getf cd :artist) artist))
     *db*))
SELECT-BY-ARTIST
> (select-by-artist "The Garbage Collectors")
((:TITLE "Cdr Care Less" :ARTIST "The Garbage Collectors" :RATING 7 :RIPPED T) (:TITLE "Lisp n Roll" :ARTIST "The Garbage Collectors" :RATING 8 :RIPPED T))

Continuing the development

The imported code works. Now you can load, compile, run, edit, and save it as any other program developed in the residential environment under the File Manager.

You no longer need the original files packages.lisp and simple-database.lisp because you work only with SIMPLEDB. But remember to load TextModules with (FILESLOAD TEXTMODULES) in every session in which you use SIMPLEDB. It's a minor inconvenience but, for automating the task, you may add a loading command to the INIT initialization file or to a script that loads the program.

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

vindarelCelebrating 1001 learners on my Common Lisp course &#129395;

· 33 days ago

I just got 1001 learners on my Common Lisp course on Udemy. Thanks everybody for your support, here or elsewhere!

Starting with CL was honestly not easy. The first thing I did was writing the “data structures” page on the Cookbook, bewildered that it didn’t exist yet. A few years and a few projects later, this course allows me to share more, learn more, have fun, and have some rewards to keep the motivation up.

I know the course isn’t complete by any means, I want to add many chapters, both advanced topics but easier material for newcomers as well (beware, my course isn’t for total beginners in a Lisp language). The next one, and soon©, will be all about CLOS. In the meantime, I don’t abandon you, I enhance the Cookbook, I publish some videos on Youtube (last one: web development in Common Lisp, part 1 and part 2), I work on starter kits or on newcomer-friendly libraries ;)

Thanks again,

don’t hesitate to share the link with a friend or a colleague ;)

Here’s a link with a coupon until March, 13th 2024. (student? Get in touch for a free link)

Joe MarshallBilinear Fractional Transformations

· 37 days ago

A bilinear fractional transformation (BiLFT) is a two-argument function of the form

(lambda (x y)
  (/ (+ (* A x y) (* B x) (* C y) D)
     (+ (* E x y) (* F x) (* G y) H)))
William Gosper figured out how to use BiLFTs to perform linear fractional operations on infinite compositions of LFTs. It isn’t as hard as it might seem.

BiLFTs do not form a group under functional composition with LFTs, but they have this interesting property: if you hold either input to the BiLFT constant, the BiLFT becomes a LFT on the other input. So if we consider just one of the inputs at a time (or the output), we can apply some group-like operations.

You can compose LFTs with BiLFTs in three ways. First, you can run the output of a BiLFT into the input of a LFT. Second and third, you can run the output of a LFT into the either the x or y input of a BiLFT. Composing a LFT with a BiLFT in any of these ways produces another BiLFT, so there is some notion of closure under composition. Composing the identity LFT with a BiLFT in any of these ways does not change anything. For any composition of a LFT with a BiLFT, you can compose the inverse of the LFT to undo the composition.

BiLFTs can also form infinite compositions, but since there are two inputs, each input can be infinite. We cannot use a Scheme stream, but we can create a two-tailed stream-like object called a binary-expression. A binary-expression is a composition of two infinite compositions. We represent a binary-expression as an object with two delayed cdrs.

(defclass binary-expression ()
  ((generation :initarg :generation
               :initform 0)
   (bilft :initarg :bilft
          :initform (error "Required initarg :bilft was omitted."))
   (delayed-left :initarg :delayed-left
                 :initform (error "Required initarg :delayed-left was omitted."))
   (delayed-right :initarg :delayed-right
                  :initform (error "Required initarg :delayed-right was omitted."))))

Like a LFT, we use binary-expressions by trying to operate on the BiLFT, but forcing one of the tails and refining the binary-expression when necessary.

(defun refine-left (binary-expression)
  (let ((delayed-left (slot-value binary-expression 'delayed-left)))
    (if (null delayed-left)
        (error "Attempt to refine-left on empty stream.")
        (let ((left (force delayed-left)))
          (make-instance 'binary-expression
                         :generation (+ (slot-value binary-expression 'generation) 1)
                         :bilft (compose-bilft-lft-x
                                 (slot-value binary-expression 'bilft)
                                 (if (empty-stream? left)
                                     (make-lft 1 1
                                               0 0)
                                     (stream-car left)))
                         :delayed-left (if (empty-stream? left)
                                           nil
                                           (stream-delayed-cdr left))
                         :delayed-right (slot-value binary-expression 'delayed-right))))))

(defun refine-right (binary-expression)
  (let ((delayed-right (slot-value binary-expression 'delayed-right)))
    (if (null delayed-right)
        (error "Attempt to refine-right on empty stream.")
        (let ((right (force delayed-right)))
          (make-instance 'binary-expression
                         :generation (+ (slot-value binary-expression 'generation) 1)
                         :bilft (compose-bilft-lft-y
                                 (slot-value binary-expression 'bilft)
                                 (if (empty-stream? right)
                                     (make-lft 1 1
                                               0 0)
                                     (stream-car right)))
                         :delayed-left (slot-value binary-expression 'delayed-left)
                         :delayed-right (if (empty-stream? right)
                                            nil
                                            (stream-delayed-cdr right)))))))

refine-fairly alternates forcing the delayed-left or delayed-right tail while refine-disjoint looks at the overlap in the ranges of the BiLFT.


(defun refine-fairly (binary-expression)
  (if (zerop (mod (slot-value binary-expression 'generation) 2))
      (if (null (slot-value binary-expression 'delayed-left))
          (refine-right binary-expression)
          (refine-left binary-expression))
      (if (null (slot-value binary-expression 'delayed-right))
          (refine-left binary-expression)
          (refine-right binary-expression))))

(defun refine-disjoint (binary-expression)
  (if (bilft-disjoint? (slot-value binary-expression 'bilft))
      (refine-right binary-expression)
      (refine-left binary-expression)))

You can do arithmetic on infinite compositions by using the appropriate BiLFT:

(defparameter bilft-add 
  (make-bilft 0 1 1 0
              0 0 0 1))

(defparameter bilft-subtract 
  (make-bilft 0 1 -1 0
              0 0 0 1))

(defparameter bilft-multiply 
  (make-bilft 1 0 0 0
              0 0 0 1))

(defparameter bilft-divide 
  (make-bilft 0 1 0 0
              0 0 1 0))

Converting a binary-expression to a LFT stream

binary-expressions can be operated on in an analagous way to an infinite composition of LFTs, but in order to nest binary expressions, we need to be able to turn one into an infinite composition of LFTs so it can become the left or right input of the other. The way to do this is to generate a stream of LFTs and their inverses. The inverses are composed into the output of the binary-expression while the LFTs are composed downstream into one of the inputs of the next binary-expression.

The math works such that we can select any LFT that has an inverse, but we want our binary-expression to represent a narrowing transformation, so we try composing a few different LFTs and their inverses and see if we still have a narrowing transformation. If we find one, we proceed with the computation, but if we cannot find a LFT and its inverse that preserves the narrowing, we refine the binary-expression by forcing one of the two delayed inputs and composing it.

(defun decompose (left composed)
  (values left (funcall (inverse-lft left) composed)))

(defun decompose-range? (left composed if-success if-failure)
  (multiple-value-bind (left right) (decompose left composed)
    (if (range? right)  ;; does it still narrow?
        (funcall if-success left right)
        (funcall if-failure))))

(defun try-decompose-digit (lft if-success if-failure)
  (decompose-range?
   (make-lft 2 1 0 1) lft
   if-success
   (lambda ()
     (decompose-range?
      (make-lft 1 0 1 2) lft
      if-success
      (lambda ()
        (decompose-range?
         (make-lft 3 1 1 3) lft
         if-success
         if-failure))))))
    
(defun binary-expression-decompose-digit (binary-expression)
  (try-decompose-digit
   (slot-value binary-expression 'bilft)
   (lambda (digit bilft)
     (values digit (make-instance 'binary-expression
                                  :generation (slot-value binary-expression 'generation)
                                  :bilft bilft
                                  :delayed-left (slot-value binary-expression 'delayed-left)
                                  :delayed-right (slot-value binary-expression 'delayed-right))))
   (lambda ()
     (binary-expression-decompose-digit (refine-disjoint binary-expression)))))

(defun binary-expression->lft-stream (binary-expression)
  (multiple-value-bind (digit remainder) (binary-expression-decompose-digit binary-expression)
    (cons-lft-stream digit 
                     (binary-expression->lft-stream remainder))))

Now we have the machinery we need to do arithmetic on pairs of LFT streams.

Infinite Expression Trees

You can only go so far with a finite number of arithmetic operations, but you can go much further with an infinite number. For example, you can create converging infinite series. We can recursively generate an infinite tree of binary expressions. We unfold the tree and call a generating function at each recursive level.

(defun unfold-expression-tree-1 (left generate counter)
  (funcall (funcall generate counter)
           left
           (delay-lft-stream (unfold-expression-tree-1 left generate (+ counter 1)))))
Often the root of the tree is special, so the usual way we call this is
(defun unfold-expression-tree (root-bilft left generate)
  (funcall root-bilft
           left
           (delay-lft-stream (unfold-expression-tree-1 left generate 1))))

Square Root of Infinite Composition

To compute the square root of an infinite composition, we create an infinite tree. Each level of the tree refines the estimate of the square root of the estimate from the next level down in the tree.

(defun sqrt-lft-stream (lft-stream)
  (unfold-expression-tree-1
    lft-stream
    (constantly (make-bilft 1 2 1 0
                            0 1 2 1))
    0))
but we need not construct an infinite tree if each level is identical. We just re-use the level:
(defun sqrt-lft-stream (lft-stream)
  (let ((stream nil))
    (setq stream
          (funcall (make-bilft 1 2 1 0
                               0 1 2 1)
                   lft-stream
                   (delay-lft-stream stream)))
    stream))
This feeds back the square root into its own computation. (These ideas are from Peter Potts and Reinhold Heckmann.)

ex and log(x)

Peter Potts gives these formulas for ex and log(x) where x is an infinite composition of LFTs:

(defun %exp-lft-stream (lft-stream)
  (unfold-expression-tree-1
   (lambda (n)
       (make-bilft (+ (* 2 n) 2) (+ (* 2 n) 1) (* 2 n) (+ (* 2 n) 1)
                   (+ (* 2 n) 1) (* 2 n) (+ (* 2 n) 1) (+ (* 2 n) 2)))
   0
   (funcall (inverse-lft (make-lft 1 -1 1 1)) lft-stream)))

(defun %log-lft-stream (lft-stream)
  (unfold-expression-tree
   (make-bilft 1 1 -1 -1
               0 1  1  0)
   (lambda (n)
     (make-bilft n (+ (* n 2) 1)       (+ n 1) 0
                 0       (+ n 1) (+ (* n 2) 1) n))
   lft-stream))
These infinite expression trees converge only on a limited range, so we need to use identities on the arguments and results to extend the range to cover a wide set of inputs.

Paolo AmorosoManaging pure Common Lisp files on Medley

· 39 days ago

Managing Lisp code in the residential environment of Medley differs from similar tasks in traditional file based Common Lisp systems.

In a previous post I explained how the residential environment of Medley works, discussed some of its facilities and tools, and introduced a workflow for managing Common Lisp code under the residential environment. This leverages the way Medley is designed to work and minimizes friction.

In this post I explain how to use Medley as a file based environment, albeit with some friction and reduced functionality.

I show how to edit, organize, and load pure Common Lisp files, i.e. source files not under the control of the File Manager. Pure files are ordinary text files that contain only Common Lisp code, with no metadata or font control commands like the code databases the File Manager maintains.

Motivation

Letting the residential environment track and control code is the most convenient and productive way of using Medley for Lisp development.

But you can still manually manage pure Common Lisp files. For example, you may want to use from Medley some code you mainly intend to load into the environment and don't change often. This is the case of external libraries and programs developed elsewhere, which you call from new code or run in Medley.

Using Common Lisp code not developed for the pre ANSI implementation of Medley may need adaptation. So let's see how to edit source files.

Editing source files

To create or modify pure Common Lisp files use the TEdit word processor of Medley, not the SEdit structure editor. SEdit doesn't directly manipulate the arbitrary, unstructured text of pure files.

TEdit can read and write ordinary text files, but not by default. To save a file as ASCII execute the command Put > Plain-Text. Execute Get > Unformatted Get to load such a file.

The main downside of using TEdit for Common Lisp is the program is not Lisp aware. It doesn't support automatic indentation, prettyprinting, perenthesis matching, and code evaluation. Another major problem is cursor keys don't work in Medley, which severely limits text editing with TEdit. The Medley team is well aware of the issue.

An alternative is to use your favorite Lisp editor to edit the Common Lisp sources outside of Medley and then load the files from Medley.

Format of Pure Common Lisp files

Although Medley can load pure Common Lisp sources, these files do have some minimal formatting requirements. A pure Common Lisp file must begin with a semicolon ; character, otherwise the system assumes it's an Interlisp file.

Other than that a pure file can contain any Common Lisp feature Medley supports, in any order the language allows.

Loading source files

Once a pure Lisp file is available, load it in Medley with the usual file loading forms such as CL:LOAD. The way symbols are interned depends on whether the file defines or makes current any Common Lisp packages.

Let's go over the main cases: the code has no packages; a single file contains both package and function definitions; the code is split between a file for the package and another for the function definitions.

No package definition

The simplest type of pure file contains no package forms, just function definitions and other Common Lisp expressions.

In this case Medley interns symbols in the USER package without exporting them. The Medley implementation of Common Lisp is closer to CLtLx than ANSI, so it provides the standardized packages LISP (nicknames: COMMON-LISP and CL) and USER instead of COMMON-LISP and COMMON-LISP-USER.

For example, suppose this file SQUARE.LISP defines a function calc-square to compute the square of its argument:

;;; SQUARE.LISP

(defun calc-square (x)
  "Return the square of X."
  (* x x))

After evaluating (load "SQUARE.LISP"), at the > prompt of a Common Lisp Exec you call the function using the proper package qualifier:

> (user::square 3)
9

If the USER package isn't adequate for your code, on Medley CL:LOAD also accepts the keyword parameter :package to supply a package to which *package* is bound when reading the file. For example, at the > prompt of a Xerox Common Lisp (XCL) Exec you can pass the package XCL-USER with a form like:

> (load "SQUARE.LISP" :package (find-package "XCL-USER"))

You must pass an actual package object to :package, not a package designator like :xcl-user.

Since XCL-USER is the default package of XCL Execs no qualifier is required for calling the function:

> (square 3)
9

Package and definitions in one file

Not using packages may be adequate for very small files. However, typical Common Lisp programs do define their own packages.

This example file PKGDEMO.LISP defines the package PKGDEMO with nickname PD that exports the function FUN, which just prints a message:

;;; PKGDEMO.LISP


(in-package "XCL-USER")

(defpackage "PKGDEMO"
  (:use "LISP" "XCL")
  (:nicknames "PD")
  (:export "FUN"))


(in-package "PKGDEMO")

(defun fun ()
  (format t "Hello from PKGDEMO:FUN."))

The file holds both the package definition and the rest of the code. It's important that the first form in the file be in-package to make current a known package like XCL-USER. XCL-USER is the Medley equivalent of COMMON-LISP-USER and uses the LISP package with the standard Common Lisp symbols.

After loading PKGDEMO.LISP from an XCL Exec with (load "PKGDEMO.LISP") you can call the exported function like this:

> (pkgdemo:fun)
Hello from PKGDEMO:FUN.
NIL

or via the package nickname:

> (pd:fun)
Hello from PKGDEMO:FUN.
NIL

Separate package and definition files

Unlike PKGDEMO.LISP in the previous example, most Common Lisp programs split the code between at least two files, one holding the package definition and the other the function definitions. Let's rewrite PKGDEMO.LISP by storing the package in the file PKGSPLIT-PKG.LISP:

;;; PKGSPLIT-PKG.LISP


(in-package "XCL-USER")

(defpackage "PKGDEMO"
  (:use "LISP" "XCL")
  (:nicknames "PD")
  (:export "FUN"))

The function definition goes into the file PKGSPLIT-FUN.LISP that begins with an appropriate in-package followed by the rest of the code:

;;; PKGSPLIT-FUN.LISP


(in-package "PKGDEMO")

(defun fun ()
  (format t "Hello from PKGDEMO:FUN."))

Finally, make sure to load the package first with (load "PKGSPLIT-PKG.LISP"), then the function with (load "PKGSPLIT-FUN.LISP"). Loading the files creates the same objects as the single file example PKGDEMO.LISP and you can call the function the same way:

> (pkgdemo:fun)
Hello from PKGDEMO:FUN.
NIL
> (pd:fun)
Hello from PKGDEMO:FUN.
NIL

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

Joe MarshallInfinite Composition

· 43 days ago

If you like functional composition, you’ll like linear fractional transformations (LFTs). They behave very nicely when you compose them (in fact, they are a group under functional composition). If you compose two LFTs, you get another LFT. You can keep composing as long as you wish. Let’s compose an infinite number of LFTs.

I suppose I should argue that composing an infinite number of LFTs will give you anything at all. Consider those LFTs with non-negative coefficients. When given an input in the range [0,∞], they will output a number in the range [A/C, B/D]. The output range is narrower than, and contained within, the input range. If we compose two such LFTs, the range of output of the outermost LFT will be even narrower. As we compose more and more of these narrowing LFTs, the range of output of the outermost LFT will become narrower and narrower. If we compose an infinite number of narrowing LFTs, the output range will narrow to a single point. Curiously, the limits on the range of output of a finite composition of LFTs are always rational numbers, yet the output from infinite composition of LFTs can be an irrational real number.

There are many ways to represent an infinite number of LFTs. A generator or a co-routine, for example, but I like the stream abstraction from Scheme. This is easily implemented in Common Lisp. A delay macro creates promises:

(defclass promise ()
  ((forced? :initform nil)
   (values-or-thunk :initarg :thunk
                    :initform (error "Required initarg :thunk omitted.")))
  (:documentation "A simple call-by-need thunk."))

(defmacro delay (expression)
  “Delays evaluation of an expression and returns a promise.”
  ‘(make-instance ’promise :thunk (lambda () ,expression)))
and the force function forces evaluation:
(defun force (promise)
  “Returns the values of a promise, forcing it if necessary.”
  (check-type promise promise)
  ;; Ensure the values have been memoized.
  (unless (slot-value promise ’forced?)
    (let ((values (multiple-value-list (funcall (slot-value promise ’values-or-thunk)))))
      ;; If this is not a recursive call, memoize the result.
      ;; If this is a recursive call, the result is discarded.
      (unless (slot-value promise ’forced?)
        (setf (slot-value promise ’values-or-thunk) values)
        (setf (slot-value promise ’forced?) t))))
   ;; Return the memoized values.
   (values-list (slot-value promise ’values-or-thunk)))
A stream is a cons of an element and a promise to produce the rest of the stream:
(defclass lft-stream ()
  ((car :initarg :car
        :initform (error "Required initarg :car omitted.")
        :reader stream-car)
   (delayed-cdr :initarg :delayed-cdr
                :initform (error "Required initarg :delayed-cdr omitted.")
                :reader stream-delayed-cdr)))

(defmacro cons-lft-stream (car cdr)
  ‘(make-instance ’stream :car ,car :delayed-cdr (delay ,cdr)))

(defun stream-cdr (stream)
  (force (stream-delayed-cdr stream)))

A stream of LFTs will represent an infinite composition. The first LFT is the outermost LFT in the composition. To incrementally compose the LFTs, we force the delayed cdr of the stream to get the second LFT. We compose the first LFT and the second LFT to get a new stream car, and the cdr of the delayed cdr is the new stream cdr. That is, we start with the stream {F …}, we force the tail, getting {F G …}, then we compose the first and second elements, getting {(F∘G) …}.

(defun refine-lft-stream (lft-stream)
  (let ((tail (stream-cdr lft-stream))) ;; forces cdr
    (make-instance ’stream
                   :car (compose (stream-car lft-stream) (stream-car tail))
                   :delayed-cdr (stream-delayed-cdr tail))))
so we can now write code that operates on what we have composed so far (in the car of the LFT stream), or, if we need to incrementally compose more LFTs, we repeatedly call refine-lft-stream until we have composed enough.

For example, we can compute the nearest float to an infinite composition as follows:

(defun nearest-single (lft-stream)
  (let ((f0 (funcall (stream-car lft-stream) 0))
        (finf (funcall (stream-car lft-stream) ’infinity)))
    (if (and (numberp f0)
             (numberp finf)
             (= (coerce f0 ’single-float)
                (coerce finf ’single-float)))
        (coerce f0 ’single-float)
        (nearest-single (refine-lft-stream lft-stream)))))

The problem with infinite compositions is that they may never narrow enough to proceed with a computation. For example, suppose an infinite composition converged to a number exactly in between two adjacent floating point numbers. The upper limit of the narrowing range will always round to the float that is above, while the lower limit will always round to the float that is below. The floats will never be equal no matter how many terms of the infinite composition are composed, so functions like nearest-single will never return a value. This is a tradeoff. We either continue computing, perhaps forever, with more and more precise values, or we stop at some point, perhaps giving an incorrect answer.

Operating on Infinite Compositions

You can compose a LFT with an infinite composition of LFTs. If we compose the LFT F with the infinite composition {G H …}, we can represent this one of two ways. Either we just tack the F on to the front, getting {F G H …}, or we continue further and eagerly compose the first two elements {(F∘G) H …}. If F is, for example, a LFT that adds 10 or multiplies by 7, the effect is to add 10 to or multiply by 7 the infinite composition. In this way, we can do arithmetic involving rational numbers and infinite compositions.

Sources of Infinite Compositions

It isn’t likely that you have a lot of ad hoc LFT streams you want to compose. Instead, we want some sources of infinite compositions. There are a few useful functions of finite arguments that return infinite compositions. I got these from Peter Potts's thesis. While most of these have limited ranges of arguments, you can use identity operations to divide down the input to an acceptable range and multiply up the output to the correct answer.

√(p/q)

Reinhold Heckmann came up with this one:

(defun %sqrt-rat (p q)
  (let ((diff (- p q)))
    (labels ((rollover (num den)
               (let ((d (+ (* 2 (- den num)) diff)))
                 (if (> d 0)
                     (cons-lft-stream (make-lft 1 0 1 2) (rollover (* num 4) d))
                     (cons-lft-stream (make-lft 2 1 0 1) (rollover (- d) (* den 4)))))))
       (rollover p q))))

The LFTs that this returns are curious in that when you compose them with a range, they divide the range in two and select one or other (high or low) segment. The outermost LFT in the infinite composition will represent a range that contains the square root, and the subsequent LFTs will narrow it down by repeatedly bifurcating it and selecting the top or bottom segment.

ex, where x is rational and 1/2 < x ≤ 2

(defun %exp-rat (x)
  (check-type x (rational (1/2) 2))
  (cons-lft-stream
    (make-lft (+ 2 x) x
              (- 2 x) x)
    (lft-stream-map (lambda (n)
                      (make-lft (+ (* 4 n) 2) x
                                x             0))
                    (naturals))))

log(x), where x is rational and x > 1

(defun %log-rat (x)
  (check-type x (rational (1) *))
  (lft-stream-map
    (lambda (n)
      (funcall (make-lft 0             (- x 1)
                         (- x 1) (+ (* n 2) 1))
               (make-lft 0       (+ n 1)
                         (+ n 1)       2)))
    (integers)))

xy, where x and y are rational and x > 1 and 0 < y < 1

(defun %rat-pow (x y)
  (check-type x (rational (1) *))
  (check-type y (rational (0) (1)))
  (cons-lft-stream
    (make-lft y 1
              0 1)
    (lft-stream-map
     (lambda (n)
       (funcall (make-lft 0             (- x 1)
                          (- x 1) (- (* n 2) 1))
                (make-lft 0       (- n y)
                          (+ n y)       2)))
     (naturals))))

tan(x), where x is rational and 0 < x ≤ 1

(defun %rat-tan (x)
  (check-type x (rational (0) 1))
  (lft-stream-map
   (lambda (n)
     (funcall (make-lft 0 x
                        x (+ (* n 4) 1))
              (make-lft 0 x
                        x (- (* n -4) 3))))
   (integers)))

tan-1(x), where x is rational and 0 < x ≤ 1

(defun big-k-stream (numerators denominators)
  (cons-lft-stream (make-lft 0 (stream-car numerators)
                             1 (stream-car denominators))
                   (big-k-stream (stream-cdr numerators) (stream-cdr denominators))))

(defun %rat-atan (z)
  (check-type z (rational (0) 1))
  (let ((z-squared (square z)))
    (cons-lft-stream (make-lft 0 z
                               1 1)
                     (big-k-stream (stream-map (lambda (square)
                                                 (* z-squared square))
                                               (squares))
                                   (stream-cdr (odds))))))

These functions have limited applicability. In my next couple of posts, I’ll show some functions that transform LFT streams into other LFT streams, which can be used to increase the range of inputs and outputs of these primitive sources.

Eugene ZaikonnikovTweaking SLIME Xref for Remote Images

· 48 days ago

By the nature of embedded development one spends a lot of time debugging on target devices. SLIME experience for the most part is as smooth as on local host with the exception of cross referencing. Swank backend on target is reporting local paths in xref records which the frontend on your host then tries to open.

The canonical workaround from the user manual is using slime-tramp contrib, allowing you to navigate the source tree on remote target over SSH. I however greatly prefer to work on the local copy of the source code, with much lower latency (SSH over VPN over cellular to a remote site is no fun) and ability to stage and commit changes immediately. A somehwat kludgy workflow that does the trick is using a hacked slime-postprocess-xref below to substitute $HOME in xref records. The source tree in remote home has to be placed in the same relative location as on your host. One should also remember copying their local tree to remote at the end of debugging session, in case the instance has to be restarted.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
;;; A modified version that substitutes HOME in remote images with local ones
;;; enabling local code navigation with the usual SLIME xref tools
(defun slime-postprocess-xref (original-xref)
  "Process (for normalization purposes) an Xref comming directly
from SWANK before the rest of Slime sees it. In particular,
convert ETAGS based xrefs to actual file+position based
locations."
  (if (not (slime-xref-has-location-p original-xref))
      (list original-xref)
    (let ((loc (slime-xref.location original-xref)))
      (slime-dcase (slime-location.buffer loc)
        ((:etags-file tags-file)
         (slime-dcase (slime-location.position loc)
           ((:tag &rest tags)
            (visit-tags-table tags-file)
            (mapcar (lambda (xref)
                      (let ((old-dspec (slime-xref.dspec original-xref))
                            (new-dspec (slime-xref.dspec xref)))
                        (setf (slime-xref.dspec xref)
                              (format "%s: %s" old-dspec new-dspec))
                        xref))
                    (cl-mapcan #'slime-etags-definitions tags)))))
        (t
         ;; Find the /home/someuser/ path component and replace it
         ;; with our local $HOME, if any
         (setf (cadr (slime-location.buffer loc))
               (replace-regexp-in-string "^/[^/]+/[^/]+"
                                         (getenv "HOME")
                                         (cadr (slime-location.buffer loc))))
         (list original-xref))))))

Luís OliveiraManuel Simoni on CL's control flow primitives

· 48 days ago

Manuel Simoni dusts his Axis of Eval blog off and writes about Common Lisp's BLOCK / RETURN-FROM and UNWIND-PROTECT. A summary for non-Lispers.

TurtleWareWriting an ad hoc GUI for Coleslaw

· 49 days ago

Table of Contents

  1. Preliminary steps
  2. Embracing the chaos
  3. Presentations
  4. Managing a blog collection
  5. Managing a blog instance
  6. Big ball of mud
  7. Closing thoughts

Coleslaw is a "Flexible Lisp Blogware". It is a program that I'm using to manage my blogs and allows for an offline blog compilation. The functionality of the website may be extended with plugins and the visual appearance is defined by configurable themes.

Its design is straightforward (if not a bit messy), so it is a good candidate to show how to slap a CLIM interface on top of existing software. The integration will be very shallow to not encroach into Coleslaw responsibilities, yet deep enough to provide a convenience utility over the library.

Preliminary steps

In this post we will use a few dependencies. Of course one of them is mcclim. Please make sure that you are using an up-to-date version; i.e clone it from the repository to ~/quicklisp/local-projects. There are other dependencies too. Load them all in the REPL with:

(ql:quickload '(coleslaw-cli cl-fad alexandria local-time
                mcclim clouseau hunchentoot)
              :verbose t)

The whole program described in this tutorial is defined in a single package:

(defpackage "COLESLAW-GUI"
  (:use "CLIM-LISP"))
(in-package "COLESLAW-GUI")

We are good to go now.

Embracing the chaos

The README.md in the project's repository mentions a few commands that may be invoked from the command line and from the lisp REPL. What they have in common is that they assume, that the blog resides in the current working directory. Here we are going to introduce a macro that estabilishes a necessary context:

(defmacro with-current-directory ((path value) &body body)
  `(let* ((,path (cl-fad:pathname-as-directory ,value))
          (*default-pathname-defaults* ,path))
     (ensure-directories-exist ,path)
     (uiop:chdir ,path)
     ,@body))

Moreover Coleslaw assumes that only one blog will be loaded during its lifetime and many objects are treated as singletons. We will embrace this chaos and provide a macro that estabilishes an appropriate environment for a blog. The key to each environment is its directory pathname:

;;; Allow for passing "env" here.
(defun blog-key (blog)
  (etypecase blog
    (null nil)
    (cons (coleslaw:repo-dir (first blog)))
    (coleslaw::blog (coleslaw:repo-dir blog))))

(defun blog () coleslaw:*config*)
(defun site () coleslaw::*site*)

(defun make-null-env ()
  (list nil (make-hash-table :test #'equal)))

(defun copy-blog-env ()
  (list coleslaw:*config*
        coleslaw::*site*))

(defun load-blog-env (env)
  (destructuring-bind (blog site)
      (or env (make-null-env))
    (setf coleslaw:*config* blog
          coleslaw::*site* site)
    ;; Populates *ALL-TAGS* and *ALL-MONTHS* using *SITE*.
    (coleslaw::update-content-metadata)))

(defun save-blog-env (table)
  (when table
    (setf (gethash (blog-key coleslaw:*config*) table)
          (copy-blog-env))))

(defmacro with-blog-env ((env table) &body body)
  `(let (coleslaw:*config*
         coleslaw::*site*)
     (load-blog-env ,env)
     (multiple-value-prog1 (progn ,@body)
       (save-blog-env ,table))))

Presentations

First we will define presentation types, so we can associate them with objects on the screen. The blog environment is composed of a pair:

  • a blog: an instance of the class coleslaw::blog

  • a site: a hash table that contains posts

    (clim:define-presentation-type coleslaw::blog () :description "(Configuration)")

    (clim:define-presentation-type blog-env () :description "(Blog)")

Presentation types are like types denoted by classes, but with a twist - they may be additionally parametrized; i.e (INTEGER 3) is a presentation type. There are also an abstract presentation types that are not tied to a single class. For example we may have presentation types "red team" and "blue team", where some arbitrary objects are presented as one or the another.

The presentation method present is used to associate the object with the presentation type and put it on the screen as the presentation. In other words the presentation is a pair (object type). The method specializes arguments:

  • object: most notably the object class, sometimes left unspecialized
  • type: obligatory specialization to the presentation type (may be abstract)
  • stream: typically left unspecialized, but may be utilized for serialization
  • view: customizes how the object is presented depending on the local context

The most primitive view is the textual view. Methods specializing to it should treat the stream as if it handles only text, so the representation should be a string. Note that presentations may be nested, like in our case:

(clim:define-presentation-method clim:present
    (self (type coleslaw::blog) stream (view clim:textual-view) &key acceptably for-context-type)
  (declare (ignore view acceptably for-context-type))
  (format stream "~a" (blog-key self)))

(clim:define-presentation-method clim:present
    (env (type blog-env) stream (view clim:textual-view) &key acceptably for-context-type)
  (declare (ignore acceptably for-context-type))
  (with-blog-env (env nil)
    (princ "[" stream)
    (clim:present (blog) 'coleslaw::blog :stream stream :view view)
    (princ "]" stream)))

Managing a blog collection

The system coleslaw-cli that is bundled with coleslaw defines commands that allow for creating a blog, adding (stub) post files to it, compiling the blog to the staging area and deploying the blog using plugins.

We are going to extend this set of operations to allow working with a collection of blogs. Since we are not barbarians, we are going to encapsulate the state in the application frame, and not in a global variable.

(clim:define-application-frame coleslaw-cli ()
  ((envs :initform (make-hash-table :test #'equal) :reader envs)))

Adding new blogs to the collection is a result of opening them or creating new ones. Both operations require for the program to operate in a target directory:

(clim:define-command (com-open-blog :name t :command-table coleslaw-cli)
    ((directory 'pathname))
  (clim:with-application-frame (frame)
    (with-current-directory (path directory)
      (with-blog-env (nil (envs frame))
        (format *query-io* "Opening a blog in ~s.~%" path)
        (coleslaw::load-config path)))))

(clim:define-command (com-make-blog :name t :command-table coleslaw-cli)
    ((directory 'pathname))
  (clim:with-application-frame (frame)
    (with-current-directory (path directory)
      (with-blog-env (nil (envs frame))
        (format *query-io* "Creating a new blog in ~s. " path)
        (coleslaw-cli:setup)
        (coleslaw::load-config path)))))

We need a command to list loaded blogs. All remaining operations will specialize to presentation types blog-env and coleslaw::blog, so we will present them with the function present:

(clim:define-command (com-list-blogs :name t :command-table coleslaw-cli)
    ()
  (clim:with-application-frame (frame)
    (dolist (env (alexandria:hash-table-values (envs frame)))
      (clim:present env 'blog-env :stream (clim:frame-query-io frame)
                                  :single-box t)
      (terpri (clim:frame-query-io frame)))))

For completness we need a command that will remove a blog from the collection.

(clim:define-command (com-close-blog :name t :command-table coleslaw-cli)
    ((self 'blog-env))
  (clim:with-application-frame (frame)
    (remhash (blog-key self) (envs frame))))

Finally there are two very important commands that compile the blog. Note that both commands will fail if there are no posts in the blog (coleslaw behavior).

(clim:define-command (com-stage-blog :name t :command-table coleslaw-cli)
    ((self 'blog-env))
  (clim:with-application-frame (frame)
    (with-current-directory (path (blog-key self))
      (with-blog-env (self (envs frame))
        (format *query-io* "Staging the blog from ~s. " path)
        (coleslaw-cli:stage)))))

(clim:define-command (com-deploy-blog :name t :command-table coleslaw-cli)
    ((self 'blog-env))
  (clim:with-application-frame (frame)
    (with-current-directory (path (blog-key self))
      (with-blog-env (self (envs frame))
        (format *query-io* "Deploying the blog from ~s. " path)
        (coleslaw-cli:deploy)))))

Additionally we define a few convenience commands:

  • creating test data
  • clearing the screen
(clim:define-command (com-spam-blogs :name t :command-table coleslaw-cli)
    ()
  (dotimes (i 8)
    (let ((d (format nil "/tmp/blogs/specimen-~2,'0d/" i)))
      (if (probe-file d)
          (com-open-blog d)
          (with-current-directory (path d)
            (com-make-blog d)
            ;; Without any content "Stage" and "Deploy" will fail.
            (coleslaw-cli:new))))))

(clim:define-command (com-clear :name t :command-table coleslaw-cli)
    ()
  (clim:with-application-frame (frame)
    (clim:window-clear (clim:frame-query-io frame))))

Now to execute a command on the blog we may type the command name and select an element from the list (with a pointer). We want also to allow the user to click on the blog with the right pointer button and select the operation without explicitly typing the command, so we define presentation to command translators:

(macrolet ((def (name command short-description long-description)
             `(clim:define-presentation-to-command-translator ,name
                  (blog-env ,command coleslaw-cli
                   :gesture nil
                   :documentation ,short-description
                   :pointer-documentation ,long-description)
                  (self)
                `(,self))))
  (def trn-close-blog  com-close-blog  "Close"  "Remove blog from collection")
  (def trn-stage-blog  com-stage-blog  "Stage"  "Compile blog to staging area")
  (def trn-deploy-blog com-deploy-blog "Deploy" "Compile blog to production"))

Moreover we'd like to be able to type the blog from the keyboard, so we define a presentation method accept that matches the blog against loaded ones.

(clim:define-presentation-method clim:accept
    ((type blog-env) stream (view clim:textual-view) &rest args)
  (declare (ignore args))
  (clim:with-application-frame (frame)
    (clim:completing-from-suggestions (stream)
      (maphash (lambda (key val)
                 (clim:suggest (namestring key) val))
               (envs frame)))))

This concludes our command line blog manager. We've mentioned the following topics:

  • application frame: defines the dynamic context of the application
  • command table: defines available commands and translators
  • presentation types: specify ontologies that may be shared among programs
  • presentation methods: specify interactions like present and accept

Managing a blog instance

Until now we've been working with the interactor and the textual view. Focusing first on presentation types and commands is good, because it captures an essence of the application interface and delays distracting stuff like visuals. Now, to make this post more appealing (less appalling?), we will extend the application with additional functionality.

The display function is responsible for presenting content on the application stream. It may be anything really, but we will defer it to a method PRESENT specialized to the frame itself. That's the purest approach. We also define a few utilities for later.

(defun display (object stream)
  (clim:present object (clim:presentation-type-of object) :stream stream))

(defun present* (object stream)
  (clim:present object (clim:presentation-type-of object) :stream stream))

(defmacro dohash (((key val) hash &optional result) &body body)
  (let ((cont (gensym)))
    `(flet ((,cont (,key ,val) ,@body))
       (declare (dynamic-extent (function ,cont)))
       (maphash (function ,cont) ,hash)
       ,result)))

(defun format-today ()
  (local-time:format-timestring nil (local-time:now)
                                :format '((:year 4) "-" (:month 2) "-" (:day 2) "-"
                                          (:hour 2) "-" (:min 2) "-" (:sec 2))))

Our application frame will feature graphics and other fluff to cater to people who are into this kind of thing. To do that we define a separate view class that extends the textual-view. While we are technically subclassing it, this is not a semantically correct description. In reality we are extending the class with non-textual capabilities. If you were looking for CLOS conceptual limits, then here you have one.

;;; KLUDGE: FANCY-VIEW extends (not specializes) the TEXTUAL-VIEW.
(defclass fancy-view (clim:textual-view) ())
(defvar +fancy-view+ (make-instance 'fancy-view))

Finally the application frame definition. It inherits from coleslaw-cli and adds a new application pane to show the frame state.

(clim:define-application-frame coleslaw-gui (coleslaw-cli)
  ((current-blog :initform nil :accessor current-blog))
  (:command-table (coleslaw-gui :inherit-from (coleslaw-cli)))
  (:reinitialize-frames t)
  (:panes (app :application :display-function 'display :default-view +fancy-view+
               :text-margins '(:left 20 :top 10))
          (int :interactor)))

Now we define new commands to load content, select a loaded blog and create a new blog. Loading the content is the operation that walks directories and adds found resources to the model.

(clim:define-command (com-update :command-table coleslaw-gui)
    ((self 'blog-env))
  (clim:with-application-frame (frame)
    (with-current-directory (path (blog-key self))
      (with-blog-env (self (envs frame))
        ;; This function removes content from *site* before adding it back.
        (coleslaw::load-content)))))

(clim:define-command (com-select :command-table coleslaw-gui)
    ((self 'blog-env))
  (clim:with-application-frame (frame)
    (setf (current-blog frame) self)
    (com-update self)))

(clim:define-command (com-create :name t :command-table coleslaw-cli)
    ((self 'blog-env)
     (type 'string :default "post")
     (name 'string :default (format-today)))
  (clim:with-application-frame (frame)
    (with-current-directory (path (blog-key self))
      (with-blog-env (self (envs frame))
        ;; This function removes content from *site* before adding it back.
        (coleslaw-cli:new type name)
        (com-update self)))))

(macrolet ((def (name command gesture short-description long-description)
             `(clim:define-presentation-to-command-translator ,name
                  (blog-env ,command coleslaw-gui
                   :gesture ,gesture
                   :documentation ,short-description
                   :pointer-documentation ,long-description)
                  (self)
                `(,self))))
  (def trn-update com-update nil     "Update"  "Update blog from disk")
  (def trn-select com-select :select "Select"  "Show blog details")
  (def trn-create com-create nil     "Create"  "Create new content"))

The implementation of the present method specializes to fancy-view. First we show the list of opened blogs (using the textual view), and then we show the selected blog. Rendering of the current blog is defered to another present method.

The current blog will show the same content as it is presented on the list above, until we define a specialized method. Note that we present it so it is not sensitive to pointer clicks. This is to avoid unnecessary noise.

(clim:define-presentation-method clim:present
    ((frame coleslaw-gui) (type coleslaw-gui) stream (view fancy-view) &rest args)
  (declare (ignore args))
  (clim:formatting-item-list (stream)
   (dohash ((dir env) (envs frame))
     (declare (ignore dir))
     (clim:formatting-cell (stream)
       (clim:with-drawing-options (stream :ink (if (eql env (current-blog frame))
                                                   clim:+dark-red+
                                                   clim:+foreground-ink+))
         (clim:present env 'blog-env :stream stream :view clim:+textual-view+ :single-box t)))))
  (terpri stream)
  (clim:present (current-blog frame) 'blog-env :stream stream :view view
                                               :sensitive nil
                                               :allow-sensitive-inferiors t))

Presenting the current blog will be implemented as follows:

  1. Show the blog title – header text style
  2. Show available commands – deliberely goofy icons
  3. Show the blog content – defered to the next method
(clim:define-presentation-type coleslaw::index ()
  :description "(Index)")

(clim:define-presentation-type site ()
  :description "(Site)")

(clim:define-presentation-type post ()
  :description "(Post)")

(clim:define-presentation-type post ()
  :description "(Page)")

(defun gap-the-gap (stream command label color)
  (clim:with-output-as-presentation (stream command '(clim:command :command-table coleslaw-gui))
    (clim:with-room-for-graphics (stream :first-quadrant nil)
      (clim:draw-circle* stream 0 0 40 :ink clim:+dark-red+ :filled nil :line-thickness 20)
      (clim:surrounding-output-with-border (stream :filled t :ink color)
        (clim:draw-text* stream label 0 0 :align-x :center :align-y :center
                                          :text-size :small
                                          :text-family :fix
                                          :ink clim:+white+)))))

(clim:define-presentation-method clim:present
    ((self cons) (type blog-env) stream (view fancy-view) &rest; args)
  (declare (ignore args))
  (clim:with-application-frame (frame)
    (with-blog-env (self (envs frame))
      ;; Blog title
      (clim:with-text-style (stream (clim:make-text-style :serif :bold :large))
        (format stream "~a" (coleslaw:title (blog))))
      (terpri stream)
      ;; Update the blog bleeper
      (gap-the-gap stream `(com-update ,self) "Mind the gap!" clim:+dark-blue+)
      (princ " " stream)
      (gap-the-gap stream `(com-create ,self "post" ,(format-today)) "Fill the gap!" clim:+dark-green+)
      (princ " " stream)
      (gap-the-gap stream `(com-create ,self "page" ,(format-today)) "Keep the gap!" clim:+dark-red+)
      (terpri stream)
      ;; The content
      (clim:present (site) 'site :stream stream :view view))))

(clim:define-presentation-method clim:present
    (self (type site) stream (view fancy-view) &rest; args)
  (declare (ignore args))
  (clim:formatting-table (stream)
   (dohash ((key val) self)
     (clim:formatting-row (stream)
       (clim:formatting-cell (stream) (present* key stream))
       (clim:formatting-cell (stream) (present* val stream))))))

The discovered content is stored in a hash table. Keys are URL addresses and values are content objects: posts, rss feeds, tag feeds and indexes. Values are presented, so these presentations may be selected with a pointer when the input context matches. For example we may invoke the inspector or a file editor:

(clim:define-presentation-action act-open-content
    (coleslaw::content nil coleslaw-gui
     :documentation "Open file"
     :pointer-documentation "Open the content file")
    (object)
  (uiop:launch-program (format nil "xdg-open ~a" (coleslaw::content-file object))))

(clim:define-presentation-action act-kill-content
    (coleslaw::content nil coleslaw-gui
     :documentation "Kill file"
     :pointer-documentation "Kill the content file")
    (object)
  (clim:with-application-frame (frame)
    (with-current-directory (dir (blog-key (current-blog frame)))
      (uiop:launch-program (format nil "rm ~a" (coleslaw::content-file object))))
    (clim:execute-frame-command frame `(com-update ,(current-blog frame)))))

(clim:define-presentation-action act-inspect
    ((or coleslaw::blog coleslaw::content coleslaw::feed coleslaw::index) nil coleslaw-gui
     :gesture nil
     :documentation "Inspect content"
     :pointer-documentation "Inspect site content")
  (object)
  (clouseau:inspect object :new-process t))

A difference between actions and commands is that actions are not expected to change the internal model, so they don't progress the display loop. Now we may click on a post and the default program that opens the file will be launched. We may also right-click on the content value and inspect it with clouseau.

In this section I've mentioned the following topics:

  • the textual view may be extended with graphical capabilities (i.e colors)
  • display function is a function that creates presentations on the stream
  • presentation translators may be used to call a command from a presentation
  • presentation method present may be nested inside another one
  • presentation types are used as specializers in presentation methods
  • it is possible to present on the stream a command along with arguments
  • presentation actions, unlike commands, are executed immedietely

Big ball of mud

Previously we've extended the application by specifying a new display function. Now we will extend it further by adding a web server to preview a blog.

(clim:define-application-frame durk (coleslaw-gui)
  ((acceptor :initarg :acceptor :accessor acceptor))
  (:panes
   (app :application :display-function 'display :default-view +fancy-view+)
   (int :interactor :height 100))
  (:reinitialize-frames t)
  (:command-table (durk :inherit-from (coleslaw-gui)))
  (:default-initargs :acceptor nil))

;;; We could enable and disable commands by calilng (SETF CLIM:COMMAND-ENABLED).
(defmethod clim:command-enabled (name (frame durk))
  (case name
    (com-stop-acceptor (hunchentoot:started-p (acceptor frame)))
    (com-start-acceptor (not (hunchentoot:started-p (acceptor frame))))
    (otherwise (call-next-method))))

(defmethod clim:adopt-frame :after (fm (self durk))
  (format *debug-io* "Booting up.~%")
  (setf (acceptor self) (make-instance 'hunchentoot:easy-acceptor :port 4242))
  (setf hunchentoot:*dispatch-table*
        (list (hunchentoot:create-static-file-dispatcher-and-handler "/" "/tmp/coleslaw/index.html")
              (hunchentoot:create-folder-dispatcher-and-handler "/" "/tmp/coleslaw/"))))

(defmethod clim:disown-frame :before (fm (self durk))
  (format *debug-io* "Cleaning up.~%")
  (when (hunchentoot:started-p (acceptor self))
    (hunchentoot:stop (acceptor self))))

(define-durk-command (com-start-acceptor)
    ((self 'hunchentoot:acceptor :gesture :select))
  (format *debug-io* "Starting acceptor.~%")
  (hunchentoot:start self))

(define-durk-command (com-stop-acceptor)
    ((self 'hunchentoot:acceptor :gesture :select))
  (format *debug-io* "Stopping acceptor.~%")
  (hunchentoot:stop self))

Here's the key part: instead of defining single method for presenting the frame, we define a :before method that presents named commands and the acceptor:

(clim:define-presentation-method clim:present :before ((self durk) (type durk) stream view &rest args)
  (declare (ignore args))
  (clim:formatting-item-list (stream)
    (clim:map-over-command-table-names
     (lambda (name command)
       (declare (ignore name))
       (clim:formatting-cell (stream)
         (clim:surrounding-output-with-border (stream)
          (clim:present command 'clim:command :stream stream))))
     (clim:find-command-table 'durk)))
  (terpri stream)
  (present* (acceptor self) stream)
  (terpri stream))

(clim:define-presentation-method clim:present
    ((self hunchentoot:acceptor) (type hunchentoot:acceptor) stream view &rest args)
  (declare (ignore view args))
  (clim:with-drawing-options (stream :ink (if (hunchentoot:started-p self)
                                              clim:+dark-green+
                                              clim:+dark-red+))
    (format stream "~a~%" self)))

In this section I mentioned the following topics:

  • presentation methods may have auxiliary methods like :after
  • we may extend existing applications by tweaking presentation methods and view
  • it is possible to enable and disable commands depending on the frame state
  • the frame life cycle starts when it is adopted, and ends when it is disowned
  • we may mix formatting macros, drawing options and stream output freely

And voila, now we can preview the blog:

Closing thoughts

In this post we've covered many CLIM features that are useful for writing applications. Some takeaways are:

  • commands have a straightforward interpretation compatible with CLI
  • command tables encapsulate commands and may inherit from each other
  • frames encapsulate the dynamic context and organize windows
  • presentations allow for associating a presentation type with an object
  • presentation types may be used to specialize numerous presentation methods
  • views provide an easy way to customize the interface depending on context
  • presentation translators may be used to coerce object to the input context
  • presentation actions allow for triggering immediate handlers
  • commands may be conditionally disabled
  • the display function may be extended by specializing the function present

Adding an ad-hoc GUI to existing libraries amounts for not so many lines of code and is moderately easy task. You may find the source code of this tutorial here:

/static/misc/coleslaw-gui.lisp

While the tool is rather on the simplistic side, I'm already using it to preview and manage a few of my blogs. Some extensions are due, but they'd rather make the tutorial more complex - contrary to the intention of this post.

Happy hacking,
Daniel

Joe MarshallExponentiating Functions

· 50 days ago

If we compose a function F(x) with itself, we get F(F(x)) or F∘F. If we compose it again, we get F(F(F(x))) or F∘F∘F. Rather than writing ‘F’ over and over again, we can abuse exponential notation and write (F∘F∘F) as F3, where the superscript indicates how many times we compose the function. F1 is, naturally, just F. F0 would be zero applications of F, which would be the function that leaves its argument unchanged, i.e. the identity function.

The analogy with exponentiation goes deeper. We can quickly exponentiate a number with a divide and conquer algorithm:

(defun my-expt (base exponent)
  (cond ((zerop exponent) 1)
        ((evenp exponent) (my-expt (* base base) (/ exponent 2)))
        (t (* base (my-expt base (- exponent 1))))))
The analagous algorithm will exponentiate a function:
(defun fexpt (f exponent)
  (cond ((zerop exponent) #'identity)
        ((evenp exponent) (fexpt (compose f f) (/ exponent 2)))
        (t (compose f (fexpt f (- exponent 1))))))

The function (lambda (x) (+ 1 (/ 1 x))) takes the reciprocal of its input and adds one to it. What happens if you compose it with itself? We can rewrite it as a linear fractional transformation (LFT) (make-lft 1 1 1 0;) and try it out:

> (make-lft 1 1 1 0)
#<LFT 1 + 1/x>

> (compose * (make-lft 1 1 1 0))
#<LFT (2x + 1)/(x + 1)>

> (compose * (make-lft 1 1 1 0))
#<LFT (3x + 2)/(2x + 1)>

> (compose * (make-lft 1 1 1 0))
#<LFT (5x + 3)/(3x + 2)>

> (compose * (make-lft 1 1 1 0))
#<LFT (8x + 5)/(5x + 3)>

> (compose * (make-lft 1 1 1 0))
#<LFT (13x + 8)/(8x + 5)>

> (compose * (make-lft 1 1 1 0))
#<LFT (21x + 13)/(13x + 8)>
Notice how the coefficients are Fibonacci numbers.

We can compute Fibonacci numbers efficiently by exponentiating the LFT 1 + 1/x.

> (fexpt (make-lft 1 1 1 0) 10)
#<LFT (89x + 55)/(55x + 34)>

> (fexpt (make-lft 1 1 1 0) 32)
#<LFT (3524578x + 2178309)/(2178309x + 1346269)>
Since we’re using a divide and conquer algorithm, raising to the 32nd power involves only five matrix multiplies.

Fixed points

If an input x maps to itself under function f, we say that x is a fixed point of f. So suppose we have a function f with a fixed point x. We consider the function f’ which ignores its argument and outputs x. If we compose f with f’, it won’t make difference that we run the result through f again, so f’ = f∘f’ = f. You can find a fixed point of a function by composing the function with its fixed point. Unfortunately, that only works in a lazy language, so you have two options: either choose a finite number of compositions up front, or compose on demand.

You can approximate the fixed point of a function by exponentiating the function to a large number.

(defun approx-fixed-point (f) 
  (funcall (fexpt f 100) 1))

> (float (approx-fixed-point (make-lft 1 1 1 0)))
1.618034

> (float (funcall (make-lft 1 1 1 0) *))
1.618034

Alternatively, we could incrementally compose f with itself as needed. To tell if we are done, we need to determine if we have reached a function that ignores its input and outputs the fixed point. If the function is a LFT, we need only check that the limits of the LFT are equal (up to the desired precision).

(defun fixed-point (lft)
  (let ((f0   (funcall lft 0))
        (finf (funcall lft ’infinity)))  
    (if (and (numberp f0)
             (numberp finf)
             (= (coerce f0 ’single-float) 
                (coerce finf ’single-float)))
        (coerce f0 ’single-float)
        (fixed-point (compose lft lft)))))

> (fixed-point (make-lft 1 1 1 0))
1.618034    

Joe MarshallRoll Your Own Linear Fractional Transformations

· 53 days ago

Looking for a fun weekend project? Allow me to suggest linear fractional transformations.

A linear fractional transformation (LFT), also known as a Möbius transformation or a homographic function, is a function of the form

(lambda (x)
  (/ (+ (* A x) B)
     (+ (* C x) D)))
You could just close over the coefficients,
(defun make-lft (A B C D)
  (lambda (x)
    (/ (+ (* A x) B)
       (+ (* C x) D))))
but you’ll want access to A, B, C, and D. If you implement LFTs as funcallable CLOS instances, you can read out the coefficients from slot values.

Constructor

The coefficients A, B, C, and D could in theory be any complex number, but we can restrict them to being integers and retain a lot of the functionality. If we multiply all the coefficients by the same factor, it doesn't change the output of the LFT. If you have a rational coefficient instead of an integer, you can multiply all the coefficients by the denominator of the rational. If there is a common divisor among the coefficients, you can divide it out to reduce to lowest form. (In practice, the common divisor will likely be 2 if anything, so if the coefficients are all even, divide them all by 2.) We can also canonicalize the sign of the coefficients by multiplying all the coefficients by -1 if necessary.

(defun canonicalize-lft-coefficients (a b
                                      c d receiver)
  (cond ((or (minusp c)
             (and (zerop c)
                  (minusp d)))
         ;; canonicalize sign
         (canonicalize-lft-coefficients (- a) (- b)
                                        (- c) (- d) receiver))
        ((and (evenp a)
              (evenp b)
              (evenp c)
              (evenp d))
         ;; reduce if possible
         (canonicalize-lft-coefficients (/ a 2) (/ b 2)
                                        (/ c 2) (/ d 2) receiver))
        (t (funcall receiver
                    a b
                    c d))))
  
(defun %make-lft (a b c d)
  ;; Constructor used when we know A, B, C, and D are integers.
  (canonicalize-lft-coefficients
   a b
   c d
   (lambda (a* b*
            c* d*)
     (make-instance 'lft
                    :a a* :b b*
                    :c c* :d d*))))

(defun make-lft (a b c d)
  (etypecase a
    (float (make-lft (rational a) b c d))
    (integer
     (etypecase b
       (float (make-lft a (rational b) c d))
       (integer
        (etypecase c
          (float (make-lft a b (rational c) d))
          (integer
           (etypecase d
             (float (make-lft a b c (rational d)))
             (integer (%make-lft a b
                                 c d))
             (rational (make-lft (* a (denominator d)) (* b (denominator d))
                                 (* c (denominator d)) (numerator d)))))
          (rational (make-lft (* a (denominator c)) (* b (denominator c))
                              (numerator c)         (* d (denominator c))))))
       (rational (make-lft (* a (denominator b)) (numerator b)
                           (* c (denominator b)) (* d (denominator b))))))
    (rational (make-lft (numerator a)         (* b (denominator a))
                        (* c (denominator a)) (* d (denominator a))))))

Printer

One advantage of making LFTs be funcallable CLOS objects is that you can define a print-object method on them. For my LFTs, I defined print-object to print the LFT in algabraic form. This will take a couple of hours to write because of all the edge cases, but it enhances the use of LFTs.

> (make-lft 3 2 4 -3)
#<LFT (3x + 2)/(4x - 3)>

Cases where some of the coefficients are 1 or 0.

> (make-lft 1 0 3 -2)
#<LFT x/(3x - 2)>

> (make-lft 2 7 0 1)
#<LFT 2x + 7>

> (make-lft 3 1 1 0)
#<LFT 3 + 1/x>

Application

The most mundane way to use a LFT is to apply it to a number.

> (defvar *my-lft* (make-lft 3 2 4 3))
*MY-LFT*
    
> (funcall *my-lft* 1/5)
13/19

Dividing by zero

In general, LFTs approach the limit A/C as the input x grows without bound. We can make our funcallable CLOS instance behave this way when called on the special symbol ’infinity.

> (funcall *my-lft* ’infinity)
3/4

In general, LFTs have a pole when the value of x is -D/C, which makes the denominator of the LFT zero. Rather than throwing an error, we’ll make the LFT return ’infinity

> (funcall *my-lft* -3/4))
INFINITY

Inverse LFTs

Another advantage of using a funcallable CLOS instance is that we can find the inverse of a LFT. You can compute the inverse of a LFT by swapping A and D and toggling the signs of B and C.

(defun inverse-lft (lft)
  (make-lft (slot-value lft ’d)
            (- (slot-value lft ’b))
            (- (slot-value lft ’c))
            (slot-value lft ’a)))

Composing LFTs

LFTs are closed under functional composition — if you pipe the output of one LFT into the input of another, the composite function is equivalent to another LFT. The coefficients of the composite LFT are the matrix multiply of the coefficients of the separate terms.

> (compose (make-lft 2 3 5 7) (make-lft 11 13 17 19))
#<LFT (73x + 83)/(174x + 198)>

Using LFTs as linear functions

A LFT can obviously be used as the simple linear function it is. For instance, the “multiply by 3” function is (make-lft 3 0 0 1) and the “subtract 7” function is (make-lft 1 -7 0 1). (make-lft 0 1 1 0) takes the reciprocal of its argument, and (make-lft 1 0 0 1) is just the identity function.

Using LFTs as ranges

LFTs are monotonic except for the pole. If the pole of the LFT is non-positive, and the input is non-negative, then the output of the LFT is somewhere in the range [A/C, B/D]. We can use those LFTs with a non-positive pole to represent ranges of rational numbers. The limits of the range are the LFT evaluated at zero and ’infinity.

We can apply simple linear functions to ranges by composing the LFT that represents the linear function with the LFT that represents the range. The output will be a LFT that represents the modified range. For example, the LFT (make-lft 3 2 4 3) represents the range [2/3, 3/4]. We add 7 to this range by composing the LFT (make-lft 1 7 0 1).

> (compose (make-lft 1 7 0 1) (make-lft 3 2 4 3))
#<LFT (31x + 23)/(4x + 3)>

Application (redux)

It makes sense to define what it means to funcall a LFT on another LFT as being the composition of the LFTs.

> (defvar *add-seven* (make-lft 1 7 0 1))
*ADD-SEVEN*

> (funcall *add-seven* 4)
11

> (funcall *add-seven* (make-lft 4 13 1 2))
#<LFT (11x + 27)/(x + 2)>

> (funcall * ’infinity)
11

Conclusion

This should be enough information for you to implement LFTs in a couple of hours. If you don’t want to implement them yourself, just crib my code from https://github.com/jrm-code-project/homographic/blob/main/lisp/lft.lisp

Paolo AmorosoA single package Common Lisp workflow for Medley

· 65 days ago

My exploration of Medley as a Common Lisp development environment proceeds with setting up a workflow for writing and saving code.

The workflow consists of a series of steps in a specific order using appropriate Lisp REPLs and tools. It supports writing the simplest type of Common Lisp software, i.e. programs or libraries in a single package that exports some symbols. I'll eventually extend the workflow to more complex cases such as programs with more than one package.

Since the steps are not intuitive, especially for Medley novices, in this post I'll describe the workflow in detail. But first there are a few concepts to introduce.

Why do you need a workflow in the first place? Because the differences between Medley and modern environments constrain how and in what order Common Lisp code may be written and managed. Before examining the constraints let's describe the differences.

The residential environment of Medley

Writing single package programs is straightforward in modern file based Common Lisp environments. In a new file you just define the package with DEFPACKAGE, then in the rest of the file or at the top of a new one you place a matching IN-PACKAGE followed by the code.

Although it's technically possible to do the same in Medley this doesn't take advantage of its facilities, and you may actually need to fight the system to accomplish what you want. Indeed, Medley is not an ordinary environment. Not only it predates current Common Lisp implementations, it supports a different development process.

In file based Common Lisps you directly edit source files and evaluate or load the code into the running Lisp image.

Medley instead is a "residential environment" in which you edit and evaluate Lisp objects that reside in the image — hence "residential". Then you save the code to files that are more like code databases than traditional source files.

You don't edit the code databases, which Medley calls "symbolic files". Rather, you use the SEdit structure editor to modify the code in memory and the "File Manager" to save the code to disk. The beginning of a symbolic file defines metadata, the "file environment", which describes the Common Lisp package and readtable associated with the code.

The File Manager, also known as "File Package" (not to be confused with Common Lisp packages), is a facility that coordinates the development tools and code management tasks. It notices the changes to Lisp objects edited with SEdit or manipulated in memory, tracks what changed functions and objects need to be saved to symbolic files, and carries out the actions for building programs such as compiling or listing them. The File Manager has some of the functionality of Unix Make.

Figuring what changed is easy in modern Common Lisp environments, as you know which files you edited and need action like saving or compiling. System definition tools like ASDF can track this for you. In Medley it's the File Manager which tracks the changes that take place in the running image and need to be synchronized to disk.

Motivation

How do the peculiarities of Medley constrain writing Common Lisp code and require a tailored workflow?

The functions and objects of Interlisp programs usually live in the same namespace of Interlisp and its tools. As a consequence, functions and Lisp objects may be mostly defined in any order and accessed without package qualifiers. No special handling is necessary with the File Manager either.

With Common Lisp code, however, a subtle complication arises due to packages and exported symbols.

A good explanation of why things are different, and how the File Manager and file environment interact, is in the documentation of TextModules, a tool for importing Common Lisp code created outside of Medley. Although in the context of TextModules, these remarks give an overview of the same issues the File Manager faces with other Common Lisp code. The TextModules chapter of the Lisp Library Modules manual says on page 311 (page 341 of the PDF):

It is important to separate the environment of the file from its contents because the File Manager (not TextModules) first reads all the forms in the file, and then evaluates them. Text based source files sometimes change the package as needed. This cannot work for the File Manager since the file's forms are all read and then executed, i.e. the package changes would not occur until after the entire file had been read, and forms after any IN-PACKAGE form would have been read incorrectly.

In other words, unless you define packages and access symbols in the proper order, you'll get subtle errors. The solution is a workflow that avoids such errors.

More information on dealing with packages in Medley is in the sources referenced in section "Documentation" of my post on using Common Lisp on Medley.

The workflow

How does the workflow order the development tasks to achieve its goal?

At any one time the workflow accesses only defined symbols and ensures the running Lisp image stays synchronized with the symbolic file. It's not the only or the best possible workflow, just one that works. I put it together after extensively reading the documentation and experimenting.

As I said in the overview of Medley as a Common Lisp environment, when coding in Common Lisp I use two Executives (Lisp REPLs), a Xerox Common Lisp (XCL) Exec and an Interlisp one.

The former is for testing, running, and evaluating Common Lisp code. I use the Interlisp Exec for running system tools and interacting with the File Manager. Since the tools and File Manager facilities are in the Interlisp package, referencing them from a Common Lisp Exec would require qualifying all symbols with the IL: package.

What follows assume you're familiar with basic Interlisp and File Manager features such as SEdit, file coms, FILES?, and MAKEFILE. If not I recommend reading the Medley primer, particulary Chapter 7 "Editing and Saving". Also, unless otherwise noted, you should carry out the steps in sequence in the same Medley session.

Let's start.

Defining the file environment and a minimal package

Suppose you want to write a Common Lisp program or library stored in the file SINGLEPKG. The package SINGLEPKG, nicknamed SP, will export the two functions FUN1 and FUN2.

From now on, denotes the prompt of an Interlisp Exec and > that of a Xerox Common Lisp (XCL) Exec. Sometimes I'll tell you in which Exec to evaluate expressions.

The first step is to define the file environment. At an Interlisp Exec evaluate:

← (XCL:DEFINE-FILE-ENVIRONMENT SINGLEPKG :PACKAGE (DEFPACKAGE "SINGLEPKG" (:USE "LISP" "XCL")) :READTABLE "XCL")

For now don't use other packages or export any symbols, just enter the form as is.

Although the file environment references package SINGLEPKG, the package doesn't exist yet in the running image. To synchronize the image with the file environment evaluate the definition of a minimal package from an Interlisp Exec:

← (DEFPACKAGE "SINGLEPKG" (:USE "LISP" "XCL"))

Next, from an Interlisp Exec evaluate (FILES?) and, when asked where the SINGLEPKG file info should go, respond yes, enter SINGLEPKG as the file name, and confirm the creation of the file.

Defining the first function

Everything is ready to define the first function FUN1. At an Interlisp Exec call SEdit with (ED 'SINGLEPKG::FUN1 '(FUNCTIONS :DONTWAIT)) and select DEFUN from the menu. Enter the code of FUN1:

(DEFUN FUN1 ()
  (FORMAT T "Hello from FUN1."))

Save and exit with Ctrl-Alt-X and test the function at an XCL Exec (an Interlisp Exec will do too):

> (SINGLEPKG::FUN1)
Hello from FUN1.
NIL

It works, so at an Interlisp Exec evaluate (FILES?) to associate FUN1 with the file SINGLEPKG.

Completing the package definition

The Lisp image contains the new symbol FUN1 in package SINGLEPKG but there's no symbolic file yet, let alone an exported symbol in the file. Therefore, to keep things in sync you need to update the package definition by exporting the function name. This is also an opportunity for adding the SP nickname to the package.

At an Interlisp Exec evaluate (DC SINGLEPKG) to open the file coms in SEdit. Just after the XCL:FILE-ENVIRONMENTS form enter:

(P (DEFPACKAGE "SINGLEPKG"
     (:USE "LISP" "XCL")
     (:NICKNAMES "SP")
     (:EXPORT SINGLEPKG::FUN1)))

Both colon characters : are required in the function name. The P File Manager command tells the system to execute the following Lisp expressions at load time, so loading SINGLEPKG will define the package and export the symbol.

Save and exit with Ctrl-Alt-X and, at an Interlisp Exec, save the file with (MAKEFILE 'SINGLEPKG) to reflect the updated definitions. This creates the file SINGLEPKG on disk.

Before doing anything else it's better to make sure the package is properly defined and the function exported.

The most reliable way is to exit the Medley session with (IL:LOGOUT), start a new session and, from an Interlisp Exec, evaluate (LOAD 'SINGLEPKG). If there are errors you will have to go back and check whether you went through all the steps correctly. You may need to delete the latest or all versions of the file SINGLEPKG.

Assuming there are no errors the package is properly defined and the function exported. To double check, from an XCL Exec call the exported function like this:

> (SINGLEPKG:FUN1)
Hello from FUN1.
NIL

Since it works you may proceed development by editing the existing function or defining new ones. In the former case you employ SEdit, FILES?, and MAKEFILE as usual. Just make sure to pass to ED the package-qualified function name like (ED 'SINGLEPKG:FUN1 :DONTWAIT).

Things change if you want to define new functions.

Defining more functions

As planned you proceed to define a second function FUN2 exported from package SINGLEPKG. If you're still in the Medley session in which you've just loaded the file SINGLEPKG, continue from there. Otherwise start a new session and load the file with (LOAD 'SINGLEPKG) from any Exec.

At an Interlisp Exec define the new function with (ED 'SINGLEPKG::FUN2 '(FUNCTIONS :DONTWAIT)) and select DEFUN from the menu:

(DEFUN FUN2 ()
  (FORMAT T "Hello from FUN2."))

Save and exit with Ctrl-Alt-X and test the function at an XCL Exec:

> (SINGLEPKG::FUN2)
Hello from FUN2.
NIL

Call FILES? to associated FUN2 with file SINGLEPKG.

As already done for FUN1 you need to modify the package definition to export the new function. At an Interlisp Exec edit the file coms with (DC SINGLEPKG) and add the symbol FUN2 to the export clause, which will now look like this:

(P (DEFPACKAGE "SINGLEPKG"
     (:USE "LISP" "XCL")
     (:NICKNAMES "SP")
     (:EXPORT SINGLEPKG:FUN1 SINGLEPKG::FUN2)))

SEdit replaced the double colon of SINGLEPKG::FUN1 with a single colon as in SINGLEPKG:FUN1. But you still have to type :: for FUN2 because FUN2 hasn't been exported yet.

Save and exit with Ctrl-Alt-X and, at an Interlisp Exec, save the file with (MAKEFILE 'SINGLEPKG).

To synchronize the Lisp image with the symbolic file, which will allow to call the exported function as (SINGLEPKG:FUN2), either load the file SINGLEPKG from a fresh session or, just after editing the file coms, evaluate a revised package definition at an Interlisp Exec:

← (DEFPACKAGE "SINGLEPKG"
    (:USE "LISP" "XCL")
    (:NICKNAMES "SP")
    (:EXPORT SINGLEPKG::FUN1 SINGLEPKG::FUN2))

Either way, to check that everything works evaluate at an XCL Exec:

> (SINGLEPKG:FUN2)
Hello from FUN2.
NIL

Success!

For every new function or Lisp object you want to export from the package, go through the steps of this section again, making sure the Lisp image and the symbolic file stay synchronized. The steps are, in order:

  1. edit the new function
  2. call FILES? to tell the File Manager about the function
  3. edit the file coms to update the package
  4. save the file with MAKEFILE
  5. evaluate the revised package definition

That's all, you're finally ready to develop single package programs. This workflow may seem convoluted at first but things will come more natural as you gain experience with Medley.

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

Marco AntoniottiA Parenthetical Year and New Open Parentheses

· 74 days ago

It has been a few years that I have spent the New Year Holiday to update my Common Lisp libraries and to think about all the open parentheses that are left unclosed. This year is no different, only I have had even less time to work on CL. Therefore, my updates to my libraries on common-lisp.net and Sourceforge all have had some minor repairs and copyright updates. As usual, I direct you to HEΛP and CLAST; most of my other libraries are support for these two.

One effort, I would like to finalize in the coming year (that is; closing this parenthesis) is the CL-LIA layer/library. Help is wanted for this! Drop me a line if you want to chip in. The main issues are API designs around the floating point environment condition handling.

Apart from that, I have been looking more and more to the future, that is, Julia (I should have listened more to some people in the CL community years ago), and... in the past.

Julia and Cancer Research

I have advised a student to work on Julia and the result is this nice (very rough) simulator for solid tumors with input from images and tracking of cells lineages: SOPHYSM. SOPHYSM builds on J-Space which is the simulator core; you can find the publication at BMC Bioinformatics.

I am currently pushing Julia to my research group, especially for new AI and ML applications to Data Analisys in Cancer Research (cfr., Data and Computational Biology Laboratory at Università degli Studi di Milano-Bicocca). This is because new Julia ML and AI frameworks have become much more solid in recent months; in particular Flux and SciML.ai.

Programming Archeology

If you have followed my older posts, you know I fell into a rabbit hole (with several side tunnels). I have been dabbling with very old technology, which turned out to be ... fun (don't tell my Department Head!).

I will just say that now I can understand the vagaries of PL/I (where do you think that loop and generic functions come from?) and that I can write decent Fortran IV (66) (downgrading from Fortran 77). And yes! I can compile my program on Hercules (of course using Emacs to submit jobs to MVS TK5; some work required).

Happy Hacking and Science Year!

That's it. These are my report for 2023, and my wishes for you all in 2024.

Just one more thing: you reader know that I am becoming and old geezer. I somehow rationalize my dabbling with parentheses and even older stuff as "taking care" of things that should not be lost; even if now we have Julia.

Let's all try to take care of each other and the world, of the old things and of the new ones that will inherit it.


'(peace)

Paolo AmorosoMy Common Lisp setup on Linux

· 82 days ago

Now that I'm back to Lisp I'm actively exploring Interlisp as a Common Lisp environment too.

But to code in Common Lisp also on my Crostini Linux system, the Linux container of chromeOS I use on a Chromebox, I'm setting up a suitable development environment. In addition to console programs I want to write GUI applications with McCLIM.

The Common Lisp implementation I chose, SBCL, is a no brainer given its features, performance, and active maintenance. As for the development environment I won't go with the default on Linux, Emacs. Although I used Emacs for years and loved it, now it feels overkill and I'd prefer not to re-learn its intricacies.

Instead I'm using Lem, a great Emacs-like IDE written in Common Lisp with a user interface and keybindings similar to those of Emacs and SLIME. Thus my familiarity with Emacs is enough to get me up to speed. A nice side effect of Lem's implementation language is the IDE can be configured and extended in Common Lisp, which I feel more at home with than Emacs Lisp.

Despite some initial installation issues Lem works well on Crostini, is fast, and has a nice SDL2 backend. I really like the IDE.

If I need to write or run some Common Lisp code on my Raspberry Pi 400 I can easily replicate this setup there.

#CommonLisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

vindarelLatest string manipulation functions in the STR library

· 87 days ago

We just released cl-str v0.21. It’s been a while since the last release, and many enhancements make it more useful than ever. Let’s review the changes, the newest first.

But first, I want to you thank everyone who contributed, by sending pull requests or feedback. Special thanks to @kilianmh who suddenly appeared one day, helped with new features as well as grunt work, and who is now a co-maintainer.

split by regex

The latest addition sent by ccQpein is that str:split now accepts a :regex key argument to split by regular expressions. The functions rsplit and split-omit-nulls have it too.

(str:split "[,|;]" "foo,bar;baz" :regex t)
;; => ("foo" "bar" "baz")

That’s handy for advent of code ;)

You can also use ppcre:split, this is the function that str:split relies on anyways, except that by default, str:split ensures that the split argument is not a regex. We need this:

(ppcre:split `(:sequence ,(string separator)) s ...)

(and eventually, we remove null elements if :omit-nulls t was set)

replace by regex

str:replace-all, str:replace-first and str:replace-using got a :regex keyword too:

(str:replace-all "(?i)fo+" "frob" "FOO bar FOO" :regex t)
;; => "frob bar frob"

The ensure functions

These were added in March.

The “ensure-” functions return a string that has the specified prefix or suffix, appended if necessary.

ensure encapsulates the other two.

ensure-prefix, ensure-suffix (start/end s)

Ensure that s starts with start/end (or ends with start/end, respectively).

Return a new string with its prefix (or suffix) added, if necessary.

Example:

(str:ensure-prefix "/" "abc/") => "/abc/" (a prefix was added)
;; and
(str:ensure-prefix "/" "/abc/") => "/abc/" (does nothing)

We also have a couple functions to find the prefixes or the suffixes, please see our README.

ensure-wrapped-in (start/end s)

Ensure that s both starts and ends with start/end.

Return a new string with the necessary added bits, if required.

It simply calls str:ensure-suffix followed by str:ensure-prefix.

See also str:wrapped-in-p and uiop:string-enclosed-p prefix s suffix.

(str:ensure-wrapped-in "/" "abc") ;; => "/abc/"  (added both a prefix and a suffix)
(str:ensure-wrapped-in "/" "/abc/") ;; => "/abc/" (does nothing)

ensure (s &key wrapped-in prefix suffix)

This str:ensure function looks for the following key parameters, in order:

  • :wrapped-in: if non nil, call str:ensure-wrapped-in. This checks that s both starts and ends with the supplied string or character.
  • :prefix and :suffix: if both are supplied and non-nil, call str:ensure-suffix followed by str:ensure-prefix.
  • :prefix: call str:ensure-prefix
  • :suffix: call str:ensure-suffix.

Example:

(str:ensure "abc" :wrapped-in "/")  ;; => "/abc/"
(str:ensure "/abc" :prefix "/")  ;; => "/abc"  => no change, still one "/"
(str:ensure "/abc" :suffix "/")  ;; => "/abc/" => added a "/" suffix.

These functions accept strings and characters:

(str:ensure "/abc" :prefix #\/)

warn: if both :wrapped-in and :prefix (and/or :suffix) are supplied together, :wrapped-in takes precedence and :prefix (and/or :suffix) is ignored.

:char-bag parameter to trim, trim-left, trim-right

This was added in January.

str:trim removes all characters in char-bag (default: whitespaces) at the beginning and end of s.

If supplied, char-bag has to be a sequence (e.g. string or list of characters).

(str:trim "cdoooh" :char-bag (str:concat "c" "d" "h")) => "ooo"

fit a string to some length

This is older, it was added in February of 2022.

Fit this string to the given length:

  • if it’s too long, shorten it (showing the ellipsis),
  • if it’s too short, add paddding (to the side pad-side, adding the character pad-char).

As such, it accepts the same key arguments as str:shorten and str:pad: ellipsis, pad-side, pad-char...

CL-USER> (str:fit 10 "hello" :pad-char "+")
"hello+++++"

CL-USER> (str:fit 10 "hello world" :ellipsis "...")
"hello wor..."

If, like me, you want to print a list of data as a table, see:

CL-USER> (ql:quickload "cl-ansi-term")
CL-USER> (term:table '(("name" "age" "email")
              ("me" 7 "some@blah")
              ("me" 7 "some@with-some-longer.email"))
             :column-width '(10 4 20))
+---------+---+-------------------+
|name     |age|email              |
+---------+---+-------------------+
|me       |7  |some@blah          |
+---------+---+-------------------+
|me       |7  |some@with-some-l(...)|
+---------+---+-------------------+
CL-USER> (ql:quickload "cl-ascii-table")
CL-USER> (let ((table (ascii-table:make-table '("Id" "Name" "Amount") :header "Infos")))
  (ascii-table:add-row table '(1 "Bob" 150))
  (ascii-table:add-row table '(2 "Joe" 200))
  (ascii-table:add-separator table)
  (ascii-table:add-row table '("" "Total" 350))
  (ascii-table:display table))

.---------------------.
|        Infos        |
+----+-------+--------+
| Id | Name  | Amount |
+----+-------+--------+
|  1 | Bob   |    150 |
|  2 | Joe   |    200 |
+----+-------+--------+
|    | Total |    350 |
+----+-------+--------+
NIL

fixed string-case

The str:string-case macro was missing an implicit progn, so with more than one s-expression in the clauses, it didn’t fail... but it didn’t work as expected either.

Fixed for LispWorks

Characters are named differently, like #\NewLine. We are still awaiting input on one issue.

We reimplemented str:replace-using to fix it on LispWorks.

Misc

We added type declaration e.g. for concat, join.

str:ends-with-p now works with a character.

Small breaking change: fixed str:prefixp when used with a smaller prefix: “f” was not recognized as a prefix of “foobar” and “foobuz”, only “foo” was. Now it is fixed. Same for str:suffixp.

We added str:ascii-p and str:ascii-char-p (in 2021).

More functions now work with characters as well.

We sped up str:join (measured: 4x). We use with-output-to-string and a loop instead of format’s iteration directive.

We use uninterned symbols in defpackage.

We deprecated predicates ending with “?” (but they are still there).

We made casing-functions consistent to inbuilt cl casing functions (we use cl-change-case, but the functions also allow symbols and characters (not only strings) and return NIL when given NIL).

We added :ignore-case to str:count-substring.

We switched the testing framework from prove to fiveam (that was grunt work by the new maintainer yay o/ )


That’s it, thanks again for helping make this lil’ lib useful since day 1.

The “str” library defines many more functions. Look at our table of content on the README: https://github.com/vindarel/cl-str

Install it with

(ql:quickload "str")

Yukari HafnerThe State of MacOS Support

· 112 days ago
https://studio.tymoon.eu/api/studio/file?id=2517

I've been writing libraries for Common Lisp for over a decade now (lord almighty), and for most of that time I've tried to ensure that the libraries would, in the very least, work on all three major operating systems: Windows, Linux, and MacOS.

Usually doing so isn't hard, as I can rely on the implementation and the language standard, but especially for libraries that deal with foreign code or operating system interfaces, a bit more work is needed. For the longest time I went the extra mile of providing that support myself, despite not being a MacOS user, and despite vehemently disapproving of Apple as a company and their treatment of users and developers.

About two years ago, I stopped. I had had enough of all the extra work the platform put on me, for zero personal gain. Especially I had had enough of all the extra work it continued to put on me for things that had already been working before. The amount of work only ever increased, with barely any thanks or compensation for this work. Apple's war against its own users and developers only ever increased as well.

I cannot in good conscience support MacOS, but I understand that a lot of people are stuck on that platform for one reason or another, and I do not wish to punish them, either. However, I lack a working MacOS setup these days, especially for the newer M1/2/3 systems. And so I appeal to you, MacOS users: if you have any interest in any of the following libraries, please contribute patches.

Requiring only C library builds:

The C library projects should not be much work to fix, with the binaries for AMD64/ARM64 they should pretty much be done. A couple require Lisp patches, though:

  • file-notify
    The darwin implementation is buggy and I don't know why. The documentation for MacOS sucks.

  • machine-state
    Needs testing of the posix APIs and possibly darwin-specific fixups

  • Kandria
    No idea how much needs doing here, probably a bunch of backend specific things to test and implement in Trial.

If you decide to contribute, I'm sure a lot of your fellow MacOS users would be very thankful!

And if you like what I do in general, please consider supporting my work on Patreon!

Joe MarshallGitHub Co-pilot Review

· 115 days ago

I recently tried out GitHub CoPilot. It is a system that uses generative AI to help you write code.

The tool interfaces to your IDE — I used VSCode — and acts as an autocomplete on steroids … or acid. Suggested comments and code appear as you move the cursor and you can often choose from a couple of different completions. The way to get it to write code was to simply document what you wanted it to write in a comment. (There is a chat interface where you can give it more directions, but I did not play with that.)

I decided to give it my standard interview question: write a simple TicTacToe class, include a method to detect a winner. The tool spit out a method that checked an array for three in a row horizontally, vertically, and along the two diagonals. Almost correct. While it would detect three ‘X’s or ‘O’s, it also would detect three nulls in a row and declare null the winner.

I went into the class definition and simply typed a comment character. It suggested an __init__ method. It decided on a board representation of a 1-dimensional array of 9 characters, ‘X’ or ‘O’ (or null), and a character that determined whose turn it was. Simply by moving the cursor down I was able to get it to suggest methods to return the board array, return the current turn, list the valid moves, and make a move. The suggested code was straightforward and didn’t have bugs.

I then decided to try it out on something more realistic. I have a linear fractional transform library I wrote in Common Lisp and I tried porting it to Python. Co-pilot made numerous suggestions as I was porting, to various degrees of success. It was able to complete the equations for a 2x2 matrix multiply, but it got hopelessly confused on higher order matrices. For the print method of a linear fractional transform, it produced many lines of plausible looking code. Unfortunately, the code has to be better than “plausible looking” in order to run.

As a completion tool, co-pilot muddled its way along. Occasionally, it would get a completion impressively right, but just as frequently — or more often — it would get the completion wrong, either grossly or subtly. It is the latter that made me nervous. Co-pilot would produce code that looked plausible, but it required a careful reading to determine if it was correct. It would be all too easy to be careless and accept buggy code.

The code Co-Pilot produced was serviceable and pedestrian, but often not what I would have written. I consider myself a “mostly functional” programmer. I use mutation sparingly, and prefer to code by specifying mappings and transformations rather than sequential steps. Co-pilot, drawing from a large amount of code written by a variety of authors, seems to prefer to program sequentially and imperatively. This isn’t surprising, but it isn’t helpful, either.

Co-pilot is not going to put any programmers out of work. It simply isn’t anywhere near good enough. It doesn’t understand what you are attempting to accomplish with your program, it just pattern matches against other code. A fair amount of code is full of patterns and the pattern matching does a fair job. But exceptions are the norm, and Co-pilot won’t handle edge cases unless the edge case is extremely common.

I found myself accepting Co-pilot’s suggestions on occasion. Often I’d accept an obviously wrong suggestion because it was close enough and the editing seemed less. But I always had to guard against code that seemed plausible but was not correct. I found that I spent a lot of time reading and considering the code suggestions. Any time savings from generating these suggestions was used up in vetting the suggestions.

One danger of Co-pilot is using it as a coding standard. It produces “lowest common denominator” code — code that an undergraduate that hadn’t completed the course might produce. For those of us that think the current standard of coding is woefully inadequate, Co-pilot just reinforces this style of coding.

Co-pilot is kind of fun to use, but I don’t think it helps me be more productive. It is a bit quicker than looking things up on stackoverflow, but its results have less context. You wouldn’t go to stackoverflow and just copy code blindly. Co-pilot isn’t quite that — it will at least rename the variables — but it produces code that is more likely buggy than not.

Nicolas MartyanoffInteractive Common Lisp development

· 120 days ago

Common Lisp programming is often presented as “interactive”. In most languages, modifications to your program are applied by recompiling it and restarting it. In contrast, Common Lisp lets you incrementally modify your program while it is running.

While this approach is convenient, especially for exploratory programming, it also means that the state of your program during execution does not always reflect the source code. You do not just define new constructs: you look them up, inspect them, modify them or delete them. I had to learn a lot of subtleties the hard way. This article is a compendium of information related to the interactive nature of Common Lisp.

Variables

In Common Lisp variables are identified by symbols. Evaluating (SETQ A 42) creates or updates a variable with the integer 42 as value, and associates it to the A symbol. After the call to SETQ, (BOUNDP 'A) will return T and (SYMBOL-VALUE 'A) will return 42.

You do not delete a variable: instead, you remove the association between the symbol and the variable. You do so with MAKUNBOUND. Following the previous example, (MAKUNBOUND 'A) will remove the association between the A symbol and the variable. And (BOUNDP 'A) returns NIL as expected. As for (SYMBOL-VALUE 'A), it now signals an UNBOUND-VARIABLE error as mandated by the standard.

What about DEFVAR and DEFPARAMETER? They are also used to declare variables (globally defined ones), associating them with symbols. Both define “special” variables (i.e. variables for which all bindings are dynamic; see CLtL2 9.2). The difference is that the initial value passed to DEFVAR is not evaluated if it already has a value. MAKUNBOUND will work on variables declared with DEFVAR or DEFPARAMETER as expected.

DEFCONSTANT is a bit more complicated. CLtL21 5.3.2 states that “once a name has been declared by defconstant to be constant, any further assignment to or binding of that special variable is an error”, but does not clearly define whether MAKUNBOUND should or should not be able to be used on constants. However, CLtL2 5.3.2 also states that “defconstant [...] does assert that the value of the variable name is fixed and does license the compiler to build assumptions about the value into programs being compiled”. If the compiler is allowed to rely on the value associated with the variable name, it would make sense not to allow the deletion of the binding. Thus it is recommended to only use constants for values that are guaranteed to never change, e.g. mathematical constants. Most of the time you want DEFPARAMETER.

Note that MAKUNBOUND does not apply to lexical variables.

Functions

Common Lisp is a Lisp-2, meaning that variables and functions are part of two separate namespaces. Despite this clear separation, functions behave similarly to variables.

Using DEFUN will either create or update the global function associated with a symbol. SYMBOL-FUNCTION returns the globally defined function associated with a symbol, and FMAKUNBOUND deletes this association.

Let us point out a common mistake when referencing functions: (QUOTE F) (abbreviated as 'F) yields a symbol while (FUNCTION F) (abbreviated as #'F) yields a function. The function argument of FUNCALL and APPLY can be either a symbol or a function (see CLtL2 7.3) It has two consequences:

First, one can write a function referencing F as (QUOTE F) with the expectation that F will later be bound to a function. The following function definition is perfectly valid even though F has not been defined yet:

(defun foo (a b)
  (funcall 'f a b))

Second, redefining the F function will update its association (or binding) to the F symbol, but the previous function will still be available if it has been referenced somewhere before the update. For example:

(setf (symbol-function 'foo) #'1+)
(let ((old-foo #'foo))
  (setf (symbol-function 'foo) #'1-)
  (funcall old-foo 42))

What about macros? Since macros are a specific kind of functions (CLtL2 5.1.4 “a macro is essentially a function from forms to forms”), it is not surprising that they share the same namespace and can be manipulated in the same way as functions with FBOUNDP, SYMBOL-FUNCTION and FMAKUNBOUND.

Symbols and packages

While functions and variables are familiar concepts to developers, Common Lisp symbols and packages are a bit more peculiar.

A symbol is interned when it is part of a package. The most explicit way to create an interned symbol is to use INTERN, e.g. (INTERN "FOO"). INTERN interns the symbol in the current package by default, but one can pass a package as second argument. After that, (FIND-SYMBOL "FOO") will return our interned symbol as expected.

More surprisingly, the reader automatically interns symbols. You can test it by evaluating (READ-FROM-STRING "BAR"). After evaluation, BAR is a symbol interned in the current package. This also means that it is very easy to pollute a package with symbols in ways you did not necessarily expect. To clean up, simply use UNINTERN. Remember to refer to the right symbol: to remove the symbol BAR from the package FOO, use (UNINTERN 'FOO::BAR "BAR").

A symbol is either internal or external. EXPORT will make a symbol external to its package while UNEXPORT will make it internal. As for UNINTERN, confusion usually arises around which symbol is affected. (UNEXPORT 'FOO:BAR "FOO") correctly refers to the external symbol in the FOO package and makes it internal again. (UNEXPORT 'BAR "FOO") will signal an error since the BAR symbol is not part of the FOO package (unless of course the current package happens to be FOO).

Packages themselves can be created with MAKE-PACKAGE and destroyed with DELETE-PACKAGE. Developers are usually more familiar with DEFPACKAGE, a macro allowing the creation of a package and its configuration (package use list, imported and exported symbols, etc.) in a declarative way. A surprising and frustrating behavior is that evaluating a DEFPACKAGE form for a package that already exists will result in undefined behavior if the new declaration “is not consistent” (CLtL2 11.7) with the current state of the package. As an example, adding symbols to the export list is perfectly fine. Removing one will result in undefined behavior (usually an error) due to the inconsistency of the export list. Fortunately, Common Lisp offers all the necessary functions to manipulate packages and their symbols: use them!

Classes

The Common Lisp standard includes CLOS, the Common Lisp Object System. Unsurprisingly it provides multiple ways to interact with classes and objects dynamically.

As variables or functions, classes are identified by symbols and FIND-CLASS returns the class associated with a symbol. Class names are part of a separate namespace shared with structures and types.

The DEFCLASS macro is the only way to define or redefine a class. Redefining a class means that instances created afterward with MAKE-INSTANCE will use the new definition. Existing instances are updated: newly added slots are added (either unbound or using the value associated with :INITFORM) and slots that are not defined anymore are deleted. UPDATE-INSTANCE-FOR-REDEFINED-CLASS is particularly interesting: developers can define methods for this generic function in order to control how instances are updated when their class is redefined.

Defining classes may imply implicitly defining methods: the :ACCESSOR, :READER and :WRITER slot keyword arguments will lead to the creation of generic functions. When a class is redefined, methods associated with slots that have been removed will live on.

A limitation of CLOS is that classes cannot be deleted. FIND-CLASS can be used as a place, and (SETF (FIND-CLASS 'FOO) NIL) will remove the association between the FOO symbol and the class, but the class itself and its instances will not disappear. While this limitation may seem strange, ask yourself how an implementation should handle instances of a class that has been deleted.

The class of an instance can be changed with CHANGE-CLASS: slots that exist in the new class will be conserved while those that do not are deleted. New slots are either unbound or set to the value associated with :INITFORM in the new class. In a way similar to UPDATE-INSTANCE-FOR-REDEFINED-CLASS, UPDATE-INSTANCE-FOR-DIFFERENT-CLASS lets developers control precisely the process.

Generics and methods

Generics are functions which can be specialized based on the class (and not type as one could expect) of their arguments and which can have a method combination type.

Generics can be created explicitly with DEFGENERIC or implicitly when DEFMETHOD is called and the list of parameter specializers and method combination does not match any existing generic function. Since generics are functions, FBOUNDP, SYMBOL-FUNCTION and FMAKUNBOUND will work as expected.

Methods themselves are either defined as part of the DEFGENERIC call or separately with DEFMETHOD. Discovering the different methods associated with a generic function is a bit more complicated. There is no standard way to list the methods associated with a generic, but it is at least possible to look up a method with FIND-METHOD. Do remember to pass a function (and not a symbol) as the generic, and to pass classes (and not symbols naming classes) in the list of specializers.

Redefinition is not as obvious as for non-generic functions. When redefining a generic with DEFGENERIC all methods defined as part of the previous DEFGENERIC form are removed and methods defined in the redefinition are added. However, methods defined separately with DEFMETHOD are not affected.

For example, in the following code, the second call to DEFGENERIC will replace the two methods specialized on INTEGER and FLOAT respectively by a single one specialized on a STREAM, but the method specialized on STRING will remain unaffected.

(defgeneric foo (a)
  (:method ((a integer))
    (format nil "~A is an integer" a))
  (:method ((a float))
    (format nil "~A is a float" a)))

(defmethod foo ((a string))
  (format nil "~S is a string" a))

(defgeneric foo (a)
  (:method ((a stream))
    (format nil "~A is a stream" a)))

Note that trying to redefine a generic with a different parameter lambda list will cause the removal of all previously defined methods since none of them can match the new parameters.

Removing a method will require you to find it first using FIND-METHOD and then use REMOVE-METHOD. With the previous example, removing the method specialized on a STRING argument is done with:

(remove-method #'foo (find-method #'foo nil (list (find-class 'string)) nil))

Working with methods is not always easy, and two errors are very common.

First, remember that changing the combinator in a DEFMETHOD will define a new method. If you realize that your :AFTER method should use :AROUND and reevaluate the DEFMETHOD form, remember to delete the method with the :AFTER combinator or you will end up with two methods being called.

Second, when defining a method for a generic from another package, remember to correctly refer to the generic. If you want to define a method on the BAR generic from package FOO, use (DEFMETHOD FOO:BAR (...) ...) and not (DEFMETHOD BAR (...) ...). In the latter case, you will define a new BAR generic in the current package.

Meta Object Protocol

While CLOS is already quite powerful, various interactions are impossible. One cannot create classes or methods programmatically, introspect classes or instances for example to list their slots or obtain all their superclasses, or list all the methods associated with a generic function.

In addition of an example of a CLOS implementation, The Art of the Metaobject Protocol2 defines multiple extensions to CLOS including metaclasses, metaobjects, dynamic class and generic creation, class introspection and much more.

Most Common Lisp implementations implement at least part of these extensions, usually abbreviated as “MOP”, for “MetaObject Protocol”. The well-known closer-mop system can be used as a compatibility layer for multiple implementations.

Structures

Structures are record constructs defined with DEFSTRUCT. At a glance they may seem very similar to classes, but they have a fundamental limitation: the results of redefining a structure are undefined (CLtL2 19.2).

While this property allows implementations to handle structures in a more efficient way than classes, it makes structures unsuitable for incremental development. As such, they should only be used as a last resort, when a regular class has been proved to be a performance bottleneck.

Conditions

While conditions look very similar to classes the Common Lisp standard does not define them as classes. This is one of the few differences between the standard and CLtL2 which clearly states in 29.3.4 that “Common Lisp condition types are in fact CLOS classes, and condition objects are ordinary CLOS objects”.

This is why one uses DEFINE-CONDITION instead of DEFCLASS and MAKE-CONDITION instead of MAKE-INSTANCE. This also means that one should not use slot-related functions (including the very useful WITH-SLOTS macro) with conditions.

In practice, most modern implementations follow CLtL2 and the CLOS-CONDITIONS:INTEGRATE X3J13 Cleanup Issue and implement conditions as CLOS classes, meaning that conditions can be manipulated and redefined as any other classes. And the same way as any other classes, they cannot be deleted.

Types

Types are identified by symbols and are part of the same namespace as classes (which should not be surprising since defining a class automatically defines a type with the same name).

Types are defined with DEFTYPE, but documentation is surprisingly silent on the effects of type redefinition. This can lead to interesting situations. On some implementations (e.g. SBCL and CCL), if a class slot is defined as having the type FOO, redefining FOO will not be taken into account and the type checking operation (which is not mandated by the standard) will use the previous definition of the type. Infortunatly Common Lisp does not mandate any specific behavior on slot type mismatches (CLtL2 28.1.3.2).

Thus developers should not expect any useful effect from redefining types. Restarting the implementation after substantial type changes is probably best.

In the same vein interactions with types are very limited. You cannot find a type by its symbol or even check whether a type exists or not. Calling TYPE-OF on a value will return a type this value satisfies, but the nature of the type is implementation-dependent (CLtL2 4.9): it could be any supertype. In other words, TYPE-OF could absolutly return T for all values but NIL. At least SUBTYPE-P lets you check whether a type is a subtype of another type.

Going further

Common Lisp is a complex language with a lot of subtleties, way more than what can be covered in a blog post. The curious reader will probably skip the standard (not because you have to buy it, but because it is a low quality scan of a printed document and jump directly to CLtL2 or the Common Lisp HyperSpec. The Art of the Metaobject Protocol is of course the normative reference for the CLOS extensions usually referred to as “MOP”.


  1. Guy L. Steele Jr. Common Lisp the Language, 2nd edition. 1990. ↩︎

  2. Gregor Kiczales, Jim des Rivieres and Daniel G. Bobrow. The Art of the Metaobject Protocol. 1991. ↩︎

Quicklisp newsOctober 2023 Quicklisp dist update now available

· 141 days ago

 New projects

  • 3d-math — A library implementing the necessary linear algebra math for 2D and 3D computations — zlib
  • ansi-test-harness — A testing harness that fetches ansi-test and allows subsets and extrinsic systems — MIT
  • babylon — Jürgen Walther's modular, configurable, hybrid knowledge engineering systems framework for Common Lisp, restored from the CMU AI Repository. — MIT
  • calm — CALM - Canvas Aided Lisp Magic — GNU General Public License, version 2
  • cffi-object — A Common Lisp library that enables fast and convenient interoperation with foreign objects. — Apache-2.0
  • cffi-ops — A library that helps write concise CFFI-related code. — Apache-2.0
  • cl-brewer — Provides CI settings for cl-brewer. — Unlicense
  • cl-jwk — Common Lisp system for decoding public JSON Web Keys (JWK) — BSD 2-Clause
  • cl-plus-ssl-osx-fix — A fix for CL+SSL library paths on OSX needed when you have Intel and Arm64 Homebrew installations. Should be loaded before CL+SSL. — Unlicense
  • cl-server-manager — Manage port-based servers (e.g., Swank and Hunchentoot) through a unified interface. — MIT
  • cl-transducers — Ergonomic, efficient data processing. — LGPL-3.0-only
  • cl-transit — Transit library for Common Lisp — MIT
  • clog-collection — A set of CLOG Plugins — MIT
  • clohost — A client library for the Cohost API — zlib
  • deptree — ASDF systems dependency listing and archiving tool for Common Lisp — MIT
  • enhanced-unwind-protect — Provides an enhanced UNWIND-PROTECT that makes it easy to detect whether the protected form performed a non-local exit or returned normally. — Unlicense
  • file-lock — File lock library on POSIX systems — MIT License
  • fmcs — Flavors Meta-Class System (FMCS) for Demonic Metaprogramming in Common Lisp, an alternative to CLOS+MOP, restored from the CMU AI Repository. — MIT
  • fuzzy-dates — A library to fuzzily parse date strings — zlib
  • glfw — An up-to-date bindings library to the most recent GLFW OpenGL context management library — zlib
  • hyperlattices — Generalized Lattice algebraic datatypes, incl., LATTICE, HYPERLATTICE, PROBABILISTIC-LATTICE, and PROBABILISTIC-HYPERLATTICE. — MIT
  • lemmy-api — Most recently generated bindings to the lemmy api — GPLv3
  • manifolds — Various manifold mesh algorithms — zlib
  • mutils — A collection of Common Lisp modules. — MIT
  • ptc — Proper Tail Calls for CL — MIT
  • type-templates — A library for defining and expanding templated functions — zlib

Updated projects: 3bmd, 3d-matrices, 3d-quaternions, 3d-spaces, 3d-transforms, 3d-vectors, 40ants-asdf-system, 40ants-slynk, action-list, adhoc, adp, alexandria, also-alsa, anypool, april, architecture.builder-protocol, array-operations, array-utils, asdf-flv, async-process, atomics, bdef, bike, binary-structures, binding-arrows, bordeaux-threads, bp, bubble-operator-upwards, cari3s, cephes.cl, cffi, chirp, chlorophyll, chunga, ci, cl+ssl, cl-6502, cl-all, cl-async, cl-atelier, cl-autowrap, cl-bcrypt, cl-bmp, cl-change-case, cl-clon, cl-collider, cl-colors2, cl-confidence, cl-containers, cl-cron, cl-data-structures, cl-dbi, cl-digraph, cl-fast-ecs, cl-fbx, cl-flac, cl-forms, cl-gamepad, cl-gists, cl-glib, cl-gltf, cl-gobject-introspection, cl-gobject-introspection-wrapper, cl-gopher, cl-gpio, cl-gserver, cl-hash-util, cl-html-parse, cl-i18n, cl-isaac, cl-jingle, cl-jsonl, cl-k8055, cl-kanren, cl-ktx, cl-lib-helper, cl-liballegro, cl-liballegro-nuklear, cl-markless, cl-marshal, cl-messagepack, cl-migratum, cl-mixed, cl-mlep, cl-modio, cl-moneris, cl-monitors, cl-mount-info, cl-mpg123, cl-naive-store, cl-opus, cl-out123, cl-patterns, cl-pdf, cl-permutation, cl-project, cl-protobufs, cl-pslib, cl-pslib-barcode, cl-rashell, cl-readline, cl-rfc4251, cl-sdl2, cl-sdl2-image, cl-sendgrid, cl-skkserv, cl-soloud, cl-spidev, cl-ssh-keys, cl-steamworks, cl-str, cl-tcod, cl-tiled, cl-tls, cl-utils, cl-veq, cl-voipms, cl-vorbis, cl-wavefront, cl-webkit, cl-webmachine, cl-wol, cl-yxorp, clack, classowary, clingon, clip, clog, closer-mop, clss, clunit2, codex, coleslaw, colored, com-on, common-lisp-jupyter, computable-reals, conduit-packages, croatoan, crypto-shortcuts, ctype, cytoscape-clj, dartsclhashtree, data-frame, data-lens, data-table, datafly, datamuse, decompress, deeds, deferred, definitions, deploy, depot, dexador, dissect, djula, dml, dns-client, doc, documentation-utils, drakma, dynamic-classes, easter-gauss, easy-routes, ecclesia, eclector, enhanced-eval-when, enhanced-multiple-value-bind, erjoalgo-webutil, extensible-compound-types, f2cl, fare-scripts, fast-http, feeder, file-attributes, file-notify, file-select, filesystem-utils, fiveam, fiveam-matchers, flare, float-features, flow, font-discovery, for, form-fiddle, function-cache, functional-trees, gendl, github-api-cl, glsl-toolkit, gtirb-capstone, gtwiwtg, harmony, helambdap, humbler, hunchentoot-errors, iclendar, imago, inkwell, ironclad, journal, json-mop, jsonrpc, jzon, kekule-clj, khazern, lack, lambda-fiddle, language-codes, lass, legion, legit, let-over-lambda, lev, lichat-ldap, lichat-protocol, lichat-serverlib, lichat-tcp-client, lichat-tcp-server, lichat-ws-server, lift, lisp-binary, lisp-critic, lisp-interface-library, lisp-pay, lisp-stat, local-time, lquery, lru-cache, luckless, macro-level, maiden, math, mcclim, memory-regions, messagebox, mgl-mat, mgl-pax, mito, mmap, mnas-path, mnas-string, modularize, modularize-hooks, modularize-interfaces, multilang-documentation, multiposter, mutility, named-readtables, nibbles, ningle, nodgui, north, numerical-utilities, numpy-file-format, nytpu.lisp-utils, omglib, one-more-re-nightmare, openapi-generator, orizuru-orm, osicat, ospm, oxenfurt, pango-markup, parachute, parseq, pathname-utils, petalisp, piping, plot, plump, plump-bundle, plump-sexp, plump-tex, policy-cond, posix-shm, postmodern, ppath, prettier-builtins, promise, psychiq, punycode, purgatory, py4cl2-cffi, qlot, quickhull, random-state, ratify, reblocks, reblocks-auth, reblocks-prometheus, redirect-stream, rove, s-dot2, sc-extensions, scribble, sel, serapeum, sha3, shasht, shop3, si-kanren, simple-inferiors, simple-tasks, sketch, slite, sly, softdrink, south, speechless, spinneret, staple, statistics, stopclock, studio-client, stumpwm, sxql, system-locale, terrable, testiere, tfeb-lisp-hax, tfeb-lisp-tools, tiny-routes, tooter, trivial-arguments, trivial-benchmark, trivial-clipboard, trivial-custom-debugger, trivial-extensible-sequences, trivial-garbage, trivial-gray-streams, trivial-indent, trivial-main-thread, trivial-mimes, trivial-sanitize, trivial-thumbnail, trivial-timeout, trivial-utf-8, trucler, try, typo, uax-14, uax-9, ubiquitous, unboxables, vellum, vellum-csv, vellum-postmodern, verbose, websocket-driver, woo, xmls, yah, zippy.

Removed projects: cl-bson, cl-fastcgi, more-cffi, myweb, parse-number-range, quilc, qvm.

To get this update, use (ql:update-dist "quicklisp")

What's up with Quicklisp updates taking way longer than usual? A couple things.

First, life has been pretty crazy for me, and I'm the only one working on Quicklisp updates. If anyone wants to collaborate, please let me know. There are some simple things that could improve the time between releases.

Second, there are now enough things in Quicklisp that every month something is broken at a critical time when I'm planning a release. I need to work around this with some better management software, but that takes time and things are pretty crazy for me (see above).

I hope to get back on track for regular monthly releases soon. Thanks for your support and thanks for using Quicklisp.

Eugene ZaikonnikovAnnouncing deptree

· 148 days ago

Deptree is a tool to list and archive dependency snapshots of (ASDF-defined) projects. We at Norphonic use it in the product build pipeline, but it can be useful for integration workflows as well. The task sounds common enough so there's little doubt am reinventing the wheel with this. Alas, I couldn't find any readily available solutions nor good folks at #commonlisp could recall of any, so there.

Available in the latest Quicklisp.

Eugene ZaikonnikovAlso ALSA gets Mixer API

· 149 days ago

Also ALSA now has a simple ALSA Mixer API support. See set-mixer-element-volume for sample use.

Available in the latest Quicklisp.

vindarelCommon Lisp on the web: enrich your stacktrace with request and session data

· 157 days ago

A short post to show the usefulness of Hunchentoot-errors and to thank Mariano again.

This library adds the current request and session data to your stacktrace, either in the REPL (base case) or in the browser.

TLDR;

Use it like this:

;; (ql:quickload "hunchentoot-errors)
;;
;; We also use easy-routes: (ql:quickload "easy-routes")

(defclass acceptor (easy-routes:easy-routes-acceptor hunchentoot-errors:errors-acceptor)
  ()
  (:documentation "Our Hunchentoot acceptor that uses easy-routes and hunchentoot-errors, for easier route definition and enhanced stacktraces with request and session data."))

then (make-instance 'acceptor :port 4242).

Base case

Imagine you have a bug in your route:

(easy-routes:defroute route-card-page ("/card/:slug" :method :GET :decorators ((@check-roles admin-role)))
    (&get debug)
  (error "oh no"))

When you access localhost:4242/card/100-common-lisp-recipes, you will see this in the REPL:

[2023-10-13 16:48:21 [ERROR]] oh no
Backtrace for: #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:53896" RUNNING {10019A21A3}>
0: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #<SB-IMPL::CHARACTER-STRING-OSTREAM {1006E9ED43}>)
1: (HUNCHENTOOT::GET-BACKTRACE)
2: ((FLET "H0" :IN HUNCHENTOOT:HANDLE-REQUEST) #<SIMPLE-ERROR "oh no" {1006E9EBE3}>)
3: (SB-KERNEL::%SIGNAL #<SIMPLE-ERROR "oh no" {1006E9EBE3}>)
4: (ERROR "oh no")
5: (MYWEBAPP/WEB::ROUTE-CARD-PAGE "100-common-lisp-recipes")
6: ((:METHOD HUNCHENTOOT:ACCEPTOR-DISPATCH-REQUEST (EASY-ROUTES:EASY-ROUTES-ACCEPTOR T)) #<MYWEBAPP/WEB::ACCEPTOR (host *, port 4242)> #<HUNCHENTOOT:REQUEST {1006C55F33}>) [fast-method]
7: ((:METHOD HUNCHENTOOT:HANDLE-REQUEST (HUNCHENTOOT:ACCEPTOR HUNCHENTOOT:REQUEST)) #<MYWEBAPP/WEB::ACCEPTOR (host *, port 4242)> #<HUNCHENTOOT:REQUEST {1006C55F33}>) [fast-method]
[...]

And, by default, you see a basic error message in the browser:

Show errors

Set this:

(setf hunchentoot:*show-lisp-errors-p* t)

Now you can see a backtrace in the browser window, which is of course super useful during development:

BTW, if you unset this one:

(setf hunchentoot:*show-lisp-backtraces-p* nil)  ;; t by default

You will see the error message, but not the backtrace:

And I remind you that if you set *catch-errors-p* to nil, you’ll get the debugger inside your IDE (Hunchentoot will not catch the errors, and will pass it to you).

Now with request and session data

Now create your server with our new acceptor, inheriting hunchentoot-errors.

You’ll see the current request and session paramaters both in the REPL:

[...]
19: (SB-THREAD::NEW-LISP-THREAD-TRAMPOLINE #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:48756" RUNNING {100AAAFC43}> NIL #<CLOSURE (LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS) {100AAAFBEB}> NIL)
20: ("foreign function: call_into_lisp")
21: ("foreign function: new_thread_trampoline")

HTTP REQUEST:
  uri: /card/100-common-lisp-recipes
  method: GET
  headers:
    HOST: localhost:4242
    USER-AGENT: Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0
    ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
    ACCEPT-LANGUAGE: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
    ACCEPT-ENCODING: gzip, deflate, br
    DNT: 1
    CONNECTION: keep-alive
    COOKIE: "..."
    UPGRADE-INSECURE-REQUESTS: 1
    SEC-FETCH-DEST: document
    SEC-FETCH-MODE: navigate
    SEC-FETCH-SITE: none
    SEC-FETCH-USER: ?1

SESSION:
  :USER: #<MYWEBAPP.MODELS:USER {100EA8C753}>

127.0.0.1 - [2023-10-13 17:32:18] "GET /card/100-common-lisp-recipes HTTP/1.1" 500 5203 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0"

and in the browser:

(notice the #<USER {...}> at the bottom? You’ll need a commit from today to see it, instead of # only)

Final words

These Hunchentoot variables were kinda explained on the Cookbook/web.html, I’ll augment that.

Clack users can use the clack-errors midleware.

Who wants to send a PR for colourful stacktraces?

Joe MarshallSyntax-rules Primer

· 157 days ago

I recently had an inquiry about the copyright status of my JRM’s Syntax Rules Primer for the Merely Eccentric. I don’t want to put it into public domain as that would allow anyone to rewrite it at will and leave that title. Instead, I'd like to release it on an MIT style license: feel free to copy it and distribute it, correct any errors, but please retain the general gist of the article and the title and the authorship.

Tim BradshawSymbol nicknames: a broken toy

· 158 days ago

Symbol nicknames allows multiple names to refer to the same symbol in supported implementations of Common Lisp. That may or may not be useful.

People often say the Common Lisp package system is deficient. But a lot of the same people write code which is absolutely full of explicit package prefixes in what I can only suppose is an attempt to make programs harder to read. Somehow this is meant to be made better by using package-local nicknames for packages. And let’s not mention the unspeakable idiocy that is thinking that a package name like, say, XML is suitable for any kind of general use at all. So forgive me if I don’t take their concerns too seriously.

The CL package system can’t do all the things something like the Racket module system can do. But it’s not clear that, given its job of collecting symbols into, well, packages, it could do that much more than it currently does. Probably some kind of ‘package universe’ notion such as Symbolics Genera had would be useful. But the namespace has to be anchored somewhere, and if you’re willing to give packages domain-structured names in the obvious way and spend time actually constructing a namespace for the language you want to use, it’s perfectly pleasant in my experience.

One thing that might be useful is to allow multiple names to refer to the same symbol. So for instance you might want to have eq? be the same symbol as eq:

> (setf (nickname-symbol "EQ?") 'eq)
eq

> (eq 'eq? 'eq)
t

> (eq? 'eq 'eq?)
t

This allows you to construct languages which have different names for things, but where the names are translated to the underlying name efficiently. As another example, let’s say you wanted to call eql equivalent-p:

> (setf (nickname-symbol "EQUIVALENT-P") 'eql)
eql

> (eql 'eql 'equivalent-p)
t

Well, now you can use equivalent-p as a synonym for eql wherever it occurs:

> (defmethod foo ((x (equivalent-p 1)))
    "x is 1")
#<standard-method foo nil ((eql 1)) 801005BD23>

> (foo 1)
"x is 1"

Symbol nicknames is not completely portable as it requires hooking string-to-symbol lookup. It is supported in LispWorks and SBCL currently: it will load in other Lisps but will complain that it can’t infect them.

Symbol nicknames is also not completely compatible with CL. In CL you can assume that (find-symbol "FOO") either returns a symbol whose name is "FOO" or nil and nil: with symbol nicknames you can’t. In the case where a nickname link has been followed the second value of find-symbol will be :nickname.

Symbol nicknames is a toy. I am not convinced that the idea is even useful, and if it is it probably needs to be thought about more than I have.

But it exists.

TurtleWareProxy Generic Function

· 168 days ago

It is often hard to refactor software implementing an independent specification. There are already clients of the API so we can't remove operators, and newly added operators must play by the specified rules. There are a few possibilities: break the user contract and make pre-existing software obsolete, or abandon some improvements. There is also an option that software is written in Common Lisp, so you can eat your cake and have it too.

CLIM has two protocols that have a big overlap: sheets and output records. Both abstractions are organized in a similar way and have equivalent operators. In this example let's consider a part of the protocol for managing hierarchies:

;; Sheet hierarchy (sub-)protocol with an example implementation.
(defclass sheet () ()) ; protocol class
(defclass example-sheet (sheet)
  ((children :initform '() :accessor sheet-children)))

(defgeneric note-sheet-adopted (sheet)
  (:method (sheet) nil))

(defgeneric note-sheet-disowned (sheet)
  (:method (sheet) nil))

(defgeneric adopt-sheet (parent child)
  (:method ((parent example-sheet) child)
    (push child (sheet-children parent))
    (note-sheet-adopted child)))

(defgeneric disown-sheet (parent child &optional errorp)
  (:method ((parent example-sheet) child &optional (errorp t))
    (and errorp (assert (member child (sheet-children parent))))
    (setf (sheet-children parent)
          (remove child (sheet-children parent)))
    (note-sheet-disowned child)))

;; Output record hierarchy (sub-)protocol with an example implementation.
(defclass output-record () ()) ; protocol class
(defclass example-record (output-record)
  ((children :initform '() :accessor output-record-children)))

(defgeneric add-output-record (child parent)
  (:method (child (parent example-record))
    (push child (output-record-children parent))))

(defgeneric delete-output-record (child parent &optional errorp)
  (:method (child (parent example-record) &optional (errorp t))
    (and errorp (assert (member child (sheet-children parent))))
    (setf (output-record-children parent)
          (remove child (output-record-children parent)))))

Both protocols are very similar and do roughly the same thing. We are tempted to flesh out a single protocol to reduce the cognitive overhead when dealing with hierarchies.

;; The mixin is not strictly necessary - output records and sheets may have
;; wildly different internal structures - this is for the sake of simplicity;
;; most notably it is _not_ a protocol class. We don't do protocol classes.
(defclass node-mixin ()
  ((scions :initform '() :accessor node-scions)))

(defgeneric note-node-parent-changed (node parent adopted-p)
  (:method (node parent adopted-p)
    (declare (ignore node parent adopted-p))
    nil))

(defgeneric insert-node (elder scion)
  (:method :after (elder scion)
    (note-node-parent-changed scion elder t))
  (:method ((elder node-mixin) scion)
    (push scion (node-scions elder))))

(defgeneric delete-node (elder scion)
  (:method :after (elder scion)
    (note-node-parent-changed scion elder nil))
  (:method ((elder node-mixin) scion)
    (setf (node-scions elder) (remove scion (node-scions elder)))))

We define a mixin class for simplicity. In principle we care only about the new protocol and different classes may have different internal representations. Now that we have a brand new unified protocol, it is time to rewrite the old code:

;; Sheet hierarchy (sub-)protocol with an example implementation.
(defclass sheet () ()) ; protocol class
(defclass example-sheet (node-mixin sheet) ())

(defgeneric note-sheet-adopted (sheet)
  (:method (sheet)
    (declare (ignore sheet))
    nil))

(defgeneric note-sheet-disowned (sheet)
  (:method (sheet)
    (declare (ignore sheet))
    nil))

(defmethod note-node-parent-changed :after ((sheet sheet) parent adopted-p)
  (declare (ignore parent))
  (if adopted-p
      (note-sheet-adopted sheet)
      (note-sheet-disowned sheet)))

(defgeneric adopt-sheet (parent child)
  (:method (parent child)
    (insert-node parent child)))

(defgeneric disown-sheet (parent child &optional errorp)
  (:method (parent child &optional (errorp t))
    (and errorp (assert (member child (node-scions parent))))
    (delete-node parent child)))

;; Output record hierarchy (sub-)protocol with an example implementation.
(defclass output-record () ()) ; protocol class
(defclass example-record (node-mixin output-record) ())

(defgeneric add-output-record (child parent)
  (:method (child parent)
    (insert-node parent child)))

(defgeneric delete-output-record (child parent &optional errorp)
  (:method (child parent &optional (errorp t))
    (and errorp (assert (member child (node-scions parent))))
    (delete-node parent child)))

Peachy! Now we can call (delete-node parent child) and this will work equally well for both sheets and output records. It is time to ship the code and boost how clever we are (and advertise the new API). After a weekend we realize that there is a problem with our solution!

Since the old API is alive and kicking, the user may still call adopt-sheet, or if they want to switch to the new api they may call insert-node. This is fine and we have rewritten all our code so that the new element will always be added. But what about user methods?

There may be a legacy code that defines its additional constraints, for example:

(defvar *temporary-freeze* nil)
(defmethod add-output-record :before (child (record output-record))
  (declare (ignore child record))
  (when *temporary-freeze*
    (error "No-can-do's-ville, baby doll!")))

When the new code calls insert-node, then this method won't be called and the constraint will fail. There is an interesting idea, that perhaps instead of trampolining from the sheet protocol to the node protocol functions we could do it the other way around: specialized node protocol methods will call the sheet protocol functions. This is futile - the problem is symmetrical. In that case if some legacy code calls adopt-sheet, then our node methods won't be called.

That's quite a pickle we are in. The main problem is that we are not in control of all definitions and the cat is out of the bag. So what about the cake? The cake is a lie of course! … I'm kidding, of course there is the cake.

When Common Lisp programmers encounter a problem that seems impossible to solve, they usually think of one of three solutions: write a macro, write a dsl compiler or use the metaobject protocol. Usually the solution is a mix of these three things. We are dealing with generic functions - the MOP it is.

The problem could be summarized as follows:

  1. We have under our control a new function that implements the program logic
  2. We have under our control old functions that call the new function
  3. We have legacy methods outside of our control defined on old functions
  4. We will have new methods outside of our control defined on the new function
  5. Sometimes lambda lists between protocols are not compatible

We want the new function to call legacy methods when invoked, and we want to ensure that old functions always call the new function (i.e it is not possible for legacy (sheet-disown-child :around) methods to bypass delete-node).

In order to do that, we will define a new generic function class responsible for mangling arguments when the method is called with make-method-lambda, and proxying add-method to the target class. That's all. When a new legacy method is added to the generic function sheet-disown-child, then it will be hijacked and added to the generic function delete-node instead.

First some syntactic sugar. defgeneric is a good operator except that it does error when we pass options that are not specified. Moreover some compilers are tempted to macroexpand methods at compile time, so we'll expand the new macro in the dynamic environment of a definition:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defun mappend (fun &rest lists)
    (loop for results in (apply #'mapcar fun lists) append results)))

;;; syntactic sugar -- like defgeneric but accepts unknown options
(defmacro define-generic (name lambda-list &rest options)
  (let ((declarations '())
        (methods '()))
    (labels ((parse-option (option)
               (destructuring-bind (name . value) option
                 (case name
                   (cl:declare
                    (setf declarations (append declarations value))
                    nil)
                   (:method
                     (push value methods)
                     nil)
                   ((:documentation :generic-function-class :method-class)
                    `(,name (quote ,@value)))
                   ((:argument-precedence-order :method-combination)
                    `(,name (quote ,value)))
                   (otherwise
                    `(,name (quote ,value))))))
             (expand-generic (options)
               `(c2mop:ensure-generic-function
                 ',name
                 :name ',name :lambda-list ',lambda-list
                 :declarations ',declarations ,@options))
             (expand-method (method)
               `(c2mop:ensure-method (function ,name) '(lambda ,@method))))
      ;; We always expand to ENSURE-FOO because we want dynamic variables like
      ;; *INSIDE-DEFINE-PROXY-P* to be correctly bound during the creation..
      `(progn
         ,(expand-generic (mappend #'parse-option options))
         ,@(mapcar #'expand-method methods)))))

Now we will add a macro that defines a proxy generic function. We include a dynamic flag that will communicte to make-method-lambda and add-method function, that we are still in the initialization phase and methods should be added to the proxy generic function:

(defvar *inside-define-proxy-p* nil)

(defmacro define-proxy-gf (name lambda-list &rest options)
  `(let ((*inside-define-proxy-p* t))
     (define-generic ,name ,lambda-list
       (:generic-function-class proxy-generic-function)
       ,@options)))

The proxy generic function may have a different lambda list than the target. That's indeed the case with our protocol - we don't have the argument errorp in the function delete-node. We want to allow default methods in order to implement that missing behavior. We will mangle arguments according to the specified template in :mangle-args in the function mangle-args-expressoin.

(defclass proxy-generic-function (c2mop:standard-generic-function)
  ((target-gfun                       :reader target-gfun)
   (target-args :initarg :target-args :reader target-args)
   (mangle-args :initarg :mangle-args :reader mangle-args))
  (:metaclass c2mop:funcallable-standard-class)
  (:default-initargs :target-gfun (error "~s required" :target-gfun)
                     :target-args nil
                     :mangle-args nil))

(defmethod shared-initialize :after ((gf proxy-generic-function) slot-names
                                     &key (target-gfun nil target-gfun-p))
  (when target-gfun-p
    (assert (null (rest target-gfun)))
    (setf (slot-value gf 'target-gfun)
          (ensure-generic-function (first target-gfun)))))

To ensure that a proxied method can invoke call-next-method we must be able to mangle arguments both ways. The target generic functions lambda list is stated verbatim in :target-args argument, while the source generic function lambda list is read from c2mop:generic-function-lambda-list.

The function make-method-lambda is tricky to get it right, but it gives quite a bit of control over the method invocation. Default methods are added normally so we don't mangle arguments in the trampoline method, otherwise we convert the target call into the lambda list of a defined method:

;;; MAKE-METHOD-LAMBDA is expected to return a lambda expression compatible with
;;; CALL-METHOD invocations in the method combination. The first argument are
;;; the prototype generic function arguments (the function a method is initially
;;; defined for) and the reminder are all arguments passed to CALL-METHOD - in a
;;; default combination there is one such argument - next-methods. The second
;;; returned value are extra initialization arguments for the method instance.
;;; 
;;; Our goal is to construct a lambda expression that will construct a function
;;; which instead of the prototype argument list accepts the proxied function
;;; arguments and mangles them to call the defined method body. Something like:
;;;
#+ (or)
(lambda (proxy-gfun-call-args &rest call-method-args)
  (flet ((original-method (method-arg-1 method-arg-2 ...)))
    (apply #'original-method (mangle-args proxy-gfun-call-args))))

(defun mangle-args-expression (gf type args)
  (let ((lambda-list (ecase type
                       (:target (target-args gf))
                       (:source (c2mop:generic-function-lambda-list gf)))))
    `(destructuring-bind ,lambda-list ,args
       (list ,@(mangle-args gf)))))

(defun mangle-method (gf gf-args lambda-expression)
  (let ((mfun (gensym)))
    `(lambda ,(second lambda-expression)
       ;; XXX It is not conforming to shadow locally CALL-NEXT-METHOD. That said
       ;; we subclass C2MOP:STANDARD-GENERIC-FUNCTION and they do that too(!).
       (flet ((call-next-method (&rest args)
                (if (null args)
                    (call-next-method)
                    ;; CALL-NEXT-METHOD is called with arguments are meant for
                    ;; the proxy function lambda list. We first need to destruct
                    ;; them and then mangle again.
                    (apply #'call-next-method 
                           ,(mangle-args-expression gf :target
                             (mangle-args-expression gf :source 'args))))))
         (flet ((,mfun ,@(rest lambda-expression)))
           (apply (function ,mfun) ,(mangle-args-expression gf :target gf-args)))))))

(defmethod c2mop:make-method-lambda
    ((gf proxy-generic-function) method lambda-expression environment)
  (declare (ignorable method lambda-expression environment))
  (if (or *inside-define-proxy-p* (null (mangle-args gf)))
      (call-next-method)
      `(lambda (proxy-args &rest call-method-args)
         (apply ,(call-next-method gf method (mangle-method gf 'proxy-args lambda-expression) environment)
                proxy-args call-method-args))))

That leaves us with the last method add-method that decides where to add the method - to the proxy function or to the target function.

(defmethod add-method ((gf proxy-generic-function) method)
  (when *inside-define-proxy-p*
    (return-from add-method (call-next-method)))
  ;; The warning will go away in the production code because we don't want to
  ;; barf at a normal client code.
  (warn "~s is deprecated, please use ~s instead."
        (c2mop:generic-function-name gf)
        (c2mop:generic-function-name (target-gfun gf)))
  (if (or (typep method 'c2mop:standard-accessor-method) (null (mangle-args gf)))
      ;; XXX readers and writers always have congruent lambda lists so this should
      ;; be fine. Besides we don't know how to construct working accessors on some
      ;; (ekhm sbcl) implementations, because they have problems with invoking
      ;; user-constructed standard accessors (with passed :SLOT-DEFINITION SLOTD).
      (add-method (target-gfun gf) method)
      (let* ((method-class (class-of method))
             (old-lambda-list (c2mop:generic-function-lambda-list gf))
             (new-lambda-list (target-args gf))
             (new-specializers (loop with spec = (c2mop:method-specializers method)
                                     for arg in new-lambda-list
                                     until (member arg '(&rest &optional &key))
                                     collect (nth (position arg old-lambda-list) spec)))
             ;; It would be nice if we could reinitialize the method.. but we can't.
             (new-method (make-instance method-class
                                        :lambda-list new-lambda-list
                                        :specializers new-specializers
                                        :qualifiers (method-qualifiers method)
                                        :function (c2mop:method-function method))))
        (add-method (target-gfun gf) new-method))))

That's it. We've defined a new generic function class that allows specifying proxies. Now we can replace definitions of generic functions that are under our control. The new (the final) implementation looks like this:

;; Sheet hierarchy (sub-)protocol with an example implementation.
(defclass sheet () ()) ; protocol class
(defclass example-sheet (node-mixin sheet) ())

(defgeneric note-sheet-adopted (sheet)
  (:method (sheet)
    (declare (ignore sheet))
    nil))

(defgeneric note-sheet-disowned (sheet)
  (:method (sheet)
    (declare (ignore sheet))
    nil))

(defmethod note-node-parent-changed :after ((sheet sheet) parent adopted-p)
  (declare (ignore parent))
  (if adopted-p
      (note-sheet-adopted sheet)
      (note-sheet-disowned sheet)))

(define-proxy-gf adopt-sheet (parent child)
  (:target-gfun insert-node)
  (:target-args parent child)
  (:mangle-args parent child)
  (:method (parent child)
    (insert-node parent child)))

(define-proxy-gf disown-sheet (parent child &optional errorp)
  (:target-gfun delete-node)
  (:target-args parent child)
  (:mangle-args parent child nil)
  (:method (parent child &optional (errorp t))
    (and errorp (assert (member child (node-scions parent))))
    (delete-node parent child)))

;; Output record hierarchy (sub-)protocol with an example implementation.
(defclass output-record () ()) ; protocol class
(defclass example-record (node-mixin output-record) ())

(define-proxy-gf add-output-record (child parent)
  (:target-gfun insert-node)
  (:target-args parent child)
  (:mangle-args child parent)
  (:method (child parent)
    (insert-node parent child)))

(define-proxy-gf delete-output-record (child parent &optional errorp)
  (:target-gfun insert-node)
  (:target-args parent child)
  (:mangle-args child parent)
  (:method (child parent &optional (errorp t))
    (and errorp (assert (member child (node-scions parent))))
    (delete-node parent child)))

And this code is defined in a separate compilation unit:

;; Legacy code in a third-party library.
(defvar *temporary-freeze* nil)
(defmethod add-output-record :before (child (record output-record))
  (declare (ignore child))
  (when *temporary-freeze*
    (error "No-can-do's-ville, baby doll!")))

;; Bleeding edge code in an experimental third-party library.
(defvar *logging* nil)
(defmethod insert-node :after ((record output-record) child)
  (declare (ignore child))
  (when *logging*
    (warn "The record ~s has been extended!" record)))

Dare we try it? You bet we do!

(defparameter *parent* (make-instance 'example-record))
(defparameter *child1* (make-instance 'example-record))
(defparameter *child2* (make-instance 'example-record))
(defparameter *child3* (make-instance 'example-record))
(defparameter *child4* (make-instance 'example-record))
(defparameter *child5* (make-instance 'example-record))

(add-output-record *child1* *parent*)
(print (node-scions *parent*))        ;1 element

(insert-node *parent* *child2*)
(print (node-scions *parent*))        ;1 element

;; So far good!
(let ((*temporary-freeze* t))
  (handler-case (adopt-sheet *parent* *child3*)
    (error     (c) (print `("Good!" ,c)))
    (:no-error (c) (print `("Bad!!" ,c))))

  (handler-case (add-output-record *child3* *parent*)
    (error     (c) (print `("Good!" ,c)))
    (:no-error (c) (print `("Bad!!" ,c))))

  (handler-case (insert-node *parent* *child3*)
    (error     (c) (print `("Good!" ,c)))
    (:no-error (c) (print `("Bad!!" ,c)))))

;; Still perfect!
(let ((*logging* t))
  (handler-case (adopt-sheet *parent* *child3*)
    (error     (c) (print `("Bad!" ,c)))
    (warning   (c) (print `("Good!",c))))

  (handler-case (add-output-record *child4* *parent*)
    (error     (c) (print `("Bad!" ,c)))
    (warning   (c) (print `("Good!",c))))

  (handler-case (insert-node *parent* *child5*)
    (error     (c) (print `("Bad!" ,c)))
    (warning   (c) (print `("Good!",c)))))

(print `("We should have 5 children -- " ,(length (node-scions *parent*))))
(print (node-scions *parent*))

This solution has one possible drawback. We add methods from the proxy generic function to the target generic function without discriminating. That means that applicable methods defined on adopt-sheet are called when add-output-record is invoked (and vice versa). Moreover methods with the same set of specializers in the target function may replace each other. On the flip side this is what we arguably want – the unified protocol exhibits full behavior of all members. We could have mitigated this problem by signaling an error for conflicting methods from different proxies, but if you think about it, a conforming program must not define methods that are not specialized on a subclass of the standard class - otherwise they risk overwriting internal methods! In other words all is good.

Edit 1 Another caveat is that methods for the proxy generic function must be defined in a different compilation unit than the function. This is because of limitations of defmethod - the macro calls make-method-lambda when it is expanding the body (at compile time), while the function definition is processed at the execution time.

That means that make-method-lambda during the first compilation will be called with a standard-generic-function prototype and the proxy won't work.

Edit 2 To handle correctly call-next-method we need to shadow it. That is not conforming, but works when we subclass c2mop:standard-generic-function. As an alternative we could write a full make-method-lambda expansion that defines both call-next-method and next-method-p.

Cheers!
Daniel

P.S. if you like writing like this you may consider supporting me on Patreon.

Joe Marshall

· 173 days ago

Greenspun's tenth rule of programming states

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug ridden, slow implementation of half of Common Lisp.
Observe that the Python interpreter is written in C.

In fact, most popular computer languages can be thought of as a poorly implemented Common Lisp. There is a reason for this. Church's lambda calculus is a great foundation for reasoning about programming language semantics. Lisp can be seen as a realization of a lambda calculus interpreter. By reasoning about a language's semantics in Lisp, we're essentially reasoning about the semantics in a variation of lambda calculus.

Paolo AmorosoExploring Medley as a Common Lisp development environment

· 175 days ago

Since encountering Medley I gained considerable experience with Interlisp. Medley Interlisp is a project for preserving, reviving, and modernizing the Interlisp-D software development environment of the Lisp Machines Xerox created at PARC.

Nine months later I know enough to find my way around and confidently use most of the major system tools and features.

I read all the available documentation, books, and publications, so I know where to look for information. And I undertook Interlisp programming projects such as Stringscope, Braincons, Sysrama, and Femtounit.

Now I'm ready to explore Medley as a Common Lisp development environment.

Although most of the system, facilities, and tools are written in and designed around Interlisp, the companies that maintained and marketed Medley over time partially implemented Common Lisp and integrated it with the environment. The completion level of the implementation is somewhere between CLtL1 and CLtL2, plus CLOS via Portable Common Loops (PCL).

Motivation

I want to widen this experience to Common Lisp.

I'll leverage the more advanced Lisp dialect and interface with Interlisp's facilities as an application platform that comprises a rich set of libraries and tools such a window system, graphics primitives, menu facilities, and GUI controls for building applications. Each world can interoperate with the other, so Common Lisp functions can call Interlisp ones and the other way around.

Developing Common Lisp programs with Medley is both my goal and a way of achieving it through practice. Medley is an ideal self-contained computing universe for my personal projects and Common Lisp greatly enchances its toolbox.

Tools

The main tools for developing Common Lisp code are the same as for Interlisp: the SEdit structure editor for writing code; the File Manager, a make-like tool for tracking changes to Lisp objects in the running image and saving them to files; and the Executive (or Exec), the Lisp listener.

However, the workflow is subtly different.

In some cases taking advantage of the integration with Medley involves different steps for Common Lisp code. For example, defining and changing packages so that the File Manager notices and tracks them needs to be done in a certain order. And there are Medley extensions to the package forms.

When working with Common Lisp I open at least two Execs, a Common Lisp and an Interlisp one. The former is for testing, running, and evaluating Common Lisp code.

The Interlisp Exec is for launching system tools and interacting with the File Manager. Since all the symbols of SEdit, the File Manager, and other system tools are in the IL Interlisp package, in an Interlisp Exec it's not necessary to add package qualifiers to symbols all the time.

Exec commands such as DIR and CD work the same in both Execs.

Documentation

Medley's Common Lisp features aren't documented in the Interlisp Reference Manual, the main information source about the system. The reason is the companies that distributed and maintained the product ceased operations before the work on implementing and documenting Common Lisp was completed.

I found only a couple of good sources on Common Lisp under Medley.

The implementation notes and the release notes of Lyric, the music-themed codename of one of Interlisp-D's versions, provide an overview of the integration between Common Lisp and Medley. The release notes of Medley 1.0, a later version, expand on this. Issue 5 of HOTLINE!, a newsletter Xerox published for its Lisp customers, has useful step by step examples of creating and managing Common Lisp packages the Medley way.

Some of the system code of Medley is written in Common Lisp and may be a source of usage examples and idioms. I'm also writing Common Lisp code snippets to test my understanding of the integration with Medley.

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org


For older items, see the Planet Lisp Archives.


Last updated: 2024-03-14 12:11