Tycho Garen Common Gotchas

· 2 days ago

This is a post I wrote a long time ago and never posted, but I've started getting back into doing some work in Common Lisp and thought it'd be good to send this one off.

On my recent "(re)learn Common Lisp" journey, I've happened across a few things that I've found frustrating or confusing: this post is a collection of them, in hopes that other people don't struggle with them:

  • Implementing an existing generic function for a class of your own, and have other callers specialize use your method implementation you must import the generic function, otherwise other callers will (might?) fall back to another method. This makes sense in retrospect, but definitely wasn't clear on the first go.
  • As a related follow on, you don't have to define a generic function in order to write or use a method, and I've found that using methods is actually quite nice for doing some type checking, at the same time, it can get you into a pickle if you later add the generic function and it's not exported/imported as you want.
  • Property lists seem cool for a small light weight mapping, but they're annoying to handle as part of public APIs, mostly because they're indistinguishable from regular lists, association lists are preferable, and maybe with make-hash even hash-tables.
  • Declaring data structures inline is particularly gawky. I sometimes want to build a list or a hash map in-line an alist, and it's difficult to do that in a terse way that doesn't involve building the structure programatically. I've been writing (list (cons "a" t) (cons "b" nil)) sort of things, which I don't love.
  • If you have a variadic macro (i.e. that takes &rest args), or even I suppose any kind of macro, and you have it's arguments in a list, there's no a way, outside of eval to call the macro, which is super annoying, and makes macros significantly less appealing as part of public APIs. My current conclusion is that macros are great when you want to add syntax to make the code you're writing clearer or to introduce a new paradigm, but for things that could also be a function, or are thin wrappers on for function, just use a function.

vindarelState of Common Lisp Web Development - an overview

· 7 days ago

Caution [from 2017: what a road since then]: this is a draft. I take notes and write more in other resources (the Cookbook, my blog).

update, June 2022: see my web project skeleton, it illustrates and fixes common issues: https://github.com/vindarel/cl-cookieweb

update july, 5th 2019: I put this content into the Cookbook: https://lispcookbook.github.io/cl-cookbook/web.html, fixing a long-standing request.

new post: why and how to live-reload one’s running web application: https://lisp-journey.gitlab.io/blog/i-realized-that-to-live-reload-my-web-app-is-easy-and-convenient/

new project skeleton: lisp-web-template-productlist: Hunchentoot + easy-routes + Djula templates + Bulma CSS + a Makefile to build the project

See also the Awesome CL list.

Information is at the moment scarce and spread appart, Lisp web frameworks and libraries evolve and take different approaches.

I’d like to know what’s possible, what’s lacking, see how to quickstart everything, see code snippets and, most of all, see how to do things that I couldn’t do before such as hot reloading, building self-contained executables, shipping a multiplatform web app.


Prior notice:

Some people sell ten pages long ebooks or publish their tutorial on Gitbook to have a purchase option. I prefer to enhance the collaborative Cookbook (I am by far the main contributor). You can tip me on Kofi if you like: https://ko-fi.com/vindarel Thanks !


Table of Contents

Web application environments

Clack, Lack

Clack is to Lisp what WSGI is to Python. However it is mostly undocumented and not as battle-proofed as Hunchentoot.

Web frameworks

Hunchentoot

The de-facto web server, with the best documentation (cough looking old cough), the most websites on production. Lower level than a web framework (defining routes seems weird at first). I think worth knowing.

Its terminology is different from what we are used to (“routes” are not called routes but we create handlers), part I don’t know why and part because the Lisp image-based development allows for more, and thus needs more terminology. For example, we can run two applications on different URLs on the same image.

https://edicl.github.io/hunchentoot/

edit: here’s a modern looking page: https://digikar99.github.io/common-lisp.readthedocs/hunchentoot/

Caveman

A popular web framework, or so it seems by the github stars, written by a super-productive lisper, with nice documentation for basic stuff but lacking for the rest, based on Clack (webserver interface, think Python’s WSGI), uses Hunchentoot by default.

I feel like basic functions are too cumbersome (accessing url parameters).

https://github.com/fukamachi/caveman

Snooze

By the maintainer of Sly, Emacs’ Yasnippet,...

Defining routes is like defining functions. Built-in features that are available as extensions in Clack-based frameworks (setting to get a stacktrace on the browser, to fire up the debugger or to return a 404,...). Definitely worth exploring.

https://github.com/joaotavora/snooze

Radiance

Radiance, with extensive tutorial and existing apps.

It doesn’t look like a web framework to me. It has ready-to-use components:

  • admin page (but what does it do?)
  • auth system
  • user: provide user accounts and permissions
  • image hosting
  • there is an email marketing system in development...

cl-rest-server

cl-rest-server

a library for writing REST Web APIs in Common Lisp.

Features: validation via schemas, Swagger support, authentication, logging, caching, permission checking...

It seems complete, it is maintained, the author seems to be doing web development in CL for a living. Note to self: I want to interview him.

Wookie

https://github.com/orthecreedence/wookie

An asynchronous web server, by an impressive lisper, who built many async libraries. Used for the Turtl api backend. Dealing with async brings its own set of problems (how will be debugging ?).

Nice api to build routes, good documentation: http://wookie.lyonbros.com/

Weblocks (solving the Javascript problem)

Weblocks allows to create dynamic pages without a line of JavaScript, all in Lisp. It was started years ago and it saw a large update and refactoring lately.

It isn’t the easy path to web development in CL but there’s great potential IMO.

It doesn’t do double data binding as in modern JS frameworks. But new projects are being developed...

See our presentation below.

http://40ants.com/weblocks/quickstart.html

Tasks

Accessing url parameters

It is easy and well explained with Hunchentoot or easy-routes in the Cookbook.

Lucerne has a nice with-params macro that makes accessing post or url query parameters a breeze:

@route app (:post "/tweet")
(defview tweet ()
  (if (lucerne-auth:logged-in-p)
      (let ((user (current-user)))
        (with-params (tweet)
          (utweet.models:tweet user tweet))
        (redirect "/"))
      (render-template (+index+)
                       :error "You are not logged in.")))

Snooze’s way is simple and lispy: we define routes like methods and parameters as keys:

(defroute lispdoc (:get :text/* name &key (package :cl) (doctype 'function))
   ...

matches /lispdoc, /lispdoc/foo and /lispdoc/foo?package=arg.


On the contrary, I find Caveman’s and Ningle’s ways cumbersome.

Ningle:

(setf (ningle:route *app* "/hello/:name")
      #'(lambda (params)
          (format nil "Hello, ~A" (cdr (assoc "name" params :test #'string=)))))

The above controller will be invoked when you access to “/hello/Eitaro” or “/hello/Tomohiro”, and then (cdr (assoc “name” params :test #‘string=)) will be “Eitaro” and “Tomohiro”.

and it doesn’t say about query parameters. I had to ask:

(assoc "the-query-param" (clack.request:query-parameter lucerne:*request*) :test 'string=)

Caveman:

Parameter keys contain square brackets (”[” & “]”) will be parsed as structured parameters. You can access the parsed parameters as _parsed in routers.

(defroute "/edit" (&key _parsed)
  (format nil "~S" (cdr (assoc "person" _parsed :test #'string=))))
;=> "((\"name\" . \"Eitaro\") (\"email\" . \"e.arrows@gmail.com\") (\"birth\" . ((\"year\" . 2000) (\"month\" . 1) (\"day\" . 1))))"

Session an cookies

Data storage

SQL

Mito works for MySQL, Postgres and SQLite3 on SBCL and CCL.

https://lispcookbook.github.io/cl-cookbook/databases.html

We can define models with a regular class which has a mito:dao-table-class :metaclass:

(defclass user ()
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name)
   (email :col-type (:varchar 128)
          :initarg :email
          :accessor user-email))
  (:metaclass mito:dao-table-class)
  (:unique-keys email))

We create the table with ensure-table-exists:

(ensure-table-exists 'user)
;-> ;; CREATE TABLE IF NOT EXISTS "user" (
;       "id" BIGSERIAL NOT NULL PRIMARY KEY,
;       "name" VARCHAR(64) NOT NULL,
;       "email" VARCHAR(128),
;       "created_at" TIMESTAMP,
;       "updated_at" TIMESTAMP
;   ) () [0 rows] | MITO.DAO:ENSURE-TABLE-EXISTS

Persistent datastores

Migrations

Mito has migrations support and DB schema versioning for MySQL, Postgres and SQLite3, on SBCL and CCL. Once we have changed our model definition, we have commands to see the generated SQL and to apply the migration.

We inspect the SQL: (suppose we just added the email field into the user class above)

(mito:migration-expressions 'user)
;=> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

and we can apply the migration:

(mito:migrate-table 'user)
;-> ;; ALTER TABLE "user" ALTER COLUMN "email" TYPE character varying(128), ALTER COLUMN "email" SET NOT NULL () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;   ;; CREATE UNIQUE INDEX "unique_user_email" ON "user" ("email") () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;-> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

Crane advertises automatic migrations, i.e. it would run them after a C-c C-c. Unfortunately Crane has some issues, it doesn’t work with sqlite yet and the author is busy elsewhere. It didn’t work for me at first try.

Let’s hope the author comes back to work on this in a near future.

Forms

There are a few libraries, see the awesome-cl list. At least one is well active.

Debugging

On an error we are usually dropped into the interactive debugger by default.

Snooze gives options:

  • use the debugger,
  • print the stacktrace in the browser (like clack-errors below, but built-in),
  • display a custom 404.

clack-errors. Like a Flask or Django stacktrace in the browser. For Caveman, Ningle and family.

By default, when Clack throws an exception when rendering a page, the server waits for the response until it times out while the exception waits in the REPL. This isn’t very useful. So now there’s this.

It prints the stacktrace along with some request details on the browser. Can return a custom error page in production.

clack-pretend

Are you tired of jumping to your web browser every time you need to test your work in Clack? Clack-pretend will capture and replay calls to your clack middleware stack. When developing a web application with clack, you will often find it inconvenient to run your code from the lisp REPL because it expects a clack environment, including perhaps, cookies or a logged-in user. With clack-pretend, you can run prior web requests from your REPL, moving development back where it belongs.

Testing

Testing with a local DB: example of a testing macro.

We would use envy to switch configurations.

Misc

Oauth, Job queues, etc

Templating engines

HTML-based

Djula: as Django templates. Good documentation. Comes by default in Lucerne and Caveman.

We also use a dot to access attributes of dict-like variables (plists, alists, hash-tables, arrays and CLOS objects), such a feature being backed by the access library.

We wanted once to use structs and didn’t find how to it directly in Djula, so we resorted in a quick helper function to transform the struct in an alist.

Defining custom template filters is straightforward in Djula, really a breeze compared to Django.

Eco - a mix of html with lisp expressions.

Truncated example:

<body>
      <% if posts %>
        <h1>Recent Posts</h1>
        <ul id="post-list">
          <% loop for (title . snippet) in posts %>
            <li><%= title %> - <%= snippet %></li>
          <% end %>
        </ul>
        ...

Lisp-based

I prefer the semantics of Spinneret over cl-who. It also has more features (like embeddable markdown, warns on malformed html, and more).

Javascript

Parenscript

Parenscript is a translator from an extended subset of Common Lisp to JavaScript. Parenscript code can run almost identically on both the browser (as JavaScript) and server (as Common Lisp). Parenscript code is treated the same way as Common Lisp code, making the full power of Lisp macros available for JavaScript. This provides a web development environment that is unmatched in its ability to reduce code duplication and provide advanced meta-programming facilities to web developers.

https://common-lisp.net/project/parenscript/

JSCL

A Lisp-to-Javascript compiler bootstrapped from Common Lisp and executed from the browser.

https://github.com/jscl-project/jscl

https://t-cool.github.io/jscl-playground/

Ajax

Is it possible to write Ajax-based pages only in CL?

The case Webblocks - Reblocks, 2017

Weblocks is an “isomorphic” web frameworks that solves the “Javascript problem”. It allows to write the backend and an interactive client interface in Lisp, without a line of Javascript, in our usual Lisp development environment.

The framework evolves around widgets, that are updated server-side and are automatically redisplayed with transparent ajax calls on the client.

It is being massively refactored, simplified, rewritten and documented since 2017. See the new quickstart:

http://40ants.com/weblocks/quickstart.html

Writing a dynamic todo-app resolves in:

  • defining a widget class for a task:
(defwidget task ()
        ((title
          :initarg :title
          :accessor title)
         (done
          :initarg :done
          :initform nil
          :accessor done)))
  • doing the same for a list of tasks:
(defwidget task-list ()
        ((tasks
          :initarg :tasks
          :accessor tasks)))
  • saying how to render these widgets in html by extending the render method:
(defmethod render ((task task))
        "Render a task."
        (with-html
              (:span (if (done task)
                         (with-html
                               (:s (title task)))
                       (title task)))))

(defmethod render ((widget task-list))
        "Render a list of tasks."
        (with-html
              (:h1 "Tasks")
              (:ul
                (loop for task in (tasks widget) do
                      (:li (render task))))))
  • telling how to initialize the Weblocks app:
(defmethod weblocks/session:init ((app tasks))
         (declare (ignorable app))
         (let ((tasks (make-task-list "Make my first Weblocks app"
                                      "Deploy it somewhere"
                                      "Have a profit")))
           (make-instance 'task-list :tasks tasks)))
  • and then writing functions to interact with the widgets, for example adding a task:
(defmethod add-task ((task-list task-list) title)
        (push (make-task title)
              (tasks task-list))
        (update task-list))

Adding an html form and calling the new add-task function:

(defmethod render ((task-list task-list))
        (with-html
          (:h1 "Tasks")
          (loop for task in (tasks task-list) do
            (render task))
          (with-html-form (:POST (lambda (&key title &allow-other-keys)
                                         (add-task task-list title)))
            (:input :type "text"
                    :name "title"
                    :placeholder "Task's title")
            (:input :type "submit"
                    :value "Add"))))

Shipping

We can build our web app from sources, no worries, that works.

Building

We can build an executable also for web apps. That simplifies a deployment process drastically.

We can even get a Lisp REPL and interact with the running web app, in which we can even install new Quicklisp dependencies. That’s quite incredible, and it’s very useful, if not to hot-reload a web app (which I do anyways but that might be risky), at least to reload a user’s configuration file.

This is the general way:

(sb-ext:save-lisp-and-die #p"name-of-executable" :toplevel #'main :executable t)

But this is an SBCL-specific command, so we can be generic and use asdf:make, with a couple lines inside our system .asd declaration. See the Cookbook: https://lispcookbook.github.io/cl-cookbook/scripting.html#with-asdf

Now if you run your binary, your app will start up all fine, but it will quit instantly. We need to put the web server thread on the foreground:

(defun main ()
    ;; with bordeaux-threads. Also sb-ext: join-thread, thread-name, list-all-threads.
    (bt:join-thread (find-if (lambda (th)
                                (search "hunchentoot" (bt:thread-name th)))
                              (bt:all-threads))))

When I run it, Hunchentoot stays listening at the foreground:

$ ./my-webapp
Hunchentoot server is started.
Listening on localhost:9003.

I can use a tmux session (tmux, then C-b d to detach it) or better yet, start the app with Systemd, see below.

But we need something more for real apps, we need to ship foreign libraries. Deploy to the rescue. Edit your .asd file slightly to have:

:defsystem-depends-on (:deploy)  ;; (ql:quickload "deploy") before
:build-operation "deploy-op"     ;; instead of "program-op" as above
:build-pathname "my-webapp"  ;; doesn't change
:entry-point "my-webapp:main"  ;; doesn't change

Build the app again with asdf:make, and see how Deploy discovers and ships in the required foreign libraries: libssl.so, libosicat.so, libmagic.so...

It puts the final binary and the .so libraries in a bin/ directory. This is what you’ll have to ship.

I can now build my web app, send it to my VPS and see it live \o/

One more thing. We don’t want to ship libssl and libcrypto, so we ask Deploy to not ship them. We prefer to rely on the target OS.

;; .asd
;; Deploy may not find libcrypto on your system.
;; But anyways, we won't ship it to rely instead
;; on its presence on the target OS.
(require :cl+ssl)  ; sometimes necessary.
#+linux (deploy:define-library cl+ssl::libssl :dont-deploy T)
#+linux (deploy:define-library cl+ssl::libcrypto :dont-deploy T)

There’s also a hack if you have an issue with ASDF trying to update itself... see the skeleton: https://github.com/vindarel/cl-cookieweb

To be exhaustive, here’s how to catch a user’s C-c and stop our app correctly. By default, you would get a full backtrace. Yuk.

(defun main ()
  (start-app :port 9003)
  ;; with bordeaux-threads
  (handler-case (bt:join-thread (find-if (lambda (th)
                                             (search "hunchentoot" (bt:thread-name th)))
                                         (bt:all-threads)))
    (#+sbcl sb-sys:interactive-interrupt
      #+ccl  ccl:interrupt-signal-condition
      #+clisp system::simple-interrupt-condition
      #+ecl ext:interactive-interrupt
      #+allegro excl:interrupt-signal
      () (progn
           (format *error-output* "Aborting.~&")
           (clack:stop *server*)
           (uiop:quit 1)) ;; portable exit, included in ASDF, already loaded.
    ;; for others, unhandled errors (we might want to do the same).
    (error (c) (format t "Woops, an unknown error occured:~&~a~&" c)))))

To see:

Multiplatform delivery with Electron (Ceramic)

Ceramic makes all the work for us.

It is as simple as this:

;; Load Ceramic and our app
(ql:quickload '(:ceramic :our-app))

;; Ensure Ceramic is set up
(ceramic:setup)
(ceramic:interactive)

;; Start our app (here based on the Lucerne framework)
(lucerne:start our-app.views:app :port 8000)

;; Open a browser window to it
(defvar window (ceramic:make-window :url "http://localhost:8000/"))

;; start Ceramic
(ceramic:show-window window)

and we can ship this on Linux, Mac and Windows.

More:

Ceramic applications are compiled down to native code, ensuring both performance and enabling you to deliver closed-source, commercial applications.

(so no need to minify our JS)

with one more line:

(ceramic.bundler:bundle :ceramic-hello-world
                                 :bundle-pathname #p"/home/me/app.tar")
Copying resources...
Compiling app...
Compressing...
Done!
#P"/home/me/app.tar"

This last line was buggy for us.

Deployment

When you build a self-contained binary (see above, “Shipping”), deployment gets easy.

Manually

sbcl --load <my-app> --eval '(start-my-app)'

For example, a run Makefile target:

run:
	sbcl --load my-app.asd \
	     --eval '(ql:quickload :my-app)' \
	     --eval '(my-app:start-app)'  ;; given this function starts clack or hunchentoot.

this keeps sbcl in the foreground. We can use tmux to put it in background, or use Systemd.

Then, we need of a task supervisor, that will restart our app on failures, start it after a reboot, handle logging. See the section below and example projects.

with Clack

$ clackup app.lisp
Hunchentoot server is started.
Listening on localhost:5000.

with Docker

So we have various implementations ready to use: sbcl, ecl, ccl... with Quicklisp well configured.

https://lispcookbook.github.io/cl-cookbook/testing.html#gitlab-ci

On Heroku

See heroku-buildpack-common-lisp and the Awesome CL#deploy section.

Systemd: daemonizing, restarting in case of crashes, handling logs

Generally, this depends on your system. But most GNU/Linux distros now come with Systemd. Write a service file like this:

$ /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example

[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/bin/sbcl --load run.lisp  # your command, full path
Type=simple
Restart=always
RestartSec=10

One gotcha is that your app must be run on the foreground. See the threads snippet above in Shipping, and add it when running the app from sources too. Otherwise, you’ll see a “compilation unit aborted, caught 1 fatal error condition” error. That’s simply your Lisp quitting too early.

Run this command to start the service:

sudo systemctl start my-app.service

to check its status:

systemctl status my-app.service

Systemd handles logging for you. We write to stdout or stderr, it writes logs:

journalctl -u my-app.service

use -f -n 30 to see live updates of logs.

This tells Systemd to handle crashes and to restart the app:

Restart=always

and it can start the app after a reboot:

[Install]
WantedBy=basic.target

to enable it:

sudo systemctl enable my-app.service

Debugging SBCL error: ensure_space: failed to allocate n bytes

If you get this error with SBCL on your server:

mmap: wanted 1040384 bytes at 0x20000000, actually mapped at 0x715fa2145000
ensure_space: failed to allocate 1040384 bytes at 0x20000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)

then disable ASLR:

sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"

Connecting to a remote Swank server

Little example here: http://cvberry.com/tech_writings/howtos/remotely_modifying_a_running_program_using_swank.html.

It defines a simple function that prints forever:

;; a little common lisp swank demo
;; while this program is running, you can connect to it from another terminal or machine
;; and change the definition of doprint to print something else out!
;; (ql:quickload '(:swank :bordeaux-threads))

(require :swank)
(require :bordeaux-threads)

(defparameter *counter* 0)

(defun dostuff ()
  (format t "hello world ~a!~%" *counter*))

(defun runner ()
  (bt:make-thread (lambda ()
                    (swank:create-server :port 4006)))
  (format t "we are past go!~%")
  (loop while t do
       (sleep 5)
       (dostuff)
       (incf *counter*)))

(runner)

On our server, we run it with

sbcl --load demo.lisp

we do port forwarding on our development machine:

ssh -L4006:127.0.0.1:4006 username@example.com

this will securely forward port 4006 on the server at example.com to our local computer’s port 4006 (swanks accepts connections from localhost).

We connect to the running swank with M-x slime-connect, typing in port 4006.

We can write new code:

(defun dostuff ()
  (format t "goodbye world ~a!~%" *counter*))
(setf *counter* 0)

and eval it as usual with M-x slime-eval-region for instance. The output should change.

There are more pointers on CV Berry’s page.

Hot reload

When we run the app as a script we get a Lisp REPL, so we can hot-reload the running web app. Here we demonstrate a recipe to update it remotely.

Example taken from Quickutil.

It has a Makefile target:

hot_deploy:
	$(call $(LISP), \
		(ql:quickload :quickutil-server) (ql:quickload :swank-client), \
		(swank-client:with-slime-connection (conn "localhost" $(SWANK_PORT)) \
			(swank-client:slime-eval (quote (handler-bind ((error (function continue))) \
				(ql:quickload :quickutil-utilities) (ql:quickload :quickutil-server) \
				(funcall (symbol-function (intern "STOP" :quickutil-server))) \
				(funcall (symbol-function (intern "START" :quickutil-server)) $(start_args)))) conn)) \
		$($(LISP)-quit))

It has to be run on the server (a simple fabfile command can call this through ssh). Beforehand, a fab update has run git pull on the server, so new code is present but not running. It connects to the local swank server, loads the new code, stops and starts the app in a row.


Buy Me a Coffee at ko-fi.com

(yes that currently helps, thanks!)

Tycho Garen Methods of Adoption

· 10 days ago

Before I started actually working as a software engineer full time, writing code was this fun thing I was always trying to figure out on my own, and it was fun, and I could hardly sit down at my computer without learning something. These days, I do very little of this kind of work. I learn more about computers by doing my job and frankly, the kind of software I write for work is way more satisfying than any of the software I would end up writing for myself.

I think this is because the projects that a team of engineers can work on are necessarily larger and more impactful. When you build software with a team, most of the time the product either finds users (or your end up without a job.) When you build software with other people and for other people, the things that make software good (more rigorous design, good test discipline, scale,) are more likely to be prevalent. Those are the things that make writing software fun.

Wait, you ask "this is a lisp post?" and "where is the lisp content?" Wait for it...

In Pave the On/Off Ramps [1] I started exploring this idea that technical adoption is less a function of basic capabilities or numbers of features in general, but rather about the specific features that support and further adoption and create confidence in maintenance and interoperability. A huge part of the decision process is finding good answers to "can I use these tools as part of the larger system of tools that I'm using?" and "can I use this tool a bit without needing to commit to using it for everything?"

Technologies which are and demand ideological compliance are very difficult to move into with confidence. A lot of technologies and tools demand ideological compliance, and their adoption depends on once-in-a-generation sea changes or significant risks. [2] The alternate method, to integrate into people's existing workflows and systems, and provide great tools that work for some usecases and to prove their capability is much more reliable: if somewhat less exciting.

The great thing about Common Lisp is that it always leans towards the pragmatic rather than the ideological. Common Lisp has a bunch of tools--both in the langauge and in the ecosystem--which are great to use but also not required. You don't have to use CLOS (but it's really cool), you don't have to use ASDF, there isn't one paradigm of developing or designing software that you have to be constrained to. Do what works.


I think there are a lot of questions that sort of follow on from this, particularly about lisp and the adoption of new technologies. So let's go through the ones I can think of, FAQ style:

  • What kind of applications would a "pave the exits" support?

    It almost doesn't matter, but the answer is probably a fairly boring set of industrial applications: services that transform and analyze data, data migration tools, command-line (build, deployment) tools for developers and operators, platform orchestration tools, and the like. This is all boring (on the one hand,) but most software is boring, and it's rarely the case that programming langauge actually matters much.

    In addition, CL has a pretty mature set of tools for integrating with C libaries and might be a decent alternative to other langauges with more complex distribution stories. You could see CL being a good langauge for writing extensions on top of existing tools (for both Java with ABCL and C/C++ with ECL and CLASP), depending.

  • How does industrial adoption of Common Lisp benefit the Common Lisp community?

    First, more people writing common lisp for their jobs, which (assuming they have a good experience,) could proliferate into more projects. A larger community, maybe means a larger volume of participation in existing projects (and more projects in general.) Additionally, more industrial applications means more jobs for people who are interested in writing CL, and that seems pretty cool.

  • How can CL compete with more established languages like Java, Go, and Rust?

    I'm not sure competition is really the right model for thinking about this: there's so much software to write that "my langauge vs your langauge" is just a poor model for thinking about this: there's enough work to be done that everyone can be successful.

    At the same time, I haven't heard about people who are deeply excited about writing Java, and Go folks (which I count myself among) tend to be pretty pragmatic as well. I see lots of people who are excited about Rust, and it's definitely a cool langauge though it shines best at lower level problems than CL and has a reasonable FFI so it might be the case that there's some exciting room for using CL for higher level tasks on top of rust fundamentals.

[1]In line with the idea that product management and design is about identifying what people are doing and then institutionalizing this is similar to the urban planning idea of "paving cowpaths," I sort of think of this as "paving the exits," though I recognize that this is a bit force.d
[2]I'm thinking of things like the moment of enterprise "object oriented programing" giving rise to Java and friends, or the big-data watershed moment in 2009 (or so) giving rise to so-called NoSQL databases. Without these kinds of events you the adoption of these big paradigm-shifting technologies is spotty and relies on the force of will of a particular technical leader, for better (and often) worse.

Tycho Garen Pave the On and Off Ramps

· 17 days ago

I participated in a great conversation in the #commonlisp channel on libera (IRC) the other day, during which I found a formulation of a familar argument that felt more clear and more concrete.

The question--which comes up pretty often, realistically--centered on adoption of Common Lisp. CL has some great tools, and a bunch of great libraries (particularly these days,) why don't we see greater adoption? Its a good question, and maybe 5 year ago I would have said "the libraries and ecosystem are a bit fragmented," and this was true. It's less true now--for good reasons!--Quicklisp is just great and there's a lot of coverage for doing common things.

I think it has to do with the connectivity and support at the edges of a project, an as I think about it, this is probably true of any kind of project.

When you decide to use a new tool or technology you ask yourself three basic questions:

  1. "is this tool (e.g. language) capable of fulfilling my current needs" (for programming languages, this is very often yes,)
  2. "are there tools (libraries) to support my use so I can focus on my core business objectives," so that you're not spending the entire time writing serialization libraries and HTTP servers, which is also often the case.
  3. "will I be able to integrate what I'm building now with other tools I use and things I have built in the past." This isn't so hard, but it's a thing that CL (and lots of other projects) struggle with.

In short, you want to be able to build a thing with the confidence that it's possible to finish, that you'll be able to focus on the core parts of the product and not get distracted by what should be core library functionality, and finally that the thing you build can play nicely with all the other things you've written or already have. Without this third piece, writing a piece of software with such a tool is a bit of a trap.

We can imagine tools that expose data only via quasi-opaque APIs that require special clients or encoding schemes, or that lack drivers for common databases, or integration with other common tools (metrics! RPC!) or runtime environments. This is all very reasonable. For CL this might look like:

  • great support for gRPC

    There's a grpc library that exists, is being maintained, and has basically all the features you'd want except support for TLS (a moderately big deal for operational reasons,) and async method support (not really a big deal.) It does depend on CFFI, which makes for a potentially awkward compilation story, but that's a minor quibble.

    The point is not gRPC qua gRPC, the point is that gRPC is really prevalent globally and it makes sense to be able to meet developers who have existing gRPC services (or might like to imagine that they would,) and be able to give them confidence that whatever they build (in say CL) will be useable in the future.

  • compilation that targets WASM

    Somewhat unexpectedly (to me, given that I don't do a lot of web programming,) WebAssembly seems to be the way deploy portable machine code into environments that you don't have full control over, [1] and while I don't 100% understand all of it, I think it's generally a good thing to make it easier to build software that can run in lots of situation.

  • unequivocally excellent support for JSON (ex)

    I remember working on a small project where I thought "ah yes, I'll just write a little API server in CL that will just output JSON," and I completely got mired in various comparisons between JSON libraries and interfaces to JSON data. While this is a well understood problem it's not a very cut and dry problem.

    The thing I wanted was to be able to take input in JSON and be able to handle it in CL in a reasonable way: given a stream (or a string, or equivalent) can I turn it into an object in CL (CLOS object? hashmap?)? I'm willing to implement special methods to support it given basic interfaces, but the type conversion between CL types and JSON isn't always as straight forward as it is in other languages. Similarly with outputting data: is there a good method that will take my object and convert it to a JSON stream or string? There's always a gulf between what's possible and what's easy and ergonomic.

I present these not as a complaint, or even as a call to action to address the specific issues that I raise (though I certianly wouldn't complain if it were taken as such,) but more as an illustration of technical decision making and the things that make it possible for a team or a project to say yes to a specific technology.

There are lots of examples of technologies succeeding from a large competitive feild mostly on the basis of having great interoperability with existing solutions and tools, even if the core technology was less exciting or innovative. Technology wins on the basis of interoperability and user's trust, not (exactly) on the basis of features.

[1]I think the one real exception is runtimes that have really good static binaries and support for easy cross-compiling (e.g. Go, maybe Rust.)

Nicolas HafnerThe Kandria Kickstarter is now Live!

· 17 days ago
https://kandria.com/media/header capsule.png

It's finally here, the Kickstarter for Kandria is now live! Check a look: kickstarter.com/projects/shinmera/kandria

We've been working towards this for a long time, so please consider supporting us. There's a new trailer on the campaign page, along with the new demo which is now live in the Steam Next Fest!

vindarelNew video: how to request a REST API in Common Lisp: fetching the GitHub API

· 20 days ago

A few weeks ago, I put together a new Lisp video. It’s cool, sound on, ‘til the end ;)

I want to show how to (quickly) do practical, real-world stuff in Lisp. Here, how to request a web API. We create a new full-featured project with my project skeleton, we study the GitHub API, and we go ahead.

I develop in Emacs with Slime, but in the end we also build a binary, so we have a little application that works on the command line (note that I didn’t use SBCL’s core compression, we could have a lighter binary of around 30MB).

I use the libraries: Dexador for HTTP requests, the Jonathan and then the Shasht JSON libraries, as well as Access and Serapeum to help with accessing, creating and viewing hash-tables.

Thanks for the comments already :)

tks for that… it’s amazing.

excellent

Thank you, that was a great video! I plan to get the udemy course now based on this.

Here are my previous ones:

I want to do more but damn, it takes time. You can push me for more :)

Last notes:

I had fun with the soundtrack :)

I edit the video with the latest Kdenlive with the snap package and it’s great for me.

Nicolas HafnerSteam Next Fest and Kickstarter Next Week! - June Kandria Update

· 25 days ago
https://filebox.tymoon.eu//file/TWpRNU9BPT0=

Okey, June is the month! The Steam Next Fest and the Kickstarter are both happening this month. Be sure you won't miss them!

Steam Next Fest

One week from now the Steam Next Fest is going to be live, and with it the new demo for Kandria! We've been working on this for quite a while now, and we're excited for people to try it out in the fest. Make sure to put it onto your wishlist:

https://filebox.tymoon.eu//file/TWpRNU53PT0=

Kickstarter

One day after the Next Fest is another important date, namely the launch of our Kickstarter. We've been preparing for this for a long time now, and we're almost ready. Just a few more small things to iron out, and we're trying our best to ramp up the pre-campaign marketing to get as many people as possible onto that waitlist.

The first two days of a Kickstarter are super important for its success, so we need to make sure to make as big of a splash with it as possible, and ideally reach the funding goal by then already. If you'd like to help that become a reality, the best thing to do is to, of course, sign up yourself! But also tell friends and other people you know about it.

Thanks again, and I'm looking forward to the launch! We have a few exciting announcements to make during the campaign as well, and combined with the next fest, it's going to be a heck of a time, I'm sure.

Development Progress

May was a month fraught with some issues! First I fell ill for pretty much an entire week, and then I had to spend another week on the Digital Dragons conference in Krakow. Finally I had to take a week off just to try and recover from everything that's been building up from the two looming big events.

So as you might guess, not too much has happened this month. However, we're still making good progress. Tim has been hard at work rounding up all the main quest content and polishing it up to a good state. The game should now be fully playable from start to finish!

https://filebox.tymoon.eu//file/TWpRNU9RPT0=

We'll start full-game testing soon to make sure the flow and everything works well, and also get to fleshing things out with some more side quests and extras. There's a ton of detail work that needs doing still, and that's going to make a huge difference. But it's also going to take a lot of time. Fortunately, we still have quite a bit of time left over until our next deadline looms, so we're still clear of the frying pan for now.

The Bottom Line

Okey, finally let's look at the roadmap from may.

  • Create a new trailer and Kickstarter video

  • General improvements to UI and flow

  • Marketiiiing

  • Buuuuuugs

  • Add detail to region 2 and 3

  • Do lots of user testing on the full game content

  • Add more side quests and areas

  • Implement Steam achievements

Yeah, as mentioned, not too much of a change here. But, not to worry, we're progressing according to schedule still.

Please wishlist Kandria and join our Kickstarter waiting list, and I hope to see you during the campaigns!

Didier VernaQuickref 4.0 beta 1 "The Aftermath" is released

· 37 days ago

Dear all,

as previously announced, I have released the first Quickref version compatible with Declt 4.0 (which is now at beta 2) and which is going to keep track of the development over there. Because Declt 4 is currently considered in beta state, so is Quickref 4 for the time being.

There are a number of important changes in this new major version of the library. Not much interesting from the outside is an infrastructure overhaul which improves the parallel support. The index generation code has also been rewritten to benefit from the recent changes in Declt. Slightly more interesting from the outside is an improvement of the self-documenting aspects of the public interface when used interactively, along with more usage correctness checks. Also, the build environment has been upgraded to Debian Bullseye.

But the most critical change, of course, is the name for the 4.x series. In compliance with the general theme (Iron Maiden songs), and because Quickref 4 is meant to closely follow the brave new Declt version, I thought "Brave New World" would be nice. On the other hand, as Declt wanders through its uncharted 4.0 beta territory, Quickref 4 is likely to suffer the consequences, so perhaps "The Aftermath" is more appropriate...

Anyway, the Docker images are up to date, and so is the Quickref website, currently documenting 2110 Common Lisp libraries.

Enjoy!

Eitaro FukamachiWoo: a high-performance Common Lisp web server

· 43 days ago

Hi, all Common Lispers.

It's been 7 years since I talked about Woo at European Lisp Symposium. I have heard that several people are using Woo for running their web services. I am grateful for that.

I quit the company I was working for when I developed Woo. Today, I'm writing a payment service in Common Lisp as ever in another company. Not surprisingly, I'm using Woo there as well, and so far have had no performance problems or other operational difficulties.

But on the other hand, some people are still reckless enough to try to run web services using Hunchentoot. And, some people complain about the lack of articles about Woo.

Sorry for my negligence in not keeping updating the information and publishing articles. Therefore, it may be worthful to take Woo as the topic even today.

What is Woo?

Woo is a high-performance web server written in Common Lisp.

It's almost the same level as Go's web server in performance and several times better than other Common Lisp servers, like Hunchentoot.

Web server benchmark

What's different? Not only eliminate bottlenecks by tuning the Common Lisp code but the architecture is designed to handle many concurrent requests efficiently.

Compared to Hunchentoot

Hunchentoot is the most popular web server. According to Quicklisp download stats in April 2022, Hunchentoot is the only web server in the top 100 (ref. Woo is 302th).

An excellent point of Hunchentoot is that it's written in portable Common Lisp. It works on Linux, macOS, and Windows with many Lisp implementations. No external libraries are required.

On the other hand, there are concerns about using it as an HTTP server open to the world.

Because Hunchentoot takes a thread-per-request approach to handle requests.

Hunchentoot thread-per-request architecture

The disadvantage of this architecture is that it is not good at handling large numbers of simultaneous requests.

Hunchentoot thread-per-request architecture

Hunchentoot creates a thread when accepting a new connection, done sending a response, and terminates the thread when it's disconnected. Therefore, more concurrent threads are required when it takes longer to process a slow client (e.g., network transmission time).

It doesn't matter if every client works fast enough. In reality, however, some clients are slow or unstable, for example, smartphone users.

There is also a DoS attack, which intentionally makes large numbers of slow simultaneous connections, called Slowloris attack. Web services running on Hunchentoot can be instantly inaccessible by this attack. The bad news is that you can easily find a script to make Slowloris attack on the web.

Compared to Wookie

Event-driven is another approach to handling a massive amount of simultaneous connections. Let's take Wookie as an example.

Wookie event-driven architecture

In this model, all connection I/O is processed asynchronously, so the speed and stability of the connection do not affect other connections.

Of course, this architecture also has its drawbacks as it works in a single thread, which means only one process can be executed at a time. When a response is being sent to one client, it is not possible to read another client's request.

Wookie's throughput is slightly worse than Hunchentoot for simple HTTP request and response iterations in my benchmark.

Besides that, it is more advantageous for protocols such as WebSocket, in which small chunks are exchanged asynchronously.

Wookie depends on libuv, C library to support asynchronous I/O. Although installing an external library is bothersome, libuv is used internally in Node.js, so it is not so difficult to do it in most environments, including Windows.

Another web server house is event-driven but written in portable Common Lisp. Its event-loop is implemented with cl:loop and usocket:wait-for-input. If the performance doesn't matter, it would be another option.

Multithreaded Event-driven

Woo also adopts the event-driven model, except it has multiple event loops.

Woo multithreaded event-driven architecture

First, it accepts a connection in the main thread, dispatches to pre-created worker threads, and processes requests and responses with asynchronous I/O in each worker.

That is why its throughput is exceptionally high: it processes multiple requests simultaneously in worker threads.

In addition, Woo uses libev while Wookie uses libuv. libev runs fast since it is a pretty small library that only wraps the async I/O API between each OS, like epoll, kqueue, and poll. Its downside is less platform support. Especially, it doesn't support Windows. However, I don't think it will be a problem in most cases since it's rare to use Windows for a web server.

Woo is Clack-compliant

Woo's feature I'd like to mention is that it is Clack-compliant.

Clack is an abstraction layer for Common Lisp web servers. It allows running a web application on different web servers which follows Clack standard without any changes.

Since Woo supports Clack natively, it can run Lack applications without any libraries. I want to introduce what Clack and Lack are in the other article.

Running inside Docker

Lastly, let's run Woo inside a Docker container.

Files

These 3 files are necessary.

  • Dockerfile
  • entrypoint.sh
  • app.lisp

Dockerfile

FROM fukamachi/sbcl
ENV PORT 5000

RUN set -x; \
  apt-get update && apt-get -y install --no-install-recommends \
    libev-dev \
    gcc \
    libc6-dev && \
  rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY entrypoint.sh /srv

RUN set -x; \
  ros install clack woo

ENTRYPOINT ["/srv/entrypoint.sh"]
CMD ["app.lisp"]

entrypoint.sh

A script to start a web server, Woo, in this case.

#!/bin/bash

exec clackup --server woo --debug nil --address "0.0.0.0" --port "$PORT" "$@"

app.lisp

A Lack application is written in this file. See Lacks documentation for the detail.

(lambda (env)
  (declare (ignore env))
  '(200 () ("Hello from Woo")))

Build

"`bash $ docker build -t server-app-test .


### Run

```bash
$ docker run --rm -it -p 5000:5000 -v $PWD:/app server-app-test

Then, open localhost:5000 with your browser.

Summary

I showed how Woo is different from other web servers and how to run it with Docker.

Clack allows switching web servers easily without modifying the application code. In the case of Quickdocs.org, I use Hunchentoot for development and use Woo for the production environment.

I will introduce Clack/Lack in the next article.

vindarelVideo: Create a Common Lisp project from scratch with our project generator

· 51 days ago

In this video I want to demo real-world Lisp stuff I had trouble finding tutorials for:

  • how to create a CL project:
    • what’s in the .asd file?
    • what’s a simple package definition?
    • how do we load everything in our editor (Emacs and SLIME here)?
  • how to set up tests?
    • and how to run them from the terminal?
    • and (WTF) how to get the correct exit code????
  • how to build a binary in order to run our app from the terminal?
    • and so, how to get command-line arguments?
    • but also, can I run my app from sources, without building a binary?
  • what’s Roswell and how do I share my app with it?

Let’s find out (it’s 5 minutes long):

I use my project generator: cl-cookieproject. See setup, alternatives, limitations and TODOs there. You will also find a similar generator for web projects.

If you find it useful, share the video!

Didier VernaDeclt 4.0 beta 1 "William Riker" is released

· 53 days ago

Today, after two years and a half of (irregular) development, and 465 commits, I have released the first beta version of Declt 4.0, my reference manual generator for Common Lisp libraries.

I seldom release official beta versions of software, let alone make announcements about them, but this one deserves an exception. Since version 3, Declt has been undergoing a massive, SOC-oriented, overhaul. The intent is to reimplement it into a clean 3-stages pipeline. Stage one, the "assessment" stage is in charge of gathering information. Stage two, the "assembly" stage, creates a documentation template specifying how the information is to be organized. Finally, Stage three, the "typesetting" stage, renders the template in a specific output format. Of course, the purpose is to eventually be able to support different kinds of templates and output formats.

Declt 4.0b1 marks the achievement of Stage one, which is now complete. It also contains a lot of new (and long awaited) features and improvements. The user manual has been updated to provide an overview of the new architecture, along with some details about Stage one of the pipeline. From now on, I'm also going to release new versions of Quickref to closely follow the evolution of Declt.

Apart from the aforementioned architecture overhaul, which really boils down to internals shaking and shouldn't affect the end-user much, a lot of new features and improvements are also provided in this release. They are listed below.

Backward Incompatible Changes

For the benefit of Separation of Concerns, but also for other reasons, a number of keyword arguments to the declt function have been renamed. :hyperlinks becomes :locations, :version becomes :library-version, :texi-directory becomes :output-directory, and :texi-name becomes :file-name.

New Features

Default / Standard Values

Declt now has the ability to advertise or hide properties that get default / standard values (e.g. standard method combination, :instance slot allocation, etc.).

Support for Aliases

Declt now recognizes and advertises "aliases", that is, funcoids which have their fdefinition, macro, or compiler macro function set manually to another, original, definition.

Support for New Programmatic Entities

Typed structures and setf compiler macros are now properly detected and documented with all the specific information.

Proper Support for Uninterened Symbols

Such symbols may be encountered on several occasions such as slot names (trivialib, for example, defines structures with uninterned slot names in order to prevent access to them). Definitions named with uninterned symbols are considered private, and denoted by the empty set package (∅).

SBCL 2.1.2 Required

The following enhancements depend on it:

  • short form setf expanders now get correct source information,
  • method combinations lambda-lists are now documented.

Domestic vs. Foreign Definitions

The concept of domestic (as opposed to foreign) definition has been extended to include those created in one of the sources files of the library being documented, even if the symbol naming the definition is from a foreign package. If the source file is unknown, then the symbol's package must be domestic. The general intent is to consider language extensions as domestic, hence, always documented. For example, a new method on a standard generic function (say, initialize-instance) will now always appear in the documentation.

This refinement is accompanied by a new option allowing to include foreign definitions in the generated documentation. Of course, only foreign definitions somehow related to the library being documented will appear in the documentation. Also, those definitions will in general be partial: only the parts relevant to the library being documented will appear. Continuing with the initialize-instance example, new methods normally appear as toplevel, standalone definitions in the documentation (and their parent generic function is simply mentioned). If foreign definitions are included however, there will be a toplevel entry for the generic function, and all methods defined in the library will be documented there (truly foreign methods won't appear at all).

Introspection Heuristic

Until now, there was a single heuristic for finding domestic definitions, which was to start from domestic packages and scan as many connections between symbols as possible. That heuristic is reasonably fast, but may occasionally miss some domestic definitions. Declt now has the ability to scan all symbols in the Lisp image instead of only the ones from domestic packages. This new option ensures that all domestic definitions are found, at the expense of a much greater computation time.

Supported Licenses

The Microsoft Public License has been added.

Info Installation Category

In other words, the value of Texinfo's @direntry command is now customizable (instead to being hardwired to "Common Lisp").

Improvements

Documentation Thinning

Packages reference lists do not point to methods directly anymore (as they can be reached via the generic function's reference). Also, only slots for which the parent classoid is named from another package are now referenced.

Files reference lists do not point to slots anymore (as they can be reached via the parent classoid's reference). Also, only methods for which the parent generic function is defined in another file are now referenced.

The readability of long references has been improved. In particular, they don't advertise the type of the referenced definitions anymore, when there is no ambiguity.

Slot documentation now advertises the slot name's package only when different from that of the parent classoid.

Non standalone method documentation now advertises the source file only when different from that of the parent generic function.

The rendering of EQL specializers has been inmproved.

The documentation of setf / writer methods doesn't render the "new value" argument / specializer anymore.

Merging

Generic definitions containing only reader / writer methods are upgraded to specific reader / writer categories, and definition merging is now only attempted on those.

Lambda Lists

Uninformative parts of lambda lists are now filtered out. This includes &whole, &environment, &aux variables, along with options / keyword variables and default values.

Method specializers in lambda lists now link back to their respective class definitions.

Method Combinations

The method combination discovery scheme has been upgraded to benefit from SBCL 1.4.8's enhancements (themselves following my ELS 2018 paper). The old code didn't break, but prevented unused method combinations from being detected.

Bug Fixes and Workarounds

ASDF

Better handling of missing / unloaded components or dependencies (this can happen for instance with feature-dependent conditional inclusion).

Cope with the lack of specification of the license information in ASDF systems by coercing to a string.

Fix several cases of system files documentation duplication. Declt automatically documents .asd files as special cases of Lisp files. However, some systems have the bad (IMHO) habit of mentioning them explicitly as components (e.g. static files). When this happens, Declt silently discards that definition and keeps its own (at the expense of having a slightly incorrect system documentation).

Anchoring

After years of vain attempts at providing human-readable yet unique anchor names (which was only really useful for Info, BTW), I finally got rid of my last bit of idealism. Anchor names now use numerical definition UIDs, and since Texinfo allows me to expand references with some more information than just the anchor, it's good enough. Besides, it fixes the last remaining rare cases exhibiting the fact that it's just impossible to have anchors that are both human-readable and unique.

Setf Expanders

Fix the computation of short form setf epxander lambda lists (which didn't correctly handle the presence of optional or rest arguments before.

Handle the potential unavailability of a setf expander's update function or short form operator. Document it if applicable. Also signal a warning when the expander is domestic.

Nicolas HafnerPolish of the game and polish conference - May Kandria Update

· 53 days ago
https://filebox.tymoon.eu//file/TWpRNE5RPT0=

Another month of polish, Kickstarter prep, more polish, and also some news on conferences! Plus, a small peek at the final boss fight. Exciting!

Kickstarter

Well heck. Next month is going to be the launch of the Kickstarter already! We're still working on the new trailer for that, and polishing things for the new demo, but things are looking pretty good. Most of the Things Left To Do now are in the marketing and outreach department.

I've started contacting other teams to see about cross-promotions and such. If you know anyone that has, had, or will have a Kickstarter for a somewhat similar game to Kandria, please ask them to contact me so we can see about arranging that!

In the meantime, if you haven't yet, subscribe to the Kickstarter and tell anyone you know that might be interested in it!

https://filebox.tymoon.eu//file/TWpRM01BPT0=

It may not seem like much, but it really does help us tremendously to have more people on that pre-launch signup.

Digital Dragons in Krakˇw

Kandria was selected to be part of the Indie Showcase, so I'll be at the Digital Dragons conference in Krakˇw, May 16/17. If you're coming to the conference, please stop on by, I'd love to have a chat!

Due to plane and other related reasons, I'll actually be in Krakˇw until the 20th. Probably just going to spend the time working in the hotel, but if you're nearby and would like to go grab a drink at some point, let me know!

Development

Another month of mostly bugfixes and small, incremental improvements to the tiling of the lower regions of the game. Some more playtesting as well, and further improvements to the later acts of the game's story.

We've also started on the boss fight AI, now that Fred has completed a first round of the animations for it. It's looking really nice!

https://filebox.tymoon.eu//file/TWpRNE5BPT0=

This has also prompted some more adjustments to the overall combat behaviour. It's feeling a lot better now, but I'm still not quite happy with it. I'll have to make more improvements to the overall movement and animation timing of the player this month.

Another important improvement has been to the deployment of the game; the data files are now bundled in an archive, reducing both install size, and massively reducing initial load times.

All of the improvements we've been working on will be in the new demo that'll be available in the Steam Next Fest launching on June 13th. I hope you can wait until then!

In between this laborious incremental fixing and improving, I've also been writing some more in-depth looks at the kinds of things that make game development take so much time on our mailing list. If you're interested in reading about that, sign up here!

The Bottom Line

Okey, let's look at the roadmap from april.

All of the improvements we've been working on will be in the new demo that'll be available in the Steam Next Fest launching on June 13th. I hope you can wait until then!dmap from april.

  • Implement new enemy types

  • Finish designing the remaining main story NPCs

  • Complete the Kickstarter page

  • Create a new trailer and Kickstarter video

  • Add detail to region 2 and 3

  • Do lots of user testing on the full game content

  • Add more side quests and areas

  • Implement Steam achievements

In the meantime, we still could really use your help with spreading the word about Kandria's Kickstarter! If you know any friends or other communities, tell them about the Kickstarter!

Patrick SteinMaking Sudoku Diagrams in Lisp

· 60 days ago

A few weeks ago, I was analyzing a Sudoku variant if the board were a flat torus instead of a square. I wrote a little paper about this analysis.

For that paper, I created the diagrams by taking screenshots of the f-puzzles Sudoku editor, pulling them into the GIMP image editor, cropping them all to the same dimensions, and pulling them into my document. I discovered a mistake that I had made throughout all of the images after I had already collected all of the screenshots. Rather than fixing the mistake by doing a new screenshot, I fixed things in the image editor instead of in the Sudoku editor.

It was painful.

So, I decided to make a way to declare a diagram and then render it so that it can be modified/corrected rather painlessly.

My initial cut at this was in CL-PDF. I realized, when I had finished, that I had only used a small subset of CL-PDF and that almost all of that functionality was available in Vecto (with a tiny bit of help from ZPB-TTF which Vecto uses already).

This was all the motivation that I needed to finally start on a project that I’ve been hoping to do for years now. I wrote a compatibility library that takes something very, very close to the subset of CL-PDF that I used in the diagram and renders it either using CL-PDF or Vecto so that I can have the output as either a vector image or a raster image.

Sudoku diagram rendered showing different marking, highlighting, and labelling options of the SUDOKU-DIAGRAMS library

I’d like to extend this sometime to also output to CL-SVG, but the way that fonts are handled in SVGs might make that impossible for most things one would want to do with text in a diagram.

You can find the library here: https://github.com/nklein/sudoku-diagrams

Patrick SteinDRAW v0.2.20220430

· 60 days ago

I have released a new Lisp library. This library provides compatibility between CL-PDF and Vecto so that I can render the same drawing either as a vector drawing or a raster drawing.

This library forms an almost drop-in replacement for a small subset of CL-PDF and implements that subset with both CL-PDF and Vecto.

The functionality at the moment is somewhat limited. But, it is enough to make the Sudoku diagrams that I was working on. I had originally created the diagrams using CL-PDF. Then, I went through and wrapped them in one line of code and changed all of the PDF: package delimiters to DRAW: and ta-da. Diagrams.

Diagram showing various triangles, rectangles, and text to demonstrate the functions DRAW supports.

This is the current test image. It demonstrates all of the capabilities available at the moment.

You can find the library here: https://github.com/nklein/draw

vindarelWriting an interactive web app in Common Lisp: Hunchentoot then CLOG

· 64 days ago

We want a web app to display a list of data and have an input field to interactively filter it.

We’ll start with a simple, regular web app built with Hunchentoot. We’ll have a search input to filter our data, and we’ll see that to be more interactive, typically to filter out the results as the user types, we’ll need more than basic HTTP requests. We’ll need some JavaScript. But we’ll reach this level of interactivity with CLOG (and no JavaScript).

DISCLAIMER: this post is my entry for the CLOG contest!

Let’s install our first libraries: Hunchentoot for the web server, Djula for the HTML templates, str for a string utility.

#+(or)
(ql:quickload '("hunchentoot" "djula" "str"))

We create a package for our experiments, and we “enter” it. I use UIOP’s define-package because it throws less warnings than defpackage when we add and remove symbols. It also has more features (:reexport) that I don’t use here.

(uiop:define-package :clog-contest
    (:use :cl))

(in-package :clog-contest)

OK. We define a route. It takes one GET parameter, for demo purposes.

(hunchentoot:define-easy-handler (root-route :uri "/") (name)
  (format nil "Hey~@[ ~A~]!" name))

Start the server:

(defvar *server* (make-instance 'hunchentoot:easy-acceptor :port 6789))
(hunchentoot:start *server*)

and access http://localhost:6789/ Now let’s create our products. We quickly define a class containing an ID, a title and a price.


(defclass product ()
  ((id :initarg :id :accessor product-id :type integer
       :documentation "Unique ID")
   (title :initarg :title :accessor product-title :type string)
   (price :initarg :price :accessor product-price :type integer)))

(defvar *product-id* 1
  "Stupid counter to increment our unique product ID.
  Normally this is given by a DB.")

(defparameter *products* '() "A list of products.")

We are going to create random testing products, so let’s have a couple helpers to create random titles and prices.

(defun random-price ()
  "Return an integer between 1 and 10.000 (price is expressed in cents)."
  (1+ (random 9999)))

(defparameter *title-part-1* (list "pretty" "little" "awesome" "white" "blue"))
(defparameter *title-part-2* (list "book" "car" "laptop" "travel" "screwdiver"))
(defun random-title ()
  (let ((index (random (length *title-part-1*)))
        (index-2 (random (length *title-part-2*))))
    (format nil "~a ~a" (elt *title-part-1* index) (elt *title-part-2* index-2))))

try it out:

#+(or)
(random-title)

We get titles like “white book”, “little car”, etc. Now, for testing purposes, we create a 100 dummy product instances:

(defun gen-test-products (&optional (nb 100))
  (dotimes (i nb)
    (push (make-instance 'product
                         :id (incf *product-id*)
                         :title (random-title)
                         :price (random-price))
          *products*)))

(defun reset-test-products ()
  (setf *products* nil))

Try it and we get:


*products*
(#<PRODUCT {1005B29363}> #<PRODUCT {1005B29113}> #<PRODUCT {1005B28EC3}>
 #<PRODUCT {1005B28C73}> #<PRODUCT {1005B28A23}> #<PRODUCT {1005B287D3}>
 ...)

Implement the print-object method if you want nice-looking product literals. See the Cookbook.
Now let’s display the products in the browser. We will redefine our route. We make sure to extract the view logic in functions.


(defun print-product (it &optional (stream nil))
  "Print a product title and price on STREAM (return a new string by default)."
  (format stream "~a - ~f~&"
          (str:fit 20 (product-title it))  ;; the fit function was merged recently.
          (/ (product-price it) 100)))

(defun print-products (products)
  "Return a list of products as a string (dummy, for tests purposes)."
  (with-output-to-string (s)
    (format s "Products:~&")
    (dolist (it products)
      (print-product it s))))

CL-USER> (print-products (subseq *products* 0 10))

"Products:
pretty car           -   22.26
awesome travel       -   13.87
little screwdiver    -   35.6
white laptop         -   6.08
little book          -   27.57
white laptop         -   42.63
blue travel          -   93.8
blue car             -   29.99
pretty car           -   38.95
little screwdiver    -   46.99

We tell our route to display the list of products like this:


(hunchentoot:define-easy-handler (root-route :uri "/") ()
  (print-products *products*))

We see something, but it’s stupid to return text to the browser. We need templates. We’ll use Djula templates. And we’ll steal some ready-to-use pretty HTML :)

I’ll use Bulma CSS because it’s simple, modern (flexbox) and just because. I don’t know all CSS frameworks out there. I’ll do my shopping in this showcase of Bulma templates: https://bulmatemplates.github.io/bulma-templates/ IIRC I took the “Modal Cards” one and simplified it a bit. Our final result is: Let’s create a templates/ directory and create:

  • base.html

  • products.html, that inherits the base.

    • inside this template, we “include” the search form that we wrote in another template: search-form.html. It’s a way to factorize HTML code that we can re-use here and there.

The base template loads Bulma from a CDN, creates a navbar, defines a “content” block that our other templates will override, and a footer.

Our products template “extends” base.html and creates the “content” block. There we loop over a list of products given by our Hunchentoot root and display them. But before that happens, we need to install and configure Djula to find and compile our templates. We tell Djula to look for templates in the templates/ directory.

(djula:add-template-directory "templates/")

Note that normally, I do that relatively to an .asd file, that we didn’t create yet:


(djula:add-template-directory
 (asdf:system-relative-pathname "myproject" "src/templates/"))

If you have an issue with the path on the Lisp REPL, on SLIME you can do ,cd (a comma command) to change the current working directory. Now we define our templates:

(defparameter +base.html+ (djula:compile-template* "base.html"))
(defparameter +products.html+ (djula:compile-template* "products.html"))

As a result, you can see they are compiled templates:


CLOG-CONTEST> +products.html+
#<DJULA::COMPILED-TEMPLATE /home/vince/bacasable/bacalisp/lisp-tutorial-clog-contest/templates/products.html {2073D30B}>

OK, our route needs to return a template and give data to it. Our route returns djula:render-template*.

(hunchentoot:define-easy-handler (root-route :uri "/") ()
  (djula:render-template* +products.html+ nil
                          :products *products*))

Nice! We display all products. We will accept a search query to filter them. Pagination is for later.

We have an input field that defines an HTML form, that calls the search endpoint. We need:

  • to define the /search route

  • to write a dummy function to search in our products list.


(defun search-products (query &optional (products *products*))
  "Search for QUERY in the products' title.
  This would be a DB call."
  (loop for product in products
     when (str:containsp (str:downcase query) (str:downcase (product-title product)))
     collect product))

Try it:

#+(or)
(search-products "awesome")

Now the search route. It accepts one GET parameter, q for “query”.


(hunchentoot:define-easy-handler (search-route :uri "/search") (q)
  (let* ((products (search-products *products* q)))
    (djula:render-template* +products.html+ nil
                            :title (format nil "My products - ~a" q)
                            :query q
                            :products products
                            :no-results-p (zerop (length products)))))

Try it, it works :) I agree, the search algorithm is simplistic. What about multiple words, accents, typos, non-exact searches (stemming)...?

Your search query is seen in the URL parameters: http://localhost:6789/search?q=travel That is usually a good thing. In modern single-page applications, you loose this, or you have to handle the URL construction yourself.

The search required a page reload. If your app is fast, it might not be an issue. However, if we wanted the search to be more interactive, for example showing results as we type, we would need to use JavaScript. Enters CLOG.

CLOG

Can we make our app interactive with CLOG?

Well, we can, and what’s even cooler is that the development process is itself very interactive. CLOG sends changes to the page through websockets as you add or edit functionalities. As such we can see changes in real time. For example, change a colour:


(setf (background-color *body*) :red)

and BAM, it’s red.

Let’s create another package for this new app. I’ll “use” functions and macros provided by the :clog package, as well as our previously defined :clog-contest ones (duh... we didn’t :export any yet).


(uiop:define-package :clog-contest-with-clog
    (:use :cl :clog
          :clog-contest))

(in-package :clog-contest-with-clog)

The very first steps you can do to grasp CLOG’s interactive fun is to make changes to a browser window while on the CLOG REPL.


(ql:quickload "clog")
CL-USER> (in-package clog-user)
CLOG-USER> (clog-repl)
CLOG-USER> (setf (background-color *body*) "red")
CLOG-USER> (create-div *body* :content "Hello World!")

And voilÓ. A browser window was opened for you.

You will also find many demos here: https://github.com/rabbibotton/clog/tree/main/tutorial You can run them with (clog:run-tutorial 1) (by their number id).

For the following, I invite you to have a look at CLOG’s common elements: https://rabbibotton.github.io/clog/clog-manual.html#toc-8-common-clog-elements

Typically, to create a div on a DOM element, we use create-div.

The first thing we want to start our CLOG app is the initialize function. Its signature:


initialize (on-new-window-handler &key (host 0.0.0.0) (port 8080) (server hunchentoot)
 (extended-routing nil) (long-poll-first nil) (boot-file /boot.html)
 (boot-function nil) (static-boot-html nil) (static-boot-js nil)
 (static-root (merge-pathnames ./static-files/ (system-source-directory clog))))

Inititalize CLOG on a socket using HOST and PORT to serve BOOT-FILE
as the default route to establish web-socket connections and static
files located at STATIC-ROOT. [...]

The following calls our add-products function with a body (CLOG object) as argument.

(defun start-tutorial ()
  "Start tutorial."
  (initialize 'add-products)
  (open-browser))

OK so what do we want to do? We want to create a search input field, and to display our products below. When the user types something, we want to immediately filter the products, and re-display them.

A first version where we only display products would be this:


(defun add-products (body)
  (let* ((result-div (create-div body :content "")))
    (display-products result-div (subseq clog-contest::*products* 0 10))))

And the display-products function is below:


(defun display-products (body products)
  "Display these products in the page.
  Create a div per product, with a string to present the product.
  We don't create nice-looking Bulma product cards here."
  (dolist (it products)
      (create-div body :content
                  (format nil "~a - ~a"
                          (clog-contest::product-id it)
                          (clog-contest::print-product it)))))

Now we want to handle the interactivity. The event to watch is the key up event. In CLOG, we have the set-on-key-up method. It takes: a CLOG object (the DOM object it watches for events) and a handler function. This function takes two arguments: the CLOG object and the event.

In our add-products function below, we create the search input and we listen the key-up event:


(defun add-products (body)
  "Create the search input and a div to contain the products.
  Bind the key-up event of the input field to our filter function."
  (let* ((form (create-form body))
         (input (create-form-element form :input :name "query"
                                     :label
				     (create-label form :content "Filter product: ")))
         (result-div (create-div body :content "" )))

    (set-on-key-up input
                   (lambda (obj event)
                     (format t ":key-up, value: ~a~&" (value obj)) ; logging
                     (setf (text result-div) "") ; this is how we erase the current content.
                     (handle-filter-product result-div obj event)))

    (display-products result-div clog-contest::*products*)))

Below, to find out what is typed in the search input, we use (value obj).


(defun handle-filter-product (div obj event)
  "Search and redisplay products."
  ;TODO: wait a little latency
  (declare (ignorable event))
  (let ((query (value obj)))
    (if (> (length query) 2)
        (display-products div (clog-contest::search-products query))
        (print "waiting for more input"))))

It works \o/

There are some caveats that need to be worked on:

  • if you type a search query of 4 letters quickly, our handler waits for an input of at least 2 characters, but it will be fired 2 other times. That will probably fix the blickering.

And, as you noticed:

  • we didn’t copy-paste a nice looking HTML template, so we have a bit of work with that :/

CLOG is not at all limited to websites like this. You can create games (there is a Snake demo), multiplayer applications (there is a chat demo)... all this by doing everything in the backend, in Common Lisp, with a lot of interactivity under the fingertips. Try it out!


By the way, this post was written in a literate style with Erudite. Everything is written in a .lisp file, and exported to markdown. Read about it here and see its source on GitHub. You can wget this source, open it in your editor and compile the snippets along the way.

For more web stuff, see:

TurtleWareMultipass Translator for CLIM

· 65 days ago

One of interesting concepts in CLIM is typed output. Basically we associate a presentation type with the presented object. When we establish an input context, then presentations which types are a subtype to the input context are selectable with the pointer.

For example:

> (present 42 'integer)
42
;; -> #<presentation>

> (with-output-as-presentation (*standard-output* 15 'integer)
    (format *standard-output* "A minor"))
A minor
;; -> #<presentation>

> (with-output-as-presentation (*standard-output* 13 '(integer 11 17))
    (format *standard-output* "A teenager"))
A teenager
;; -> #<presentation>

> (accept 'integer :prompt "Give me an integer")
;; We select "a minor" with a pointer
Give me an integer: 15
;; -> 15, integer

> (accept '(integer 0 18) :prompt "Please select a minor")
;; We select "a teenager" with a pointer
Please select a minor: 13
;; -> 13, (integer 11 17)

The typed output may get a little tricky. If we try to accept a presentation type (integer 0 18) then "a minor" is not selectable. This is because the type integer is not a subtype of the type (integer 0 18). It seems a bit counterintuitive, because the object has the correct type!

> (presentation-typep 15 '(integer 0 18))
;; -> T
> (presentation-subtypep 'integer '(integer 0 18))
;; -> NIL
> (presentation-typep 13 '(integer 0 18))
;; -> T
> (presentation-subtypep '(integer 11 17) '(integer 0 18))
;; -> T

This behavior may be explained with a short story:

There was a cook who had a day off; they have decided to spend the evening with their friends. When they've entered the restaurant, someone has asked the waiter: "please compliment the cook!" - the waiter obviously ignored "our" cook and left to the kitchen. This is because they are incognito.

Sometimes it is desirable to ignore the fact that some object does not advertise its precise type. An appropriate story would involve a medical doctor who is in the plane when some other person has a heart attack. Then we want to select any suitable object that meets the criteria of being a doctor, despite them being incognito.

It is possible to achieve such relaxed constraints by defining a translator:

> (define-presentation-translator multipass-integer
      (integer nil global-command-table
               :tester ((object context-type)
                        (presentation-typep object context-type)))
      (object context-type)
    (values object context-type))
;; -> #<Translator>

> (accept '(integer 0 18) :prompt "Please select a minor")
;; We select "a minor" with a pointer
Please select a minor: 15
;; -> 15, (integer 0 18)

> (accept '(integer 0 18) :prompt "Please select a minor")
;; We select "a teenager" with a pointer
Please select a minor: 13
;; -> 13, (integer 11 17)

The first two arguments of the presentation translator are from-type and to-type. This is a little more complex, but the applicability of the translator could be determined with the following test:

(and (presentation-subtypep #<object-ptype> from-type)
     (presentation-subtypep #<context-ptype> to-type)
     (tester object #<context-ptype>)

In other words the translator multipass-integer translates all objects of the type integer to the type nil. The latter is a subtype of all types so this translator will work for any context type, because:

> (presentation-subtypep nil '(integer 0 18))
;; -> (t t)

Then we add a test whether the object meets the input context criteria. This is necessary, because otherwise the translator would happily return values that are integers, but they are for instance bigger than 18. At last the translator returns the object, and the context type.

A nice thing about this approach is that translators reside in command tables and it is possible to define such translator that works only locally in "our" application frame without affecting others.

Now we may define a macro that relaxes the constraint for selected types:

> (defmacro define-multipass-translator
      (ptype &optional (command-table 'global-command-table))
    (let ((name (alexandria:symbolicate 'multipass- ptype)))
      `(define-presentation-translator ,name
           (,ptype nil ,command-table
                   :tester ((object context-type)
                            (presentation-typep object context-type)))
           (object context-type)
         (values object context-type))))
> (define-multipass-translator integer)
> (define-multipass-translator doctor)

TurtleWareImplementing a simpleminded REPL from scratch

· 68 days ago

We will start with a very simple (and conforming!) implementation:

(defun rep ()
  (format t "~&~a> " (package-name *package*))
  (shiftf +++ ++ + - (read *standard-input* nil '%quit))
  (when (eq - '%quit)
    (throw :exit "bye!"))
  (shiftf /// // / (multiple-value-list (eval -)))
  (shiftf *** ** * (first /))
  (format t "~&~{ ~s~^~%~}~%" /))

(defun repl ()
  (catch :exit
    (loop (handler-case (rep)
            (condition (c)
              (format *error-output* "~&~a~%~a~%" (class-name (class-of c)) c))))))

Starting this REPL in McCLIM is simple:

(with-output-to-drawing-stream (stream :clx-ttf nil
                                       :text-style (make-text-style :fix nil nil)
                                       :scroll-bars :vertical)
  (let ((*standard-input* stream)
        (*standard-output* stream)
        (*error-output* stream))
    (unwind-protect (repl)
      (close stream))))

Now we may add a graphical debugger:

(defun repl ()
  (catch :exit
    (loop
      (clim-debugger:with-debugger ()
        (with-simple-restart (abort "Return to CLIM's top level.")
          (rep))))))

And create a "real" application frame:

(define-application-frame a-repl ()
  ()
  (:pane :interactor :text-style (make-text-style :fix nil nil)))

(defmethod run-frame-top-level ((frame a-repl) &rest args)
  (declare (ignore args))
  (let ((*standard-input* (frame-standard-input frame))
        (*standard-output* (frame-standard-output frame))
        (*error-output* (frame-error-output frame))
        (*query-io* (frame-query-io frame)))
    (unwind-protect (repl)
      (frame-exit frame))))

(find-application-frame 'a-repl :width 800 :height 600)

Now let's estabilish a context where graphics land below the prompt:

(defun rep ()
  (format t "~&~a> " (package-name *package*))
  (shiftf +++ ++ + - (read *standard-input* nil '%quit))
  (when (eq - '%quit)
    (throw :exit "bye!"))
  (with-room-for-graphics (t :first-quadrant nil)
    (shiftf /// // / (multiple-value-list (eval -))))
  (shiftf *** ** * (first /))
  (format t "~&~{ ~s~^~%~}~%" /))

> (in-package clim-user)
> (draw-rectangle* *standard-output* 10 10 90 90 :ink +dark-blue+)

Let's allow interleaving commands with forms for evaluation:

(defun rep ()
  (multiple-value-bind (command-or-form ptype)
      (accept 'command-or-form :prompt (package-name *package*))
    (when (presentation-subtypep ptype 'command)
      (with-application-frame (frame)
        (return-from rep (execute-frame-command frame command-or-form))))
    (shiftf +++ ++ + - command-or-form)
    (when (eq - '%quit)
      (throw :exit "bye!"))
    (with-room-for-graphics (t :first-quadrant nil)
      (shiftf /// // / (multiple-value-list (eval -))))
    (shiftf *** ** * (first /))
    (format t "~&~{ ~s~^~%~}~%" /)))

(defmethod run-frame-top-level ((frame a-repl) &rest args)
  (declare (ignore args))
  (let ((*standard-input* (frame-standard-input frame))
        (*standard-output* (frame-standard-output frame))
        (*error-output* (frame-error-output frame))
        (*query-io* (frame-query-io frame))
        (*command-dispatchers* '(#\,)))
    (unwind-protect (repl)
      (frame-exit frame))))

A fancy inspector would be nice:

(define-presentation-type result () :inherit-from t)

(define-a-repl-command com-inspect-result ((result 'result :gesture :select))
  (clouseau:inspect result))

(defun rep ()
  (multiple-value-bind (command-or-form ptype)
      (accept 'command-or-form :prompt (package-name *package*))
    (when (presentation-subtypep ptype 'command)
      (with-application-frame (frame)
        (return-from rep (execute-frame-command frame command-or-form))))
    (shiftf +++ ++ + - command-or-form)
    (when (eq - '%quit)
      (throw :exit "bye!"))
    (with-room-for-graphics (t :first-quadrant nil)
      (shiftf /// // / (multiple-value-list (eval -))))
    (shiftf *** ** * (first /))
    (format-textual-list / (lambda (object stream)
                             (format stream " ")
                             (present object 'result :stream stream))
                         :separator #\newline)
    (terpri)))

That and much more is implemented in the system clim-listener. The purpose of this blog post is to show how easy it is to build a sketchy application to help with daily tasks.

Cheers!

P.S. For the inspector and for the debugger load systems clouseau and clim-debugger.

vindarelResources

· 71 days ago

search libraries on

Individual sites:

Screencasts:

Of course, see my Udemy Lisp course!

I also post videos on Youtube, check this out:

Those ones ar great too:

and more on Cliki.

Some games:

  • Kandria - a nice platform game being actively developed and launching soon. Check it out!

  • Spycursion - “a sandbox “edutainment” MMO centered around hacking and espionage which takes place in a near-future world”.

spycursion

TurtleWareMcCLIM backends - Part 2: Stream Output Protocol

· 74 days ago

Table of Contents

  1. Introduction
  2. Creating a graft
  3. Implementing the stream output protocol
  4. Conclusions

Introduction

In "Part I" we've created an SVG backend implementing the Medium Output Protocol. This tutorial is focused on grafts, sheets and CLIM streams.

  • graft: an object representing the screen
  • sheet: an object representing a window
  • CLIM stream: a sheet with a stream interface

We can do various sorts of graphics with the current backend - draw ellipses, embed images, add styling to lines. That is not all that CLIM has to offer. For example, formatting a table requires a second pass over output records to arrange cells in the table grid.

Creating a graft

A graft is an object that represents the output device. A graft may be a computer screen, a document, or some other abstract entity (like a window!). It is used to graft sheets, to query important information about the output, and to transform between the coordinate system of CLIM and the coordinate system of the output device.

What is the output device coordinate system? That depends on the backend. The default screen orientation in SVG is "default"1.

(defclass svg-graft (graft)
  ((density :initarg :dpi    :reader density)
   (region  :initarg :region :reader sheet-native-region)
   (native  :initarg :native :reader sheet-native-transformation)))

(defmethod print-object ((graft svg-graft) stream)
  (print-unreadable-object (graft stream :type t :identity nil)
    (format stream "~ax~a"
            (graft-width graft :units :device)
            (graft-height graft :units :device))))

;;; The constructor is used as a converter - it is possible to supply any valid
;;; combination of the orientation and units and the created graft will have its
;;; native transformation convert supplied parameters to device parameters:
;;;
;;;   (ORIENTATION, UNITS) -> (:DEFAULT :DEVICE)
;;;
(defmethod climb:make-graft ((port svg-port) &key (orientation :default) (units :device))
  (destructuring-bind (port-type &key (width 640) (height 360) (dpi 96) &allow-other-keys)
      (port-server-path port)
    (declare (ignore port-type))
    (let* ((graft (make-instance 'svg-graft :orientation orientation :units units
                                            :mirror (destination port)
                                            :dpi dpi
                                            :region (make-rectangle* 0 0 width height)))
           ;; Transform graft units to 1/dpi (for example 1/96in).
           (units-transformation
             (make-scaling-transformation (/ width (graft-width graft  :units units))
                                          (/ height (graft-height graft :units units))))
           (orientation-transformation (ecase orientation
                                         (:graphics (compose-transformations
                                                    (make-translation-transformation 0 height)
                                                    (make-reflection-transformation* 0 0 1 0)))
                                         (:default +identity-transformation+)))
           (region (make-rectangle* 0 0 width height))
           (native (compose-transformations orientation-transformation units-transformation)))
      (setf (slot-value graft 'region) region)
      (setf (slot-value graft 'native) native)
      graft)))

(defmethod graft-width ((graft svg-graft) &key (units :device))
  (let ((native-width (bounding-rectangle-width (sheet-native-region graft))))
    (ecase units
      (:device native-width)
      (:inches (/ native-width (density graft)))
      (:millimeters (* (/ native-width (density graft)) 25.4))
      (:screen 1))))

(defmethod graft-height ((graft svg-graft) &key (units :device))
  (let ((native-height (bounding-rectangle-height (sheet-native-region graft))))
    (ecase units
      (:device native-height)
      (:inches (/ native-height (density graft)))
      (:millimeters (* (/ native-height (density graft)) 25.4))
      (:screen 1))))

Voila, the graft has been created. Now we can "ask" the screen for its dimensions:

CLIM-USER> (climb:with-port (port :svg  :width 640 :height 360 :dpi 96)
             (setf (mcclim-svg::destination port) :dummy)
             (let ((graft (climb:make-graft port :orientation :graphics :units :device)))
               (print graft)
               (print (sheet-native-transformation graft))
               (print (list (float (graft-height graft :units :inches))
                            (float (graft-width graft :units :inches))))
               (values)))

#<SVG-GRAFT 640x360> 
#<STANDARD-HAIRY-TRANSFORMATION 1 0 0 -1 0 360> 
(3.75 6.6666665) 

Implementing the stream output protocol

I have to disappoint you. We don't have to implement anything, we may use the class clim-stream-pane and benefit from goodies coming with the stream output protocol. To be able to pass the parameter :units and :orientation let's redefine the server path parser to accept additional parameters. To replace the medium with the stream in the drawing context we'll also redefine the method for the function invoke-with-output-to-drawing-stream:

(defun parse-server-path (server-path)
  (destructuring-bind (port-type &rest args)
      server-path
    (list* port-type :id (gensym) args)))

(defmethod invoke-with-output-to-drawing-stream
    (continuation (port svg-port) (destination stream) &rest args)
  (declare (ignore args))
  (destructuring-bind (port-type &key (units :device) (orientation :default) &allow-other-keys)
      (port-server-path port)
    (declare (ignore port-type))
    (setf (destination port) destination)
    (let ((graft (make-graft port :units units :orientation orientation)))
      (let* ((w-in (format nil "~ain" (fmt (graft-width graft :units :inches))))
             (h-in (format nil "~ain" (fmt (graft-height graft :units :inches))))
             (*viewport-w* (graft-width graft :units :device))
             (*viewport-h* (graft-height graft :units :device))
             (bbox (format nil "0 0 ~a ~a" (fmt *viewport-w*) (fmt *viewport-h*)))
             (sheet-region (untransform-region
                            (sheet-native-transformation graft)
                            (sheet-native-region graft)))
             (sheet (make-instance 'clim-stream-pane :port port :background +white+
                                   :region sheet-region)))
        (sheet-adopt-child graft sheet)
        (cl-who:with-html-output (destination destination)
          (:svg :version "1.1" :width w-in :height h-in :|viewBox| bbox
           :xmlns "http://www.w3.org/2000/svg"
           :|xmlns:xlink| "http://www.w3.org/1999/xlink"
           (funcall continuation sheet)))))))

For example, we can now write to the stream with format, draw graphs, and format tables with data:

(defun format-table (stream rows cols)
  (formatting-table (stream)
    (dotimes (row rows)
      (formatting-row (stream)
        (dotimes (col cols)
          (formatting-cell (stream)
            (surrounding-output-with-border (stream)
              (format stream "Row ~s, Col ~s" row col))))))))

(defun format-graph (stream depth breadth)
  (format-graph-from-roots
   (list depth)
   (lambda (obj str)
     (surrounding-output-with-border (stream)
      (format str "Node ~s" obj)))
   (lambda (obj)
     (when (plusp obj)
       (make-list breadth :initial-element (1- obj))))
   :stream stream :orientation :vertical))

(with-output-to-drawing-stream (stream :svg "/tmp/formatted-output.svg" :preview t
                                       :width 640 :height 360)
  (medium-clear-area stream 0 0 640 360)
  (format stream "~a ~a~%"
          (graft-units (graft stream))
          (graft-orientation (graft stream)))
  (format-graph stream 3 2)
  (terpri stream)
  (format-table stream 3 2))

Formatted output

What is the coordinate system of CLIM? That is entirely up to the graft! When we graft a sheet to the graft with units :millimeters, then native coordinates are specified in millimeters. When the orientation is :graphics, then the origin is located at the lower-left corner and Y grows upwards.

(with-output-to-drawing-stream (stream :svg "/tmp/upside-down-output.svg" :preview t
                                       :width 640 :height 360
                                       :orientation :graphics)
  (medium-clear-area stream 0 0 640 360)
  (format stream "~a ~a~%"
          (graft-units (graft stream))
          (graft-orientation (graft stream)))
  (format-graph stream 3 2)
  (terpri stream)
  (format-table stream 3 2))

Formatted output in graphics orientation

Let's throw in one extra feature to our drawing backend that scales the output to always fit in the viewport. This is similar to other drawing backends. Notice, that we compare the history size in the stream coordinates with dimensions of the graft in specified units.

(defun scale-to-fit (continuation stream)
  (with-output-recording-options (stream :record t :draw nil)
    (funcall continuation stream))
  (let* ((history (stream-output-history stream))
         (graft (graft stream))
         (scale (min (/ (graft-width graft :units (graft-units graft))
                        (bounding-rectangle-width history))
                     (/ (graft-height graft :units (graft-units graft))
                        (bounding-rectangle-height history))))
         (transformation (compose-transformation-with-scaling
                          (make-translation-transformation
                           (- (bounding-rectangle-min-x history))
                           (- (bounding-rectangle-min-y history)))
                          scale scale)))
    (with-output-recording-options (stream :draw t :record nil)
      (climi::letf (((sheet-transformation stream) transformation))
        (replay history stream)))))

(defmethod invoke-with-output-to-drawing-stream
    (continuation (port svg-port) (destination stream) &rest args)
  (declare (ignore args))
  (destructuring-bind (port-type &key (units :device) (orientation :default) (scale-to-fit nil)
                       &allow-other-keys)
      (port-server-path port)
    (declare (ignore port-type))
    (setf (destination port) destination)
    (let ((graft (make-graft port :units units :orientation orientation)))
      (let* ((w-in (format nil "~ain" (fmt (graft-width graft :units :inches))))
             (h-in (format nil "~ain" (fmt (graft-height graft :units :inches))))
             (*viewport-w* (graft-width graft :units :device))
             (*viewport-h* (graft-height graft :units :device))
             (bbox (format nil "0 0 ~a ~a" (fmt *viewport-w*) (fmt *viewport-h*)))
             (sheet-region (if scale-to-fit
                               +everywhere+ ; don't clip when fitting
                               (untransform-region
                                (sheet-native-transformation graft)
                                (sheet-native-region graft))))
             (sheet (make-instance 'clim-stream-pane :port port :background +white+
                                                     :region sheet-region)))
        (sheet-adopt-child graft sheet)
        (cl-who:with-html-output (destination destination)
          (:svg :version "1.1" :width w-in :height h-in :|viewBox| bbox
           :xmlns "http://www.w3.org/2000/svg"
           :|xmlns:xlink| "http://www.w3.org/1999/xlink"
           (if scale-to-fit
               (scale-to-fit continuation sheet)
               (funcall continuation sheet))))))))

On the first image we see the output drawn with the unit being :millimeters, and clipped by the viewport specified in the device units. Notice that the last letter of the text is wrapped - that indicates that the stream region is correct.

(with-output-to-drawing-stream (stream :svg "/tmp/truncated-millimeters.svg" :preview t
                                       :width 640 :height 360
                                       :orientation :graphics
                                       :units :millimeters
                                       :scale-to-fit nil)
  (medium-clear-area stream 0 0 640 360)
  (draw-circle* stream 0 0 100 :ink +dark-red+)
  (format stream "~a ~a~%"
          (graft-units (graft stream))
          (graft-orientation (graft stream)))
  (format-graph stream 3 2)
  (terpri stream)
  (format-table stream 10 8))

The output in millimeters is truncated to the viewport size.

On the second image we see a clipped image which units are :device (the same as the viewport).

(with-output-to-drawing-stream (stream :svg "/tmp/truncated-millimeters.svg" :preview t
                                       :width 640 :height 360
                                       :orientation :graphics
                                       :units :millimeters
                                       :scale-to-fit nil)
  (medium-clear-area stream 0 0 640 360)
  (draw-circle* stream 0 0 100 :ink +dark-red+)
  (format stream "~a ~a~%"
          (graft-units (graft stream))
          (graft-orientation (graft stream)))
  (format-graph stream 3 2)
  (terpri stream)
  (format-table stream 10 8))

The output in device units is truncated to the viewport size.

The third image scales the history to fit in the viewport. For giggles we'll specify the unit to be :inches2.

(with-output-to-drawing-stream (stream :svg "/tmp/truncated-inches.svg" :preview t
                                       :width 15 :height :compute
                                       :orientation :graphics
                                       :units :inches
                                       :scale-to-fit t)
  (surrounding-output-with-border (stream :background +white+ :filled t)
    (draw-circle* stream 0 0 100 :ink +dark-red+)
    (format stream "~a ~a~%"
            (graft-units (graft stream))
            (graft-orientation (graft stream)))
    (format-graph stream 3 2)
    (terpri stream)
    (format-table stream 10 8)))

The output in inches is scaled to the viewport size.

Conclusions

This concludes the case study of the SVG backend. The code in McCLIM will most likely evolve with time to fix some issues, and to account for new interfaces. But with this knowledge you, the backend developer, should have a good hang of things that need to be done when implementing a new drawing backend for McCLIM.

The next part of the tutorial will cover input processing in the interactive backend.

PS You may leave me a feedback on my email or join our irc channel.
PPS You may support my FLOSS work and blogging by becoming my patron.
PPPS The SVG backend is already available in the master branch of McCLIM.

Footnotes

1 PostScript on the other hand is oriented in :graphics coordinates.

2 For even more giggles and some fine keming specify the unit :screen.

Eitaro FukamachiBuilding Docker images for Common Lisp applications

· 74 days ago

Hello, all Common Lispers.

Today, I will introduce how to build a Docker container for Common Lisp applications.

Docker is designed to provide the same execution environment throughout the development, CI, and production environment.

It is losing momentum as an infrastructure to support increasingly large web applications, but it is still useful when viewed as a development tool.

Recently, Eric Timmons has stably maintained Docker images provided by the Common Lisp Foundation, and I have also published SBCL and CCL images containing Roswell.

This article will show how to write a Dockerfile based on these images and create your image to run a Quicklisp or Common Lisp application.

Using clfoundation/sbcl

First, let's discuss using Common Lisp Foundation's image.

This image provides only a Lisp implementation, which means you must install Quicklisp if necessary.

Here's an example of a Dockerfile based on the CL Foundation image with Quicklisp:

FROM clfoundation/sbcl:2.2.2-slim
ARG QUICKLISP_DIST_VERSION=2022-02-20

WORKDIR /app
COPY . /app

ADD https://beta.quicklisp.org/quicklisp.lisp /root/quicklisp.lisp

RUN set -x; \
  sbcl --load /root/quicklisp.lisp \
    --eval '(quicklisp-quickstart:install)' \
    --eval '(ql:uninstall-dist "quicklisp")' \
    --eval "(ql-dist:install-dist \"http://beta.quicklisp.org/dist/quicklisp/${QUICKLISP_DIST_VERSION}/distinfo.txt\" :prompt nil)" \
    --quit && \
  echo '#-quicklisp (load #P"/root/quicklisp/setup.lisp")' > /root/.sbclrc && \
  rm /root/quicklisp.lisp

It allows specifying explicitly to fix the Quicklisp dist version.

Dockerfile is like a step-by-step guide for creating an execution environment, and docker build allows you actually to create a Docker image.

$ docker build -t clf-sbcl .
$ docker run --rm -it clf-sbcl

# Use the dist version '2022-04-01'
$ docker build -t clf-sbcl --build-arg QUICKLISP_DIST_VERSION=2022-04-01 .

This image provides only a minimal setup, for better or worse. Notably that they provide images for Windows, if you need to run it on Windows, this image is a strong candidate.

If you are not concerned about the size of the final Docker image, the Dockerfile can be simplified a bit by using the non-slim base image (clfoundation/sbcl:2.2.2-slim -> clfoundation/sbcl:2.2.2).

FROM clfoundation/sbcl:2.2.2
ARG QUICKLISP_DIST_VERSION
ARG QUICKLISP_ADD_TO_INIT_FILE=true

WORKDIR /app
COPY . /app

RUN set -x; \
  /usr/local/bin/install-quicklisp

Using fukamachi/sbcl

Next, I will introduce the Docker images I provide.

The difference that this image has is that it contains Roswell and Quicklisp.

FROM fukamachi/sbcl:2.2.3

WORKDIR /app
COPY . /app

RUN set -x; \
  ros install qlot && qlot install

ENV QUICKLISP_HOME /app/.qlot/

Since it is not possible to specify the dist version of Quicklisp, it is recommended to use Qlot; see the previous article for how to use Qlot.

# Create qlfile.lock beforehand
$ touch qlfile
$ echo .qlot/ | tee -a .gitignore | tee -a .dockerignore
$ docker run --rm -it -v $PWD:/app fukamachi/qlot install

$ docker build -t fukamachi-sbcl .
$ docker run --rm -it fukamachi-sbcl

The good thing about this image is the Dockerfile is short, and Qlot is easy to be installed. On the other hand, the only architectures offered are AMD64 and ARM64 for Linux.

multi-stage build

If you do not want to include Qlot in the final image, you can use Docker's multi-stage build.

The "multi-stage build" is a mechanism that separates the build environment from the execution environment so that tools needed only at build time are not left in the final image.

It can be used by writing multiple FROM clauses in the Dockerfile, like this:

FROM fukamachi/qlot:1.0.1 AS build-env

WORKDIR /app
COPY . /app

RUN set -x; \
  qlot install

FROM fukamachi/sbcl:2.2.3

WORKDIR /app
COPY --from=build-env /app /app

ENV QUICKLISP_HOME /app/.qlot/

Only the .qlot directory after qlot install is copied to the final image.

The final image can also be clfoundation/sbcl based, and in this way, it is possible to create a final image that does not even include Roswell.

FROM fukamachi/qlot:1.0.1 AS build-env

WORKDIR /app
COPY . /app

RUN set -x; \
  qlot install

FROM clfoundation/sbcl:2.2.2-slim

WORKDIR /app
COPY --from=build-env /app /app

ENTRYPOINT ["sbcl", "--load", ".qlot/setup.lisp"]

Don't forget to load .qlot/setup.lisp to enable the project-local Quicklisp.

Publish the Docker image to Docker Registry

Built images can be pushed to the Docker registry to allow people to use them or deploy them to remote environments.

Several Docker registry services are available: Docker Hub, officially provided by Docker, and GitHub Container Registry, provided by GitHub, is well known for OSS use.

You can also use GitHub Actions to publish an image to the registry each time you push to GitHub. It is easily accomplished using docker/build-and-push.

vindarelI Am Creating a Common Lisp Video Course on Udemy (free video previews) &#127909;

· 77 days ago

Everyone, let me celebrate a little bit: I am creating a Common Lisp video course on the Udemy platform. I’m several dozen hours in already and it’s taking a good shape! It is so much more time consuming to create videos than to write a tutorial O_o But I like what’s in there already, although there isn’t everything I want to teach, of course. I’m working on more content. Everything will come in time, and meanwhile you can buy the course: you’ll get future content for “free” ;) Yes the course is to sell, hopefully it will help me concentrate more on my CL activities (BTW, dear reader, here’s a 50% off coupon for April, 2022, and if you are a student drop me a line). Currently 4 videos are freely viewable, and I’m also posting new videos on Youtube (that one is on how to create a new Common Lisp project with my project generator) but more on that later. You currently have more than 3 hours of learning material.

You probably know me from my blog and my Lisp activities. I am vindarel, I contribute to community-based learning resources such as the Cookbook and I use CL in productionę. Honestly, I missed a Cookbook-like resource as a beginner. I dumped there a lot of content and must-know tricks that I either learned the hard way, either learned by chance. But I still want Common Lisp to be easier to learn and, for that, there is the video media. Consequently, I truely think this course is today’s most efficient way to learn Common Lisp.

So, what can you learn already in my course?

I want it to be practical: you will learn Lisp the language in order to build real-world stuff.

Udemy

Chapter 1 is how to get started

1.1. We start by installing SBCL on our machine (showed for Unix, links for Windows). This one is 15m long and is 🆓 free to watch. We see how to start our Lisp, how to write “hello world”, we understand the output, we add readline support to the SBCL default REPL in the terminal, we disable the interactive debugger, and we have a few words on Lisp implementations and GNU CLISP in particular.

1.2. We see how to run Lisp code, the simplest way. We write a code snippet with a simple text editor and we run it with sbcl’s --script and --load flags. We use the LOAD function.

1.3. We see how to use Portacle (the ready-to-use, multiplatform image shipping Emacs, SBCL, Quicklisp, Git and a couple handy Emacs packages).

Chapter 2 is about Lisp basics

2.1. I heard Lisp beginners who needed a recap on the Lisp syntax and the evualation model. This one is also 🆓 available to everyone. We see: the prefix notation, that everything is an expression, the evaluation model (and the exception of macros). Code is data is code... right?

2.2. How to define variables, at the toplevel or locally. How to lexically re-bind dynamic variables, the gotcha, the alternative.

2.3. and conditionals (if, when, #+or...)

That’s it for now for this chapter (yes, we’ll see data structures, but for now I refer you to the Cookbook, on the page that I also authored. You are armed to read it.).

Chapter 3 is about iteration. It is made of shorter videos that typically sum up in 5 minutes

...what took me a long time to learn or discover, a way longer time to admit.

3.1 Iterating over lists and vectors (with 🆓 free preview). loop, dolist and other libraries of the ecosystem.

3.2 then: Iterating over a hash-table keys and values. We see 5 different ways in 5 minutes.

3.3 Iterating a fixed or infinite number of times, and we take the opportunity to build our first read-eval-print-loop.

3.4. Here, we take a high level overview of loop and we study some gotchas. We see a practical example from an answer to last year’s Advent Of Code.

Chapter 4 teaches everything you need to know about functions

(with a sneak peak into CLOS):

4.1. How to create named functions, how to handle all types of arguments (🆓 free preview). We see defun, returned values, required arguments, optional arguments, key arguments, how to set a default value, how to know if an argument was supplied, &rest, example of apply, feature flags...

4.2. Referencing functions, redefining functions locally, accessing documentation

4.3. Multiple Return Values (they are NOT like returning a list or a tuple!!!)

4.4. Higher Order Functions: how to give functions as arguments, member, the :test keyword, map and mapcar, lambda, how to generate functions, what are symbols, setf symbol-function. A word on currying and being a Lisp-2.

4.5. Closures

4.6. setf functions

4.7. Generic Functions (quick intro to CLOS): they allow to write functions that dynamically dispatch on the type of their arguments. What we see (quickly): defmethod, defgeneric.

and, lastly (for now):

Chapter 5 shows how to work with projects

5.1 How to work with an existing project. ASDF, Quicklisp, SLIME shortcuts, some useful asdf functions...

5.2. How to create a new project. As a bonus, see again my new video on Youtube: Common Lisp: how to create a new project (demo of my project generator).

5.3. What are systems and packages anyways? Demo. Gotcha when creating a package. Finding all external symbols of a package.


Soooo

Who is this course for by the way?

I must warn that this course is not for total newcomers in programming. You should know what variables and functions are. You should knew that Common Lisp is a language of the Lisp family! (I tell more in the presentation video, but still).

Lisp newbies are welcome because I introduce Lisp basics (syntax, evaluation model) and I show how to install everything. It would help if you know what is a language of the Lisp family.

This course is mainly targetting young(ish) profesional developers like me, who feel they deserve a more fun, comfy, compiled and fast programming language.

It is for Python or JavaScript programmers frustrated by the unstability of their ecosystem,

for students of computer science who want to discover why Lisp still has un-matched alien technology inside (and maybe for *your* students?),

for Clojurists who want to transition quickly to a bare-metal Lisp,

or simply for your friend or colleague to whom you are trying to sell the power of Lisp ;)

Closing thoughts

If you are already a programmer: you can watch the videos at speed 1.25 or higher, but try to not skip content. You can start by the chapter of your choice. Use the captions, they are manually edited.

Thanks kindly for your support in that new journey of mine! Hope you’ll enjoy the content (I know you’ll learn a few tricks).

Learn Common Lisp now! It's a tool for a lifetime.

And have fun!

   🎥   Common Lisp: from novice to effective developer    🎥


  • project’s GitHub
  • a known issue: I started with a meh microphone. I bought a new one, sound is now good, but older videos were not re-captured yet.

ADDENDUM: the Lisp philosophy revealed

Udemy auto-generates captions for your videos. You can manually edit them... but sometimes I was tempted not to, as they can reveal some hidden truth^^

I can see my new prince here

reveals my respect for “print”.

Survivor needs votes.

I never thought about that O_o (that’s for “the variable name is unbound”)

CL might be more active than you think:

Google and China’s also actively contributes, as we said.

for “Google engineers also actively contribute to the SBCL implementation”. My cute accent is well interpreted beyond hopes (and it is not wrong, hello fellow chinese lispers o/ ).

and yes, I use communism prediction.

“I use CL in production”...

Lisp is used in more places than you think:

I want to give you a few tips for when you are dealing with cruise people

aka “for when you are dealing with macros”. And yes:

my crew is special

indeed :p “the loop macro is special”.

We knew a so called Lisp curse... there’s a conspiracy too!

the conspiracy control key

is “C-c C-k”.

and we can create with controversy

instead of “and we can quit with C-c C-c”.

you have to respect the water

you have to respect the order.

discipline is nested in your code.

for “deeply nested in your code”

I’ll show you from Oslo

= “I’ll show you from Emacs and Slime”, that’s quite an interpretation but Emacs is a great place to live in too.

And I also like that one:

I’m actually a terrific

that is the translation of the name “Eitaro Fukamachi”. Add “coder” and that matches :)

TurtleWareMcCLIM backends - Part I: Medium Output Protocol

· 78 days ago

Table of Contents

  1. Introduction
  2. Using a drawing backend
  3. Defining a new backend
  4. Implementing the medium protocol
    1. The first 90 percent of drawing
    2. The following 9 percent of drawing
      1. Basic shapes
      2. The line style
      3. The text style
      4. Arbitrary clip regions
    3. The last 1% of drawing
      1. Images, tiles and transformed patterns
      2. Recursive designs
      3. Masked in- and out- composition
    4. Features that are not implemented
  5. Conclusions

Introduction

CLIM backends may be categorized as:

  • interactive: handles I/O and is used by applications
  • drawing: renders graphics in a specified format

McCLIM currently provides the following drawing backends:

  • PostScript : .ps
  • PDF : .pdf
  • RasterImage: .png, .tiff etc. (using the library opticl)

This tutorial focuses on using, defining and implementing drawing backends. It is also applicable to the rendering part of the interactive backend.

In this part we'll focus on using drawing backends, defining a new backend and on implementing output protocols.

Using a drawing backend

A typical use of a drawing backend involves creating a stream and drawing on it with operators specified in Part IV: Sheet and Medium Output Facilities. The PostScript backend is special because it is defined in the specification.

(with-open-file (file-stream "/tmp/file.ps" :direction :output :if-exists :supersede)
  (with-output-to-postscript-stream (stream file-stream)
    (draw-rectangle* stream 10 10 90 90 :ink clim:+red+)))

Executing the form above will create a postscript document with a red rectangle drawn near the top-left corner of the page. Notice that the second argument passed to the macro is a stream - this provides an important insight: drawing backends are filters that convert CLIM operations to a device-specific format1.

McCLIM provides an extension that unifies the access to drawing:

;; similar to with-output-to-postscript-stream
(with-output-to-drawing-stream (stream :ps "/tmp/file.ps")
  (draw-rectangle* stream 10 10 90 90 :ink +red+))

;; interactive backends open a window by default
(with-output-to-drawing-stream (stream :clx-ttf nil)
  (draw-rectangle* stream 10 10 90 90 :ink +red+))

;; requires loading the system "mcclim-raster-image"
(with-output-to-drawing-stream (stream :raster "/tmp/file.png" :width 200 :height 200)
  (draw-rectangle* stream 10 10 90 90 :ink +red+))

The first argument is a variable to be bound to a drawing stream, the second argument is a symbol that designates the backend, and the third argument is a destination. All remaining arguments are parsed by the backend.

Let's define a smoke test for different shapes and drawing styles:

(defun test-drawing (stream)
  (with-drawing-options (stream :transformation (make-reflection-transformation* 0 180 100 180))
    (draw-ellipse* stream 320 180 300 0 0 150 :ink +grey+)
    (draw-ellipse* stream 320 180 300 0 0 150 :ink +black+ :filled nil
                                              :line-dashes '(8 16)
                                              :line-thickness 8)
    (draw-circle* stream 220 250 25 :ink +blue+)
    (draw-circle* stream 420 250 25 :ink +blue+)
    (draw-point* stream 220 250 :ink +cyan+ :line-thickness 15)
    (draw-point* stream 420 250 :ink +cyan+ :line-thickness 15)
    (draw-rectangle* stream 125 150 150 175 :ink +deep-pink+ )
    (draw-rectangle* stream 515 150 490 175 :ink +deep-pink+ )
    (draw-polygon* stream (list 320 225 280 150 360 150) :filled t :ink +orange+)
    (draw-polygon* stream (list 320 225 280 150 360 150) :filled nil :ink +black+
                                                         :line-thickness 4
                                                         :line-joint-shape :round)
    (draw-line* stream 175 125 465 125 :ink +blue+ :line-thickness 4 :line-cap-shape :no-end-point)
    (draw-line* stream 175 130 465 130 :ink +blue+ :line-thickness 4 :line-cap-shape :butt)
    (draw-line* stream 175 135 465 135 :ink +blue+ :line-thickness 4 :line-cap-shape :square)
    (draw-line* stream 175 140 465 140 :ink +blue+ :line-thickness 4 :line-cap-shape :round)

    (let ((smile '(200 100 280 75 360 75 440 100 200 100 200 100 200 100)))
      (draw-bezigon* stream smile :filled t :ink +red+)
      (draw-bezigon* stream smile :filled nil :ink +black+ :line-dashes nil :line-thickness 4))
    (draw-circle* stream 0 0 25 :ink +red+)
    (draw-circle* stream 320 300 20 :ink +cyan+)
    (draw-line* stream 300 300 340 300 :line-thickness 1)
    (draw-text* stream "McCLIM" 320 300
                :align-x :center :align-y :center :text-size :large
                :transform-glyphs t
                :ink +dark-green+
                :transformation (make-rotation-transformation* (/ pi 6) 320 300))))

;;; (ql:quickload 'mcclim-raster-image)
(with-output-to-drawing-stream (stream :raster "smoke-test.png" :width 640 :height 360)
  (test-drawing stream))

img

Defining a new backend

It is time to define the SVG backend. The whole implementation will reside in a package mcclim-svg. Packages clime and climb export symbols for core extensions provided by McCLIM.

;; (ql:quickload '("mcclim" "mcclim-bitmaps" "mcclim-fonts/truetype"
;;                 "alexandria" "cl-who" "cl-base64" "flexi-streams"))

(defpackage #:mcclim-svg
  (:use #:clim #:clime #:climb #:clim-lisp)
  (:local-nicknames (#:alx #:alexandria)))

(in-package #:mcclim-svg)

A port is an object that represents a display server. It is used for specialization, handling input and maintaining allocated resources. New backends should subclass basic-port that implements essential defaults. The slot destination stores the stream for writing, and the slot resources is a hash table storing the defined resource identifiers.

(defclass svg-port (mcclim-truetype:ttf-port-mixin basic-port)
  ((destination :accessor destination)
   (resources :initform (make-hash-table :test #'equal) :reader resources)))

Our port will have a simple resource manager. In SVG some resources may be defined in the section <defs> and then referenced by id. The manager will try to return the identifier, and if absent, assign an unique string. A macro ensure-resource-id executes the body conditionally.

(defun resource-id (holder resource)
  (let ((ht (resources (port holder))))
    (alx:ensure-gethash resource ht
      (format nil "r~4,'0d" (hash-table-count ht)))))

(defmacro ensure-resource-id ((medium object) &body body)
  (alx:with-gensyms (foundp)
    `(multiple-value-bind (^resource-id ,foundp)
         (resource-id ,medium ,object)
       (unless ,foundp ,@body)
       ^resource-id)))

Every port is designated by a symbol and parameterized by a supplied server path. The server path looks like this: (port-designator . port-parameters). Instances of the same class are distinguished by the use of equal to compare canonical server paths.

A backend may provide a server-path parser to ensure a consistent order of arguments, to add additional options, and to sanitize parameters.

The svg-port represents a single SVG document. To avoid a situation where the same port is used for two documents we append a unique id so that find-port will always return a fresh instance.

(defun parse-server-path (server-path)
  (destructuring-bind (port-type &rest args &key dpi width height)
      server-path
    (declare (ignore dpi width height))
    (list* port-type :id (gensym) args)))

The function find-port-type is used by McCLIM to map symbols to port classes and their corresponding parsers.

(defmethod find-port-type ((port (eql :svg)))
  (values (find-class 'svg-port) 'parse-server-path))

For example:

CLIM-USER> (find-port :server-path :svg) ;-> #<MCCLIM-SVG::SVG-PORT #x302007C9D9CD>, NIL
CLIM-USER> (find-port :server-path :svg) ;-> #<MCCLIM-SVG::SVG-PORT #x302007C9ADAD>, NIL

A drawing backend is created by a macro with-output-to-drawing-stream that expands to a call to the function invoke-with-output-to-drawing-stream. We will define a DWIM method that coerces the destination to a stream. We call destroy-port in the end to ensure that it is not referenced in the core.

;;; Ensure that the "real" method receives a stream.
(defmethod invoke-with-output-to-drawing-stream
    (cont (port (eql :svg)) destination &rest args &key (preview nil))
  (let* ((args (alx:remove-from-plist args :preview))
         (port (find-port :server-path (list* port args))))
    (unwind-protect
         (etypecase destination
           ((or string pathname)
            (with-open-file (stream destination :direction :output
                                                :if-exists :supersede
                                                :if-does-not-exist :create
                                                :element-type 'character)
              (invoke-with-output-to-drawing-stream cont port stream))
            (when preview
              (when (eq preview t)
                (setf preview "xdg-open"))
              (uiop:launch-program (format nil "~a ~a" preview destination)))
            destination)
           (null
            (with-output-to-string (stream nil :element-type 'character)
              (invoke-with-output-to-drawing-stream cont port stream)))
           ((eql t)
            (let ((stream *standard-output*))
              (invoke-with-output-to-drawing-stream cont port stream))
            nil)
           (stream
            (invoke-with-output-to-drawing-stream cont port destination)))
      (destroy-port port))))

Lisp numbers can't be directly serialized to SVG - strings like 1/4 or 1.0d0 won't be parsed as numbers. We introduce a function fmt that formats the number to either an integer or a float without its type indicator:

(defun fmt (number)
  (if (integerp number)
      (format nil "~d" number)
      (format nil "~f" number)))

The "real" method is specialized to the port. CLIM and SVG default coordinate systems are the same. That is not always true, for example PDF and PostScript origins are located at the lower-left corner and Y grows upwards. We introduce a necessary boilerplate to create a document. We specify the document size in real-size units depending on the dpi.

(defvar *viewport-w*)
(defvar *viewport-h*)

(defmethod invoke-with-output-to-drawing-stream
    (continuation (port svg-port) (destination stream) &rest args)
  (declare (ignore args))
  (destructuring-bind (port-type &key (dpi 96) (width 640) (height 360) &allow-other-keys)
      (port-server-path port)
    (declare (ignore port-type))
    (setf (destination port) destination)
    (let ((medium (make-medium port nil))
          (clip (make-rectangle* 0 0 width height))
          (w-in (format nil "~ain" (fmt (/ width dpi))))
          (h-in (format nil "~ain" (fmt (/ height dpi))))
          (bbox (format nil "0 0 ~a ~a" (fmt width) (fmt height)))
          (*viewport-w* width)
          (*viewport-h* height))
      (with-drawing-options (medium :clipping-region clip)
        (cl-who:with-html-output (destination destination)
          (:svg :version "1.1" :width w-in :height h-in :|viewBox| bbox
           :xmlns "http://www.w3.org/2000/svg"
           :|xmlns:xlink| "http://www.w3.org/1999/xlink"
           (funcall continuation medium)))))))

Let's try it!

CLIM-USER> (with-output-to-drawing-stream (stream :svg nil)
             (declare (ignore stream)))
"<svg version='1.1' width='6.6666665'in height='3.75in' viewBox='0 0 640 360' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'></svg>"

Implementing the medium protocol

A drawing backend must implement CLIM's medium protocol. It should also implement CLIM's output stream protocol.

A medium is the object that is responsible for maintaining the drawing context and for performing the drawing. New mediums should subclass basic-medium that implements essential defaults. We associate the medium class with a port by specializing the method make-medium.

(defclass svg-medium (mcclim-truetype:ttf-medium-mixin basic-medium)
  ())

(defmethod make-medium ((port svg-port) sheet)
  (make-instance 'svg-medium :stream sheet :port port))

To CLIM, the drawable accessed with a reader medium-drawable is an opaque object. It may be a file stream, a pixmap, an ffi handler etc. Handling the drawable is the "know-how" contributed by the backend.

(defmethod medium-drawable ((medium svg-medium))
  (if (medium-sheet medium)
      (call-next-method)
      (destination (port medium))))

The first 90 percent of drawing

Drawing usually involves establishing a context that specifies the clipping region, the paint color, and other drawing properties. Manually handling it for each drawing method may be a bit tedious - that's why we'll introduce a macro for establishing the context for us.

(defmacro with-drawing-context ((drawable-var medium mode) &body body)
  (alx:with-gensyms (cont)
    `(flet ((,cont (,drawable-var) ,@body))
       (declare (dynamic-extent (function ,cont)))
       (invoke-with-drawing-context (function ,cont) ,medium ,mode))))

Clipping in SVG may be achieved either by defining a clip path or by defining a mask. The former is much more performant while the latter is more flexible. The function medium-clip will return the clip type and its value:

(defun url (id)
  (format nil "url(#~a)" id))

(defgeneric medium-clip (medium region)
  (:method ((medium svg-medium) (region (eql +nowhere+)))
    (values :none nil))
  (:method ((medium svg-medium) (region (eql +everywhere+)))
    (values :clip "none"))
  (:method ((medium svg-medium) (region bounding-rectangle))
    (let ((id (ensure-resource-id (medium (cons :clip region))
                (cl-who:with-html-output (drawable (medium-drawable medium))
                  (:defs (:|clipPath| :id ^resource-id (draw-design medium region)))))))
      (values :clip (url id))))
  (:method ((medium svg-medium) (region standard-region-complement))
    (error "SVG: Unsupported clip ~s." 'standard-region-complement))
  (:method ((medium svg-medium) (region standard-region-intersection))
    (error "SVG: Unsupported clip ~s." 'standard-region-intersection)))

The paint color and opacity in SVG are treated separately. The function medium-design-ink will return two values that are suitable as values for attributes "fill", "stroke" and "opacity".

;;; This would be so much nicer had SVG accept RGBA as a fill from the get-go.
(defun uniform-design-values (design)
  (multiple-value-bind (r g b a) (color-rgba design)
    (values (format nil "#~2,'0x~2,'0x~2,'0x"
                    (truncate (* r 255))
                    (truncate (* g 255))
                    (truncate (* b 255)))
            (format nil "~,2f" a))))

(defgeneric medium-design-ink (medium design)
  (:method ((medium svg-medium) (design color))
    (uniform-design-values design))
  (:method ((medium svg-medium) (design opacity))
    (uniform-design-values design))
  (:method ((medium svg-medium) (design climi::uniform-compositum))
    (uniform-design-values design))
  (:method ((medium svg-medium) (design indirect-ink))
    (medium-design-ink medium (indirect-ink-ink design)))
  (:method ((medium svg-medium) design)
    (warn "SVG: Unsupported design ~s." (class-name (class-of design)))
    (medium-design-ink medium +deep-pink+)))

Notice that unsupported clips lead to an error, while unsupported paints signal a warning and use a characteristic color. Finally, all drawing is expected to be transformed by the device transformation, which maps points from a drawing plane to the device plane. The SVG transformation matrix is specified in a column-major order - unlike McCLIM!

;;; SVG transformation matrix is in column-major order.
(defun svg-transform (transformation)
  (multiple-value-bind (mxx mxy myx myy tx ty)
      (climi::get-transformation transformation)
    (format nil "matrix(~f ~f ~f ~f ~f ~f)"
            (fmt mxx) (fmt myx)
            (fmt mxy) (fmt myy)
            (fmt  tx) (fmt  ty))))

Finally, let's write the function that establishes the drawing context. Did you notice that in the function medium-clip we call the function draw-design? We need to prevent infinite recursion, and we do that with *configuring-device-p*. We'll use the hierarchical SVG element <g> to configure the device.

(defvar *configuring-device-p* nil)

(defun invoke-with-drawing-context (cont medium mode)
  (declare (ignorable mode))
  (alx:when-let ((drawable (medium-drawable medium)))
    (when *configuring-device-p*
      (return-from invoke-with-drawing-context
        (funcall cont drawable)))
    (cl-who:with-html-output (drawable)
      (labels ((configure-clip ()
                 (multiple-value-bind (clip value)
                     (medium-clip medium (medium-clipping-region medium))
                   (when clip
                     (cl-who:htm
                      (ecase clip
                        (:clip (cl-who:htm (:g :clip-path value (configure-draw))))
                        (:mask (cl-who:htm (:g :mask      value (configure-draw))))
                        (:none))))))
               (configure-draw ()
                 (multiple-value-bind (paint opacity)
                     (medium-design-ink medium (medium-ink medium))
                   (let ((transformation (svg-transform (medium-device-transformation medium))))
                     (cl-who:htm
                      (ecase mode
                        (:area
                         (cl-who:htm
                          (:g :transform transformation :fill paint :opacity opacity
                              (configure-area))))
                        (:path
                         (cl-who:htm
                          (:g :transform transformation :fill "none" :stroke paint :opacity opacity
                              (configure-path))))
                        (:text
                         (cl-who:htm
                          (:g :transform transformation :fill paint :opacity opacity
                              (configure-text)))))))))
               (configure-area ()
                 (funcall cont drawable))
               (configure-path ()
                 (funcall cont drawable))
               (configure-text ()
                 (funcall cont drawable)))
        (let ((*configuring-device-p* t))
          (configure-clip))))))

The first drawing function we will implement is medium-draw-polygon*. It is responsible for drawing both polygons and polylines. All coordinates must be sanitized as floats.

(defmethod medium-draw-polygon* ((medium svg-medium) coord-seq closed filled)
  (alx:when-let ((drawable (medium-drawable medium)))
    (multiple-value-bind (rgb-color opacity)
        (multiple-value-bind (r g b a) (color-rgba (medium-ink medium))
          (values (format nil "#~2,'0x~2,'0x~2,'0x"
                          (truncate (* r 255))
                          (truncate (* g 255))
                          (truncate (* b 255)))
                  (format nil "~,2f" a)))
      (let* ((transformation (medium-device-transformation medium))
             (coord-seq (climi::transform-positions transformation coord-seq))
             (points (format nil "~{~f~^ ~}" (coerce coord-seq 'list)))
             (fill   (if filled rgb-color "none"))
             (stroke (if filled "none" rgb-color)))
        (if (or filled closed)
            (cl-who:with-html-output (stream drawable)
              (:polygon :points points :fill fill :stroke stroke :opacity opacity))
            (cl-who:with-html-output (stream drawable)
              (:polyline :points points :fill fill :stroke stroke :opacity opacity)))))))

For example:

CLIM-USER> (with-output-to-drawing-stream (stream :svg "first-drawing.svg" :preview t)
             (with-rotation (stream (/ pi 6))
               (test-drawing stream)))
"first-drawing.svg"

img

Wow! We have a complete picture, why? The explanation is simple - the default drawing methods fall back to medium-draw-polygon*. In other words to have a very fine working prototype it is enough to define one function. That said, there are important shortcomings of this solution:

  • approximating shapes with polygons is expensive (the file has over 400KB!)
  • incomplete drawing context (line styles, text styles)

Fret not, we will address all of them. That said, please take a moment to look back at the source code buffer and appreciate - roughly 200 lines of code is enough to have the first working prototype - we can even render the text!

The following 9 percent of drawing

Approximating drawing with polygons is fun and games but it is time to get serious. SVG provides elements that can be utilized by McCLIM for drawing.

Basic shapes

Points will be represented based on the line cap shape. The thickness of a point is returned by the function line-style-effective-thickness - it accounts for the line style unit and thickness.

(defmethod medium-draw-point* ((medium svg-medium) x y)
  (with-drawing-context (drawable medium :area)
    (let* ((line-style (medium-line-style medium))
           (thickness (line-style-effective-thickness line-style medium))
           (radius (/ thickness 2)))
      (case (line-style-cap-shape line-style)
        (:round
         (cl-who:with-html-output (drawable drawable)
           (:circle :cx (fmt x) :cy (fmt y) :r (fmt radius))))
        (:square
         (cl-who:with-html-output (drawable drawable)
           (:rect :x (fmt (- x radius)) :y (fmt (- y radius))
                  :width (fmt thickness) :height (fmt thickness))))
        (otherwise
         (let* ((coord-seq (list (- x radius) y x (- y radius) (+ x radius) y x (+ y radius)))
                (points (format nil "~{~a~^ ~}" (map 'list #'fmt coord-seq))))
           (cl-who:with-html-output (drawable drawable)
             (:polygon :points points))))))))

Drawing lines and rectangles is trivial:

(defmethod medium-draw-line* ((medium svg-medium) x1 y1 x2 y2)
  (with-drawing-context (drawable medium :path)
    (cl-who:with-html-output (stream drawable)
      (:line :x1 (fmt x1) :y1 (fmt y1) :x2 (fmt x2) :y2 (fmt y2)))))

(defmethod medium-draw-rectangle* ((medium svg-medium) x1 y1 x2 y2 filled)
  (with-drawing-context (drawable medium (if filled :area :path))
    (cl-who:with-html-output (stream drawable)
      (:rect :x (fmt x1) :y (fmt y1) :width (fmt (- x2 x1)) :height (fmt (- y2 y1))))))

To draw a bezigon we need to construct a path.

(defmethod medium-draw-bezigon* ((medium svg-medium) coord-seq filled)
  (with-drawing-context (drawable medium (if filled :area :path))
    (let* ((coord-seq (coerce coord-seq 'list))
           (points (with-output-to-string (str)
                     (destructuring-bind (x0 y0 . coords) coord-seq
                       (format str "M ~f ~f " x0 y0)
                       (loop for (x1 y1 x2 y2 x3 y3) on coords by (lambda (lst) (nthcdr 6 lst))
                             do (format str "C ~f ~f, ~f ~f, ~f ~f " x1 y1 x2 y2 x3 y3))))))
      (cl-who:with-html-output (drawable)
        (:path :d points)))))

Drawing ellipses is always a pain in CLIM. Part of the complexity comes from the fact that CLIM allows drawing only a slice of an ellipse. If it weren't for that complexity, we could have used the built-in element. There are other gotchas:

  • ellipse is specified as a paralleogram: the issue is alleviated with the function ellipse-normalized-representation*

  • angles are CCW in screen coordinates starting at [1,0]: the start point is correct so we only need to transform the direction

Here is how we draw an ellipse:

(defvar +angle-transformation+ (make-reflection-transformation* 0 0 1 0))

(defun draw-sliced-ellipse (medium cx cy rx ry trans filled eta1 eta2)
  ;; IMPORTANT compare angles /before/ the transformation.
  (let ((lf (if (< 0 (- eta2 eta1) pi) 0 1))
        (sf 0))
    (climi::with-transformed-angles (+angle-transformation+ nil eta1 eta2)
      (multiple-value-bind (x1 y1)
          (climi::ellipse-point eta1 cx cy rx ry 0)
        (multiple-value-bind (x2 y2)
            (climi::ellipse-point eta2 cx cy rx ry 0)
          (let ((points (format nil "M ~f ~f L ~f ~f A ~f ~f 0 ~a ~a ~f ~f L ~f ~f"
                                cx cy   x1 y1   rx ry   lf sf x2 y2   cx cy)))
            (with-drawing-context (drawable medium (if filled :area :path))
              (cl-who:with-html-output (drawable)
                (:path :d points :transform trans)))))))))

(defun draw-simple-ellipse (medium cx cy rx ry trans filled)
  (with-drawing-context (drawable medium (if filled :area :path))
    (cl-who:with-html-output (drawable)
      (:ellipse :cx (fmt cx) :cy (fmt cy) :rx (fmt rx) :ry (fmt ry) :transform trans))))

(defmethod medium-draw-ellipse* ((medium svg-medium) cx cy rdx1 rdy1 rdx2 rdy2 eta1 eta2 filled)
  (multiple-value-bind (rx ry rotation)
      (climi::ellipse-normalized-representation* rdx1 rdy1 rdx2 rdy2)
    (let  ((trans (format nil "rotate(~a ~a ~a)" (fmt (/ (* rotation 360) (* 2 pi))) (fmt cx) (fmt cy))))
      (if (< (- eta2 eta1) (* 2 pi))
          (draw-sliced-ellipse medium cx cy rx ry trans filled eta1 eta2)
          (draw-simple-ellipse medium cx cy rx ry trans filled)))))

And a smoke test:

(defun test-ellipse (stream)
  (draw-ellipse* stream 200 180 100 0 0 50 :ink +red+)
  (draw-ellipse* stream 440 180 100 0 0 50
                 :ink +blue+ :start-angle (/ pi 6) :end-angle (* 7/4 pi))
  (draw-ellipse* stream 440 180 100 0 0 50
                 :ink +white+ :filled nil :start-angle (/ pi 6) :end-angle (* 7/4 pi)))

(with-output-to-drawing-stream (stream :svg "/tmp/smoke-ellipse.svg" :preview t)
  (test-ellipse stream))

img

Finally the function to draw text. The text style will be configured as part of a drawing context. Arguments transform-glyphs, toward-x and toward-y will be ignored because their specification is ambiguous. Most code comes from the alignment setting2.

(defmethod medium-draw-text* ((medium svg-medium) string x y start end
                              align-x align-y toward-x toward-y
                              transform-glyphs)
  (declare (ignore toward-x toward-y transform-glyphs))
  (with-drawing-context (drawable medium :text)
    (let ((text-anchor
            (ecase align-x
              (:left "start")
              (:center "middle")
              (:right "end")))
          (dominant-baseline
            (ecase align-y
              (:top "text-before-edge")
              (:center "central")
              (:baseline "alphabetic")
              (:bottom "text-after-edge"))))
      (cl-who:with-html-output (stream drawable)
        (:text :x (fmt x)
               :y (fmt y)
               :text-anchor text-anchor
               :dominant-baseline dominant-baseline
               (cl-who:fmt (subseq string start (or end (length string)))))))))

That covers all basic shapes provided by McCLIM. Let's draw the "smiley" again!

(with-output-to-drawing-stream (stream :svg "/tmp/second-drawing.svg" :preview t)
  (with-rotation (stream (/ pi 6))
    (test-drawing stream)))

img

This looks fairly the same! The difference is in size - the result is around 5kB and is 80x smaller than the previous drawing. Not to mention that polygonalizing everything is disgusting a realm of 3d rendering.

The line style

The standard line style has five attributes:

  1. line-style-unit is a unit for measuring thickness and dash length.
    • :coordinate is a device unit (pixel). Lengths are a subject of transformations - the line thickness is scaled along with graphics

    • :point is a physical unit (1/72in). Lengths are not a subject of transformations - the line thickness is independent from transformation

    • :normal is a unit that depends on the renderer. Like a physical unit it is not a subject of scaling. This is a default value and should adapt the underlying device idiosyncrasies - for clx this will be a pixel (that doesn't scale) while the text backend will try to use glyphs that imitate a line (for a reasonably small thickness). As a rule of thumb the line of thickness 1 should match the stroke thickness of text.

  2. line-style-thickness is (hah!) a thickness of the line in units specified by the line-style-unit. The length is pixels may be easily computed by calling line-style-effective-thickness. The default method assumes that the effective thickness may be a float and that the unit :normal is commensurate with a pixel3.

    ;; Example method assuming that the underlying engine accepts only integers
    (defmethod line-style-effective-thickness
        (line-style (medium fixated-medium))
      (floor (+ .5 (call-next-method))))
    
  3. line-style-dashes defaults to nil (no dashes). It may be also specified as t, then dashes depend on the rendering engine.

    When line-style-dashes is a sequence then it must have an even number of elements where consecutive pairs are "fill" and "gap" values specified in units specified by the line-style-unit. Like with the line thickness McCLIM provides a function line-style-effective-dashes. Tentatively it is permissible for the renderer to not fully support line dashes.

  4. line-cap-shape specifies the shape for the ends of lines. Possible values are :butt, :square, :round or :no-end-point. It is permissible for the renderer to not fully support all cap shapes.

  5. line-joint-shape specifies the shapes of joints between segments of unfilled figures. The possible values are :miter, :bevel, :round and :none (the default is :miter). It is permissible for the renderer to not fully support all cap shapes.

    McCLIM specifies a generic function medium-miter-limit. When the line-style-joint-shape is :miter and the angle between two joined lines is less than the limit, then :bevel style is used.

As for the SVG renderer - most attributes map easily to the SVG element attributes and the implementation is very straightforward.

The only exception is the miter limit that must be casted between representations - McCLIM specifies the angle between lines while SVG specifies a ratio of the miter length to the stroke width.

Without further ado:

(defun svg-miter-limit (miter-limit-as-angle)
  ;; svg-miter-limit = miter-length / stroke-width = 1 / sin(theta/2)
  (/ 1 (sin (/ miter-limit-as-angle 2))))

(defun invoke-with-drawing-context (cont medium mode)
  ...
  (labels (...
           (configure-path ()
             (let* ((line-style (medium-line-style medium))
                    (thickness (fmt (line-style-effective-thickness line-style medium)))
                    (dashes (map 'list #'fmt (line-style-effective-dashes line-style medium)))
                    (cap-shape (ecase (line-style-cap-shape line-style)
                                      ((:butt :no-end-point) "butt")
                                      (:square "square")
                                      (:round "round")))
                    (joint-shape (ecase (line-style-joint-shape line-style)
                                        ((:bevel :none) "bevel")
                                        (:miter "miter")
                                        (:round "round")))
                    (miter-limit (svg-miter-limit (medium-miter-limit medium))))
               (cl-who:htm
                 (:g :stroke-width thickness
                     :stroke-dasharray (format nil "~{~a~^ ~}" dashes)
                     :stroke-linecap cap-shape
                     :stroke-linejoin joint-shape
                     :miter-limit miter-limit
                     (funcall cont drawable))))
             ...))
    ...))

For example:

(defun test-line-style (stream)
  (labels ((draw-lines (&rest options)
             (apply #'draw-line* stream 10 10 90 10 options)
             (apply #'draw-line* stream 10 20 90 20 :line-thickness 4 options)
             (apply #'draw-line* stream 10 30 90 30 :line-thickness 4 :line-dashes t options)
             (apply #'draw-line* stream 10 40 90 40 :line-thickness 4 :line-dashes #(8 8 4 4) options)
             (apply #'draw-polygon* stream  '(10 75 30 50 50 100 100 75)
                    :line-thickness 2 :filled nil options)))
    (with-translation (stream 0 0)
      (draw-lines))
    (with-translation (stream 0 100)
      (draw-lines :line-style (make-line-style :joint-shape :round :cap-shape :round)))
    (with-translation (stream 100 0)
      (with-scaling (stream 2 2)
        (draw-lines :line-style (make-line-style :unit :coordinate :joint-shape :round))))
    (with-translation (stream 300 0)
      (with-scaling (stream 2 2)
        (draw-lines :line-unit :normal)))))

(with-output-to-drawing-stream (stream :svg "/tmp/line-styles.svg" :preview t)
  (test-line-style stream))

img

The text style

The text style in CLIM is specified by three components:

  1. text-style-family specifies the family of the text style. Families that must be recognized by the backend are :serif, :sans-serif and :fix.

  2. text-style-face specifies the face of the text style. Faces that must be recognized by the backend are :roman, :bold and :italic. It is possible that the text face will be also a list (:bold :italic) or (:italic :bold).

  3. text-style-size is either specified in printer points (1/72 inch) or as a logical size (:normal, :tiny, :very-small, :small, :large, :very-large, :huge) and a relative size (:smaller or :larger) - relative sizes are merged with the *default-text-style.

McCLIM extends the legal values of family and face to include strings (in additional to portable keyword symbols). Each backend defines its specific syntax for these families and faces - using such text style is not portable across backends. When the backend can't parse the text style then it should fall back to *undefined-text-style*.

(make-text-style "fantasy" "oblique" :normal)

The function parse-text-style* takes the medium, the text style and returns a normalized text style:

  • device text styles are returned as is (backend-specific text styles)
  • unspecified components are filled from the *default-text-style*
  • the size is normalized to a number (specified in printer points)
  • when values are invalid then *undefined-text-style* is returned

SVG is more elaborate with font styles and we won't support all options, however To illustrate a possible direction for extensions we will do minor parsing of a font face. Non-portable font family will be passed verbatim.

(defun svg-parse-text-style-family (text-style-family)
  (case text-style-family
    (:serif "serif")
    (:sans-serif "sans-serif")
    (:fix "monospace")
    (otherwise text-style-family)))

;;; Returns values: font-style, font-weight and font-variant.
(defun svg-parse-text-style-face (text-style-face)
  (etypecase text-style-face
    (symbol
     (ecase text-style-face
       (:italic (values "italic" "normal" "normal"))
       (:bold   (values "normal" "bold"   "normal"))
       (:roman (values "normal" "normal" "normal"))))
    (list
     (values (if (member :italic text-style-face) "italic" "normal")
             (if (member :bold   text-style-face) "bold"   "normal")
             "normal"))
    (string
     (handler-case (let* ((s1 (position #\- text-style-face))
                          (s2 (position #\- text-style-face :start (1+ s1)))
                          (style   (subseq text-style-face 0 s1))
                          (weight  (subseq text-style-face (1+ s1) s2))
                          (variant (subseq text-style-face (1+ s2))))
                     (values style weight variant))
       (error ()
         (svg-parse-text-style-face
          (text-style-face *undefined-text-style*)))))))

(defun invoke-with-drawing-context (cont medium mode)
  ...
  (labels (...
           (configure-text ()
             (multiple-value-bind (family face size)
                 (text-style-components (parse-text-style* (medium-text-style medium)))
               (let ((font-family (svg-parse-text-style-family family)))
                 (multiple-value-bind (font-style font-weight font-variant)
                     (svg-parse-text-style-face face)
                   (cl-who:htm
                     (:g :font-family font-family
                         :font-style font-style
                         :font-weight font-weight
                         :font-variant font-variant
                         :font-size size
                         (funcall cont drawable))))))))
    ...))

For example:

(defun test-text-style (stream)
  (draw-rectangle* stream 0 0 300 300 :ink +light-cyan+)
  (loop for dy from 50 by 50
        for family in '(:serif :sans-serif :fix "fantasy")
        for style  in '(:roman :bold (:bold :italic) :italic)
        for size in   '(:normal :small :large :smaller)
        for ts = (make-text-style family style size) do
          (draw-text* stream "Hello World" 20 dy :align-y :top :text-style ts :ink +dark-red+)))

(with-output-to-drawing-stream (stream :svg "/tmp/smoke-text.svg" :preview t)
  (test-text-style stream))

img

Arbitrary clip regions

We've already implemented clipping the output to a solid degree. We rely on the function draw-design that is expected to draw the path for us. That does not work for region intersections and complements because they can't be represented as simple regions.

Conceptually there are a few possible approaches to the intersection. For example the most intuitive one, we could clip the clipping path:

<clipPath id='clip1' clip-path='none')...</clipPath>
<clipPath id='clip2' clip-path='url(#clip1)'>...</clipPath>
<clipPath id='clip3' clip-path='url(#clip2)'>...</clipPath>

But this approach doesn't work on "modern" web browsers. Alternatively we could use the feComposite filter using patterns loaded with the filter feImage:

<pattern id='clip1' ...>...</pattern>
<pattern id='clip2' ...>...</pattern>
<filter id='intersection' x=0 y=0 width=100% height=100% filterUnits='userSpaceOnUse'>
  <feImage xlink:href="#clip1" result="stencil1" />
  <feImage xlink:href="#clip2" result="stencil2" />
  <feComposite in="SourceGraphic" in2="stencil1" operator="in" />
  <feComposite
      in2="stencil2" operator="in" />
</filter>
<mask id='mask' x=0 y=0 width=100% height=100% maskUnits='userSpaceOnUse'
      fill='white' filter='url(#intersection)'>
  <rect x=0 y=0 width='100%' height='100%' />
</mask>

Needless to say this works even worse across SVG renderers. We need to use masks. We'll start with the region complement as the easier one. While the region complement in theory is infinite, in practice we are always working on a bounded region:

  • fill the drawing plane with "white"
  • fill the complementary shape with "black"

Code:

(defmethod medium-clip ((medium svg-medium) (region standard-region-complement))
  (let ((id (ensure-resource-id (medium (cons :mask region))
              (cl-who:with-html-output (drawable (medium-drawable medium) :indent t)
                (:defs nil nil
                  (let ((pattern-id
                          (ensure-resource-id (medium (cons :mask-pattern region))
                            (cl-who:htm (:pattern :id ^resource-id :|patternUnits| "userSpaceOnUse"
                                         :x 0 :y 0 :width "100%" :height "100%")
                                        (let ((*fgcolor* "white"))
                                          (draw-design medium +everywhere+))
                                        (let ((*fgcolor* "black"))
                                          (draw-design medium (region-complement region)))
                                        ))))
                    (cl-who:htm
                     (:mask :id ^resource-id :|maskUnits| "userSpaceOnUse"
                      :x 0 :y 0 :width "100%" :height "100%"
                      (:rect :x 0 :y 0 :width "100%" :height "100%" :fill (url pattern-id))))))))))
    (cons :mask (url id))))

The intersection is implemented as follows:

  • for each region being part of the intersection construct a mask from the shape's pattern
  • regions other than the first one are masked by the previous mask

For example:

  1. draw mask-1 as a rectangle with a fill pattern-1
  2. draw mask-2 as a rectangle with a fill pattern-2 and masked with mask-1
  3. draw mask-3 as a rectangle with a fill pattern~3 and masked with mask-2

Code

(defmethod medium-clip ((medium svg-medium) (clip standard-region-intersection))
  (let ((mask-id "none"))
    (cl-who:with-html-output (drawable (medium-drawable medium) :indent t)
      (:defs nil nil
        (labels ((add-pattern (region)
                   (ensure-resource-id (medium (cons :mask-pattern region))
                     (cl-who:htm (:pattern :id ^resource-id :|patternUnits| "userSpaceOnUse"
                                  :x 0 :y 0 :width "100%" :height "100%"
                                  (etypecase region
                                    (standard-region-intersection
                                     (error "BUG: not canonical form!"))
                                    (standard-region-complement
                                     (cl-who:htm (:g :fill "white" (draw-design medium +everywhere+)))
                                     (cl-who:htm (:g :fill "black" (draw-design medium (region-complement region)))))
                                    (bounding-rectangle
                                     (cl-who:htm (:g :fill "white" (draw-design medium region)))))))))
                 (add-mask (pattern-id mask-id)
                   (ensure-resource-id (medium (cons :mask pattern-id))
                     (cl-who:htm
                      (:mask :id ^resource-id :|maskUnits| "userSpaceOnUse"
                       :x 0 :y 0 :width "100%" :height "100%"
                       (:rect :x 0 :y 0 :width "100%" :height "100%"
                              :fill pattern-id :mask mask-id))))))
          (loop for region in (region-set-regions clip)
                for pattern-id = (add-pattern region)
                do (setf mask-id (url (add-mask (url pattern-id) mask-id)))))))
    (values :mask mask-id)))

This covers pretty much all possible clipping regions possible with McCLIM. Most renderers I've tried are able to handle such clip paths.

(defun test-clipping (stream show-hints-p)
  (let* ((regions (list (make-rectangle* 25 25 175 175)
                        (make-rectangle* 150 25 300 175)
                        (make-rectangle* 25 150 175 300)
                        (make-ellipse* 225 225 90 0 0 90)))
         (inter (reduce #'region-intersection regions :initial-value +everywhere+))
         (negative (make-ellipse* 162.5 162.5 8 0 0 8))
         (clipping (region-difference inter negative))
         )
    ;; Show the clipping region.
    (when show-hints-p
      (loop for region in regions
            for base-ink in (list +cyan+ +cyan+ +cyan+   +orange+)
            do (draw-design stream region :ink (compose-in base-ink (make-opacity 0.1))))
      (loop for region in regions
            do (draw-design stream region :filled nil :line-dashes t)))
    ;; ;; Draw the thing.
    (with-drawing-options (stream :clipping-region clipping)
      (draw-rectangle* stream 5 5 345 345 :filled t :ink +dark-green+))
    ;; Draw the negative region outline.
    (when show-hints-p
      (draw-design stream negative :ink +dark-red+ :filled nil :line-dashes '(4 4) :line-thickness 2))))

(with-output-to-drawing-stream (stream :svg "/tmp/advanced-clipping.svg" :preview t)
  (test-clipping stream t))

img

The last 1% of drawing

True to the ninety-ninety-ninety rule we have a few features that need to be implemented for the backend to be "100%" complete. Don't expect all backends to implement every feature mentioned in this section. SVG does not have a proper support for composition so "general designs" may be only partially implemented.

For example is the medium-clear-area - it is expected to perform a "source" composition while we are forced to rely on the default method that does an "over" composition with a solid fill. Another one is a flipping ink - it requires an access to the background image and to perform a boole xor operation on the background and the ink colors.

;;; Clearing the area is a "source"-composition - not the same as drawing the
;;; rectangle which is an "over"-composition. Given that SVG 1.1 doesn't seem to
;;; specify such compsition let's take that the default method is "good enough".
;;; -- jd 2022-03-17
#+ (or)
(defmethod medium-clear-area ((medium svg-medium) x1 y1 x2 y2)
  (multiple-value-bind (fill opacity)
      (medium-design-ink medium (medium-background medium))
    (with-drawing-context (drawable medium)
      (cl-who:with-html-output (stream drawable)
        (:rect :x x1 :y y1 :width (- x2 x1) :height (- y2 y1)
               :fill fill :opacity opacity :stroke *nocolor* :comp-op "src")))))

;;; Flipping ink requires access to the underlying picture to mix with the
;;; background. So-called "modern" browsers apparently can't implement the
;;; "BackgroundImage" input source. Imaginary implementation could look
;;; something like this. This is not entirely true - the "xor" filter in
;;; feComposite seems to work only on the alpha channel.. -- jd 2022-03-24
#+ (or)
(defmethod medium-design-ink ((medium svg-medium) (design climi::standard-flipping-ink))
  (let* ((filter-id
           (ensure-resource-id (medium (cons :filter design))
             (let* ((ink1 (slot-value design 'climi::design1))
                    ;; (ink2 (slot-value design 'climi::design2))
                    ;; (logxor (format nil "#~x" (logxor (rgb-as-hex ink1) (rgb-as-hex ink2))))
                    (logxor (uniform-design-values ink1)))
               (cl-who:with-html-output (drawable (medium-drawable medium))
                 (:defs nil nil
                   (:filter :id ^resource-id :|filterUnits| "userSpaceOnUse" :x 0 :y 0 :width "100%" :height "100%"
                            (:|feFlood| :flood-color logxor :result "xor-source")
                            (:|feComposite| :in "xor-source" :operation "xor" :in2 "BackgroundImage")))))))
         (pattern-id
           (ensure-resource-id (medium (cons :pattern design))
             (cl-who:with-html-output (drawable (medium-drawable medium))
               (:defs nil nil
                 (:pattern :id ^resource-id :x 0 :y 0 :width "100%" :height "100%"
                           :|patternUnits| "userSpaceOnUse"
                           (:rect :width "100%" :height "100%" :filter (url filter-id))))))))
    (values (url pattern-id) *opacity*)))

Without inquiring too much general designs may be characterized as:

  • bounded or unbounded
  • uniform or non-uniform
  • solid or translucent
  • colorless or colored

Designs we have worked with until now were:

design bounded uniform solid colorless
region both no yes yes
color no yes yes no
opacity no yes no yes
uniform-compositum no yes no no

An indirect ink is a special design that is a "trampoline" to another one - it doesn't have any predefined characteristics.

Other designs that weren't mentioned yet are:

  • composite designs: in-compositum, out-compositum, over-compositum
  • pattern: an indexed array with other designs as inks
  • stencil: an array with opacities
  • tiled design: a design that repeats itself
  • transformed design: a design with a transformation
  • output record design: a vector graphics design

The concept of design is an unification of a shape, a color and an opacity. Depending on the function call the design may be used as clipping region, a paint or a mask. For example in the call below ~ is used only for the opacity value, the "blue part" is ignored.

(draw-design <rectangle> :ink (compose-in +red+ <translucent-blue>))

Images, tiles and transformed patterns

It is said that cat pictures make about 90% of the internet, the rest is the content generated by AI. Time to draw a raster image. Here is a function that encodes images in a form that may be embedded in SVG:

(defun encode-svg-pattern (pattern &optional (format :jpeg))
  (check-type format (member :png :jpeg))
  (with-output-to-string (stream)
    (format stream "data:image/~(~a~);base64," format)
    (cl-base64:usb8-array-to-base64-stream
     (flexi-streams:with-output-to-sequence (octet-stream)
       (climi::write-bitmap-file pattern octet-stream :format format))
     stream)))

Patterns may be categorized in three groups:

  • simple pattern: a rectangle with the origin at [0,0]
  • rectangular-tile: a pattern that repeats itself
  • transformed-pattern: a pattern with an associated transformation

A simple pattern is limited by its dimensions. In the code bellow the size of the pattern is 100% while the image has finite width and height. We do that because SVG patterns are rectangular tiles in CLIM terminology and by making them this big we prevent the repetition.

(defmethod medium-design-ink ((medium svg-medium) (pattern image-pattern))
  (let ((id (ensure-resource-id (medium (cons :image pattern))
              (let* ((href (encode-svg-pattern pattern)) ;; :png for alpha channel
                     (width (pattern-width pattern))
                     (height (pattern-height pattern)))
                (cl-who:with-html-output (drawable (medium-drawable medium))
                  (:defs nil nil
                    (:pattern :id ^resource-id :x 0 :y 0 :width "100%" :height "100%"
                              :|patternUnits| "userSpaceOnUse"
                              (:image :|xlink:href| href :x 0 :y 0 :width (fmt width) :height (fmt height)))))))))
    (values (url id) 1.0)))

To display the image we may either use the function draw-pattern or use the image as an ink. There is an important difference between these two methods:

  • when used as an ink, then the pattern is anchored at the viewport's origin
  • when drawn with draw-pattern, then the pattern may be transformed if it is not located at the viewport's origin

Example:

(defvar *kitten*
  (make-pattern-from-bitmap-file
   (asdf:component-pathname
    (asdf:find-component "clim-examples" '("images" "kitten.jpg")))))

(defun draw-the-internet (stream precision)
  (assert (< precision 90))
  (draw-circle* stream 550 180 150 :ink *kitten*))

(with-output-to-drawing-stream (stream :svg "/tmp/internet.svg" :preview t)
  (draw-the-internet stream 88))

img

A rectangular tile takes the source design and repeats it after specified width and height. Its implementation is very similar to the image pattern, but in this case the svg pattern size is not "100%" of the viewport but the tile size, so it repeats itself.

(defmethod medium-design-ink ((medium svg-medium) (pattern rectangular-tile))
  (let ((id (ensure-resource-id (medium (cons :design-ink pattern))
              (let* ((src-pattern (rectangular-tile-design pattern))
                     (src-id (medium-design-ink medium src-pattern))
                     (src-width (fmt (pattern-width src-pattern)))
                     (src-height (fmt (pattern-height src-pattern)))
                     ;;
                     (tile-width (fmt (pattern-width pattern)))
                     (tile-height (fmt (pattern-height pattern))))
                (cl-who:with-html-output (drawable (medium-drawable medium))
                  (:defs nil nil
                    (:pattern :id ^resource-id :x 0 :y 0 :width tile-width :height tile-height
                              :|patternUnits| "userSpaceOnUse"
                              (:rect :x 0 :y 0 :width src-width :height src-height :fill src-id))))))))
    (values (url id) 1.0)))

For example:

(defvar *checkers-array*
  (let ((array (make-array '(100 100) :element-type 'fixnum :initial-element 0)))
    (loop for row from 0 below 50 do
      (loop for col from 0 below 50 do
        (setf (aref array row col) 1)
        (setf (aref array (- 99 row) (- 99 col)) 1)))
    array))

(defun make-checkers (design1 design2 width height)
  (make-rectangular-tile
   (make-pattern *checkers-array* (list design1 design2))
   width height))

(defun draw-rectangular-tile (stream)
  (let ((pattern (make-checkers +black+ +white+ 75 75)))
    (draw-circle* stream 550 180 150 :ink pattern)
    (draw-circle* stream 550 180 150 :ink +dark-blue+ :filled nil :line-thickness 4)))

(with-output-to-drawing-stream (stream :svg "/tmp/tile.svg" :preview t)
  (draw-rectangular-tile stream))

img

Designs have the same origin and orientation as the native region. It is possible to transform a design - in that case its origin and orientation may change. When the source design is bounded then the transformed design is also bounded. For tiles it is not bounded.

It is important to apply the transformation to the whole pattern instead of applying it to the source design and tile the result. Otherwise the result may be wrong when the design is rotated - tiled parts won't be connected.

(defmethod medium-design-ink ((medium svg-medium) (pattern transformed-pattern))
  (let ((id (ensure-resource-id (medium (cons :design-ink pattern))
              (let* ((transform (svg-transform (transformed-design-transformation pattern)))
                     (src-pattern (transformed-design-design pattern))
                     (src-id (medium-design-ink medium src-pattern))
                     (src-width (fmt (pattern-width src-pattern)))
                     (src-height (fmt (pattern-height src-pattern)))
                     ;;
                     (tilep (typep src-pattern 'rectangular-tile))
                     (width  (if tilep src-width  "100%"))
                     (height (if tilep src-height "100%")))
                (cl-who:with-html-output (drawable (medium-drawable medium))
                  (:defs nil nil
                    (:pattern :id ^resource-id :x 0 :y 0 :width width :height height
                              :|patternUnits| "userSpaceOnUse"
                     :|patternTransform| transform
                     (:rect :x 0 :y 0 :width src-width :height src-height :fill src-id))))))))
    (values (url id) 1.0)))

For example:

(defun draw-transformed-tile (stream)
  (let* ((pattern (transform-region
                   (make-rotation-transformation (/ pi 6))
                   (make-checkers +black+ +white+ 75 75))))
    (draw-circle* stream 550 180 150 :ink pattern)
    (draw-circle* stream 550 180 150 :ink +dark-blue+ :filled nil :line-thickness 4)))

(with-output-to-drawing-stream (stream :svg "/tmp/transformed.svg" :preview t)
  (draw-transformed-tile stream))

img

Finally a default method that should work for any other pattern:

(defmethod medium-design-ink ((medium svg-medium) (pattern pattern))
  (let ((id (ensure-resource-id (medium (cons :design-ink pattern))
              (let* ((href (encode-svg-pattern pattern :png))
                     (width (pattern-width pattern))
                     (height (pattern-height pattern)))
                (cl-who:with-html-output (drawable (medium-drawable medium))
                  (:defs nil nil
                    (:pattern :id ^resource-id :x 0 :y 0 :width "100%" :height "100%"
                              :|patternUnits| "userSpaceOnUse"
                              (:image :|xlink:href| href :x 0 :y 0 :width (fmt width) :height (fmt height)))))))))
    (values (url id) 1.0)))

Recursive designs

When the source design is "flat" then things are simple, but the source may be also a recursive pattern. For example when we tile an indexed pattern that uses non-uniform designs as inks, then it may show different design in different repetitions. The same concern applies to transformed patterns - only the pattern is transformed, not inks used as its palette. For example:

(defvar *checkers-array*
  (let ((array (make-array '(100 100) :element-type 'fixnum :initial-element 0)))
    (loop for row from 0 below 50 do
      (loop for col from 0 below 50 do
        (setf (aref array row col) 1)
        (setf (aref array (- 99 row) (- 99 col)) 1)))
    array))

(defun make-checkers (design1 design2)
  (make-rectangular-tile
   (make-pattern *checkers-array* (list design1 design2))
   100 100))

(defun draw-recursive-pattern (stream)
  (let ((pattern (make-checkers +black+ *kitten*)))
    (draw-circle* stream 550 180 150 :ink pattern)
    (draw-circle* stream 550 180 150 :ink +dark-blue+ :filled nil :line-thickness 4)))

(with-output-to-drawing-stream (stream :svg "/tmp/recursive.svg")
  (draw-recursive-pattern stream))

When implemented naively, the tile repeats the beginning of the kitten design instead of providing "holes" that show the kitten:

img

The pattern should look like this:

img

The most straightforward solution to that is by using composition and treating the indexed pattern as a stencil. Since we don't have any GPU nearby, we will analyze the pattern and decide what to do with it. When we detect a recursive pattern then we collapse it so it may be used on the whole viewport. This is not effective, because for a tile it will encode the image with the same size as the produced document.

Encoding cat pictures that way is grossly inefficient, so when we detect such situation we signal a warning.

(defun maybe-collapse-pattern (pattern)
  (let ((tile-p nil))
    (labels ((unmoveable-pattern-p (design)
               (typecase design
                 (rectangular-tile
                  (setf tile-p t)
                  (unmoveable-pattern-p (rectangular-tile-design design)))
                 (transformed-pattern
                  (unmoveable-pattern-p (transformed-design-design design)))
                 (otherwise
                  (and (typep design 'climi::indexed-pattern)
                       (some (lambda (p)
                               (not (typep p '(or color opacity climi::uniform-compositum))))
                             (climi::pattern-designs design))
                       design)))))
      (when (unmoveable-pattern-p pattern)
        ;; FIXME hardcoded viewport size.
        (if tile-p
            (climi::%collapse-pattern pattern 0 0 *viewport-w* *viewport-h*)
            (with-bounding-rectangle* (x0 y0 :width width :height height) pattern
              (transform-region (make-translation-transformation x0 y0)
                                (climi::%collapse-pattern pattern x0 y0 width height))))))))

;;; This method is very inefficient (evaluation time and memory) and very
;;; expensive (file size). We are flexing to do the right thing. Normally we'd
;;; use palette in-composition using shaders.
(defmethod medium-design-ink :around ((medium svg-medium) (pattern pattern))
  (alx:if-let ((collapsed (maybe-collapse-pattern pattern)))
    (warn "Encoding a collapsed tile as an RGBA image.")
    (let ((id (ensure-resource-id (medium (cons :design-ink pattern))
                (alx:simple-style-warning
                 "Collapsing the pattern for the viewport - this is very inefficient!")
                (let* ((rht (resources (port medium)))
                       (cid (progn (medium-design-ink medium collapsed)
                                   (resource-id medium (cons :design-ink collapsed)))))
                  (setf (gethash (cons :design-ink pattern) rht) cid)
                  (setf ^resource-id cid)))))
      (values (url id) 1.0))
    (call-next-method)))

Masked in- and out- composition

It is possible to provide a stencil when drawing the image. in-compositum and out-compositum both have an ink and a mask. Only the alpha channel is taken from the mask and applied to the ink. out-compositum has the alpha channel values inverted out(alpha) = (1- alpha). For solid masks the result is the same as if we had clipped to the region or its complement.

There two types of masks that should be supported by McCLIM:

  • uniform mask
  • stencil mask

Let's start with the former. Only uniform have methods defined on the function opacity-value:

(defmethod medium-design-ink ((medium svg-medium) (design climi::in-compositum))
  (let ((ink (climi::compositum-ink design))
        (mask (climi::compositum-mask design)))
    (alx:if-let ((opacity (ignore-errors (opacity-value mask))))
      (multiple-value-bind (ink-url ink-opacity)
          (medium-design-ink medium ink)
        (values ink-url (* ink-opacity opacity)))
      (compose-stencil medium design))))

(defmethod medium-design-ink ((medium svg-medium) (design climi::out-compositum))
  (let ((ink (climi::compositum-ink design))
        (mask (climi::compositum-mask design)))
    (alx:if-let ((opacity (ignore-errors (opacity-value mask))))
      (multiple-value-bind (ink-url ink-opacity)
          (medium-design-ink medium ink)
        (values ink-url (* ink-opacity (- 1.0 opacity))))
      (compose-stencil medium design))))

For example:

(defvar *glider*
  (make-pattern-from-bitmap-file
   (asdf:component-pathname
    (asdf:find-component "clim-examples" '("images" "glider.png")))))

(defun draw-masked-compositums (stream mask)
  (let* ((glider (make-rectangular-tile *glider* 100 100))
         (pattern-1 (compose-in glider mask))
         (pattern-2 (compose-out glider mask)))
    (draw-circle* stream 50 50 150 :ink (make-checkers +dark-green+ +dark-blue+ 75 75))
    (draw-circle* stream 50 50 50 :ink pattern-1)
    (draw-circle* stream 100 100 50 :ink pattern-2)
    (draw-circle* stream 50 50 150 :ink +dark-blue+ :filled nil :line-thickness 4)))

(with-output-to-drawing-stream (stream :svg "/tmp/in-out-uniform.svg" :preview t)
  (draw-masked-compositums stream (make-opacity 0.3)))

img

Stencil is an array of opacities. We will simply collapse the pattern. Doing it as a mask is also an option.

;;; Normally we'd use a mask, but this is yet another feature that is handled
;;; differently by every second renderer. That's why we simply flatten the ink
;;; when it is not an uniform compositum.
;;;
;;; CLIM II specification hints that handling only uniform masks is OK.  That
;;; said we still want to support stencils so let's get lazy big time.
(defun compose-stencil  (medium pattern)
  (let* ((pattern* (climi::%collapse-pattern pattern 0 0 *viewport-w* *viewport-h*)))
    (medium-design-ink medium pattern*)))

And a test

(defvar *stencil*
  (let ((array (make-array '(100 100) :element-type '(single-float 0.0 1.0) :initial-element 0.0)))
    (loop for row from 0 below 100 do
      (loop for col from 0 below 100 do
        (setf (aref array row col)
              (- 1.0
                 (/ (sqrt (+ (expt (- row 50) 2)
                             (expt (- col 50) 2)))
                    50.0)))))
    (make-stencil array)))

(defun draw-stencil (stream stencil)
  (let* ((glider (make-rectangular-tile *glider* 100 100))
         (pattern-1 (compose-in glider stencil))
         (pattern-2 (compose-out glider stencil)))
    (draw-circle* stream 50 50 150 :ink (make-checkers +dark-green+ +dark-blue+ 75 75))
    (draw-rectangle* stream 0 0 50 150 :ink pattern-1)
    (draw-rectangle* stream 50 0 100 150 :ink pattern-2)
    (draw-circle* stream 50 50 150 :ink +dark-blue+ :filled nil :line-thickness 4)))

(with-output-to-drawing-stream (stream :svg "/tmp/in-out-stencil.svg" :preview t)
  (draw-stencil stream *stencil*))

img

Features that are not implemented

Default methods defined on basic-medium are often no-op stubs or approximations (like with drawing routines that resort to using draw-polygon). Sometimes the functionality is not applicable to a drawing backend or it is unfeasible to implement it correctly. The following parts of the output protocol are not implemented:

  • src-composition: medium-{clear,copy}-area
  • xor-composition: flipping inks
  • offscreen drawing: invoke-with-output-{buffered,to-pixmap}
  • attracting the user's attention: medium-beep
  • precise font metrics: text-size and text-bounding-rectangle*
  • output record patterns: these are not implemented yet in McCLIM

We are also doing a poor job with composing things - collapsing patterns is correct but inefficient. Sometimes it may be necessary to specialize other medium or port functions, but that varies on case-to-case basis. We could also add some bonus features, like gradient patterns.

Congratulations, we've implemented the medium output protocol targeting an SVG document4!

Conclusions

Implementing the medium output protocol is relatively easy and it gets gradually harder when we exercise more niche features. In this post we've uncovered various shortcoming of the SVG format and renderers. The current ([2022-04-15 Fri]) version of this backend may be found in a feature branch.

In the next part of this tutorial the SVG port will be extended to implement the sheet output recording protocol. It will be much shorter than this one and will cover grafts and sheets.

Footnotes

1 Not all output devices operate on streams of data - sometimes "drawing a rectangle" may involve calling a function from a library. In that case the second argument to the macro could be a handler.

2 Not all SVG renderers I've tried honor the attribute dominant-baseline.

3 Currently it is 1px but I'm considering changing it to 2px so that a rectilinear line of thickness 1 with all coordinates being integers is not a subject of rounding. See the figure 12.7 in the specification.

4 It is worth noting that it is more complete (the last 1%-wise) than pdf and postscript backends.

Charles ZhangUsing Lisp libraries from other programming languages - now with sbcl

· 78 days ago

Have you ever wanted to call into your Lisp library from C? Have you ever written your nice scientific application in Lisp, only to be requested by people to rewrite it in Python, so they can use its functionality? Or, maybe you've written an RPC or pipes library to coordinate different programming languages, running things in different processes and passing messages around to simulate foreign function calls.

Calling into C from other languages is much easier in comparison. A dlopen() and a foreign function interface suffices to use library functionality exported with the C ABI, without spawning separate processes. Bindings written over such a C standard library for language X make it seem like the underlying functions really are written in X. For cross-language inter-operation, this is the most lightweight option available. Languages like Rust and C++ can use extern "C" { } blocks to wrap their own functions in the C standard ABI, to expose their functions in exactly this way for the foreign function interfaces of other programming languages.

For languages with a sizable runtime included (with, for example, automatic memory management, signal handling, etc.) such as Python or even Javascript, having a way to expose functions to be callable with the C ABI is much less common. You quickly need to deal with issues of how objects are represented internally, how to deal with type translations, and what happens when objects in memory move for GC. Some other Lisp implementations like ECL (the E stands for Embeddable Common Lisp) and some commercial Lisps do support this use-case however: that of allowing Lisp libraries to essentially be packaged up as a shared library linkable with ordinary object files where Lisp code is callable with the normal C calling convention.

Some of you may already know you can use ECL for this purpose; ECL is a great implementation, but it does lack some functionality and features that makes it unattractive for some.

If you prefer using SBCL, you can now join in on the cross-language programming frenzy too.

1.1. wait, really?

Those of you who have used foreign callbacks in libraries such as CFFI will already be somewhat familiar with how the Lisp-side interface works – you write your Lisp code normally and then define C convention callable function pointers in Lisp through a fairly comprehensive translation interface. With callbacks, you just pass these function pointers as parameters to your foreign functions. What's needed to make Lisp callable as a shared library on top of this interface is to iron out runtime management and linkage issues, e.g. how do we initialize the Lisp runtime and GC, and then how does the system linker actually find the C callable function pointers defined in Lisp so that you can call those functions normally as if they were written in C?

I'll explain how to use low-level machinery now exposed in SBCL and how it works, and wrap up by talking about a convenience library that creates a high level interface for creating bindings and manages some of the details.

1.2. example scenario: a calculator

Let's take a classic example to illustrate this new functionality: Suppose you want to write C code and call out to a library called calc. calc is written entirely in Common Lisp, full of classes and objects and methods defining ASTs for a simple symbolic calculator. It has functions like parse-expr, simplify, and pretty-print-expr, where everything works on Lisp objects.

(defclass expression ()
()
(:documentation "An abstract expression."))
(defclass int-literal (expression)
((value :reader int-literal-value))
(:documentation "An integer literal."))
(defclass sum-expression (expression)
((left-arg :reader sum-expression-left-arg)
(right-arg :reader sum-expression-right-arg)))
(defun parse (string) ...)
(defun simplify (expr) ...)
(defun expression-to-string (expr) ...)

We'd like to expose these functions so that we can call these functions from other programming languages using the C ABI. So how does it work in the simplest case of using calc from a simple C program?

First, we need to define these function pointers in Lisp. The primitives SBCL now exposes to deal with this is through the macro sb-alien:define-alien-callable. You can use this macro in the following way:

(define-alien-callable calc-parse int ((source c-string) (result (* (* t))))
(handler-case
(progn
;; The following needs to use SBCL internal functions to
;; coerce a Lisp object into a raw pointer value. This is
;; unsafe and will be fixed in the next section.
(setf (deref result) (sap-alien (int-sap (get-lisp-obj-address (parse source))) (* t)))
0)
(t (condition) (declare (ignore condition)) 1)))

(define-alien-callable calc-simplify int ((expr (* t)) (result (* (* t))))
(handler-case
(progn
;; The following needs to use SBCL internal functions to
;; coerce a raw pointer value into a Lisp object and
;; back. This is unsafe and will be fixed in the next
;; section.
(setf (deref result)
(sap-alien (int-sap (get-lisp-obj-address
(simplify (%make-lisp-obj (sap-int (alien-sap expr))))))
(* t)))
0)
(t (condition) (declare (ignore condition)) 1)))

(define-alien-callable calc-expression-to-string int ((expr (* t)) (result (* c-string)))
(handler-case
(progn
;; The following needs to use SBCL internal functions to coerce a raw
;; pointer value into a Lisp object. This is unsafe and
;; will be fixed in the next section.
(setf (deref result) (%make-lisp-obj (sap-int (alien-sap (expression-to-string result)))))
0)
(t (condition) (declare (ignore condition)) 1)))

Notice that for example purposes we just translate any exceptional Lisp conditions into the C return error code convention. The actual returned value is set through the second passing argument's out-pointer in typical C fashion.

Now we've defined C callable function pointers associated with names. You can get the underlying function pointers with the function sb-alien:alien-callable-function, which is useful for passing these callable functions as callbacks to C. SBCL actually didn't even have an external interface to callbacks before this, and CFFI had been using an internal system interface which is now superseded by this interface.

However, we're not too interested in callbacks right now. We want to be able to call these functions from C! You can save the Lisp image containing new callable function pointer definitions like so:

(sb-ext:save-lisp-and-die "calc.core"
:callable-exports '("calc_parse" "calc_simplify" "calc_expression_to_string" ...))

What the :callable-exports argument describes is the set of C symbols you want to initialize with the corresponding function pointers created with define-alien-callable. This image, when started, has only two jobs: finish starting up the Lisp system and initialize these callable exports with the proper function pointers. It then immediately passes control back to C

Here's an example C file which uses these functions:

#include "libcalc.h"
int main () {
initialize_lisp("libcalc.core");
expr_type expr;
if (calc_parse(source, &expr) != ERR_SUCCESS)
die("unable to parse expression");
char *result;
expr_type simplified_expr;
calc_simplify(expr, &simplified_expr);
if (calc_expression_to_string(simplified_expr, &result) != ERR_SUCCESS)
die("unable to print expression to string");
printf("\n%s\n", result);
return 0;
}

Notice that there is a symbol, initialize_lisp, that we can use to initialize the Lisp runtime. Its arguments are the same as the arguments you can normally pass to the main sbcl executable, so once the core name is specified and the runtime initializes the function pointers we are going to use, control is returned to the program.1

Once the Lisp runtime has been initialized, your function pointers are ready to use, and calling them from C works just like a callback would, with the same type translation machinery used.

1.3. what about those raw pointers? objects? GC???

Now comes the question of what to do with objects. There are actually several:

  1. How do we make sure that references living outside the Lisp heap keep the object alive?
  2. Solving that, how can we make sure objects in the Lisp heap don't stay alive forever and do end up getting garbage collected at some point?
  3. Even more conspicuously, how do we deal with objects getting moved by the GC?

In ECL, we don't really need to worry about this last aspect too much. Since the garbage collector in ECL uses Boehm, objects in memory never move, so you could get away with just passing the object address as a void pointer into C, and opaquely manipulating that in your non-Lisp code, as long as you have some way of telling Boehm where in your C program Lisp objects can reside.

Unfortunately, this is an unworkable scheme with SBCL. The primary garbage collector used in SBCL is generational and copying. Therefore, passing an object address to the outside world directly outside of the managed Lisp heap means that once that object moves, the pointer becomes invalid, and a segfault ensues. Luckily, the solution is not too complicated: A layer of indirection for opaque pointers is all that's needed to resolve the problem. This can be implemented in any number of ways, but arguably the simplest scheme is just to have a map associating integers (fixnums) to objects which cross the foreign function boundary. If the map exists in the Lisp heap, for example by using a global hash table stored in a global Lisp variable, then GCs simply update the map values like everything else, and because fixnums do not move, the keys to the map are therefore stable handles we can pass to the outside world. Since we usually only use object pointers opaquely, the indirection through the map can simply be handled on the Lisp side. This solution also solves problems 1) and 2) at the same time: the map lives in the Lisp heap, so it will keep any objects in its entries live. When we want to remove the reference to the object from the outside world, we simply remove the entry in the map. If that object is no longer referenced anywhere else, the garbage collector happily cleans it up.

Let's illustrate this indirection by modifying the above example to be safe and also teach the C program how to clean up memory. Assuming we have Lisp functions make-handle and dereference-handle to add this layer of indirection, we would simply change our callable definitions like so, allowing us to remove unsafe raw pointer coercions. make-handle and dereference-handle deal with Lisp fixnums coerced into opaque pointer values, so we know that they are always safe to pass around.

(define-alien-callable calc-parse int ((source c-string) (result (* (* t))))
(handler-case
(progn
(setf (deref result) (make-handle (parse source)))
0)
(t (condition) (declare (ignore condition)) 1)))

(define-alien-callable calc-simplify int ((expr (* t)) (result (* (* t))))
(handler-case
(progn
(setf (deref result) (make-handle (simplify (dereference-handle expr))))
0)
(t (condition) (declare (ignore condition)) 1)))

(define-alien-callable calc-expression-to-string int ((expr (* t)) (result (* c-string)))
(handler-case
(progn
(setf (deref result) (expression-to-string (dereference-handle result)))
0)
(t (condition) (declare (ignore condition)) 1)))

Now let's modify our C example to include this type of handling using the exported function lisp_release_handle:

#include "libcalc.h"
int main () {
initialize_lisp("libcalc.core");
expr_type expr;
if (calc_parse(source, &expr) != ERR_SUCCESS)
die("unable to parse expression");
char *result;
expr_type simplified_expr;
calc_simplify(expr, &simplified_expr);
if (calc_expression_to_string(simplified_expr, &result) != ERR_SUCCESS)
die("unable to print expression to string");
printf("\n%s\n", result);
lisp_release_handle(expr);
lisp_release_handle(simplified_expr);
return 0;
}

As you can see, we only ever need to make and dereference handles on the Lisp callable definition side, while the foreign world only needs to pass around these objects opaquely and know how to "free" their references to these objects. Any time you feel like you need to peek inside the internals of Lisp object from the foreign world, all you need to do is define an alien callable function exposing that portion of the object. That way, cross-language pointers can be kept totally opaque and safe through some indirection like through the handles illustrated above.

1.4. sharing a process

There are some more sophisticated intra-process related issues I haven't touched upon, including how the Lisp runtime interacts with the outside world in relation to signals, sessions, threading, or file descriptors such as standard input and out. In particular, users will need to be careful when using Lisp together with another managed runtime in the same process, such as a Python interpreter. However, since many of these issues are general problems anyone running multiple systems in one process has to deal with, I won't cover the myriad number of ways people use to share process stuff. However, POSIX signal handlers can all be set and removed from Lisp itself in SBCL, if the need arises. Some examples of ways Lisp implementations could use signals:

  • signals relating to handling keyboard interrupts to drop into the debugger. The debugger can of course be disabled.
  • some kind of mechanism for telling other threads a stop-the-world garbage collection is happening and they need to suspend. This can be implemented via signals, but it can be implemented by other means such as safepoints.
  • some kind of mechanism for actually starting a garbage collection, for example through a segfault on an unallocated page.

Different builds of SBCL on different operating systems require different signals or do threading in different ways, so consult the documentation for details.

1.5. wrapping it up

While the functionality that SBCL exposes is fairly low-level, a high level library called sbcl-librarian is in development (used in, for example, the Quil language stack), which defines a declarative interfaces for generating bindings for types and errors, as well as producing the corresponding Lisp callable definitions. It also defines some API exports for important Lisp-side functionality such as the handles described above, as well as exposing the Lisp loader and debugger for loading and debugging Lisp code at runtime from the foreign world. Much of the interface is still subject to design and change, but hopefully things will stabilize soon with use.

Well, so now you have another implementation on the block for embedding Lisp in larger systems. Go try it out!

Footnotes:

1

Note that behind the scenes, the symbols parse_expr, simplify_expr, etc. are all global variables that happen to hold function pointers. They are not function symbols in linker lingo, but from the perspective of C, you can call them the same as if you defined a normal function. Hence there's only one level of indirection to make the call on the C side, which is the same as the indirection to call any other function from a shared library via the normal PLT/GOT mechanism, which is pretty much optimal for cross-language shared library linkage, if you ask me.

TurtleWareTixel Viewer

· 79 days ago

I'm currently having fun with modeling motion with non-linear transformations. While doing that I've written a simple viewer for points scattered in time and decided to share it as a separate piece.

To visualise the plane over time we need to introduce a concept of the scene. The scene is a set of frames that present a time from t0 to end-time stepped by dt. As for scaling to the "observer time", a variable dt/s specifies the number of frames per second. For simplicity we will assume that t0=0.

(in-package #:clim-user)

;;; Poor man's double buffering. The proper interface is a WIP.
(defmacro with-scene ((stream) &body body)
  (check-type stream symbol)
  `(let* ((pixmap (with-output-to-pixmap (,stream ,stream)
                    ,@body))
          (width (pixmap-width pixmap))
          (height (pixmap-height pixmap)))
     (medium-copy-area pixmap 0 0 width height ,stream 0 0)
     (deallocate-pixmap pixmap)
     (finish-output ,stream)))

(defun draw-frame (stream jiff dt points)
  (draw-rectangle* stream -50 -50 50 50 :ink +grey90+)
  (draw-rectangle* stream -50 -50 50 50 :filled nil)
  (loop with thickness = 10
        for (x y time) in points
        when (= time jiff) do
          (draw-point* stream x y :ink +dark-red+ :line-thickness thickness)))

(defparameter *dt/s* 1) ;; fps-esque

(defun project-scene (stream end-time dt points)
  (loop for jiff from 0 upto end-time by dt do
    (with-scene (stream)
      (let ((blurb (format nil "Jiff = ~6,2f / ~6,2f" jiff end-time)))
        (draw-text* stream blurb 10 250
                    :align-x :left :align-y :bottom :text-family :fix))
      (with-translation (stream 60 60)
        (draw-frame stream jiff dt points)))
    (sleep (/ 1 *dt/s*))))

Let's display the scene in a loop and allow to recompile functions and points at runtime - because that's what Lisp programmers do.

;;; The scene time and dt are constant in one scene, new values will be used
;;; when the scene is replayed. We may freely manipulate the speed though.
(defparameter *scene-time* 30)
(defparameter *dt* 10)

(defparameter *points*
  (loop for time from 0 below 40 by 10
        for +dist from 10 by 10
        for -dist = (- +dist)
        collect (list 0 0 time)
        collect (list +dist +dist time)
        collect (list +dist -dist time)
        collect (list -dist -dist time)
        collect (list -dist +dist time)))

(defun start-scene ()
  (with-output-to-drawing-stream
      (stream nil nil :scroll-bars nil :borders nil :width 240 :height 260)
    (sleep .1) ;; don't ask (.. some vms delay the window creation, i.e kwin)
    (handler-case
        (loop while (sheet-grafted-p stream)
              do (project-scene stream *scene-time* *dt* *points*))
      (error (c)
        (princ c *debug-io*)))
    (close stream)))

Can we really draw a point on a plane though? Are pixels little squares1? In the code above we take a crude approach of treating the time as if it were discrete. If we change the time resolution (or a point position), then the scene will start to blink or even worse, some points will be missed by the scanline:

  • dt = 10 : all points appear as expected
  • dt = 10/4: all points appear but only for a little while (blinking)
  • dt = 20 : um, we might have dropped something
  • dt = 0.1 : good luck finding your points (remember to set dt/s 100)

When we step by dt=10 we implicitly make the point thickness in time to be 10. It is time (ha!) to make the point thickness explicit and independent from the value of dt. Instead of checking (= jiff time) we'll see whether the tixel's is "inside" the point or not. This is the very same approach as suggested in the Rendering Conventions for Geometric Shapes (CLIM).

But what is the tixel? Let's come up with a definition:

A tixel (as in "time pixel") is a little cube.. No, I'm joking, see the first footnote. Let's try again.

A tixel (as in "time pixel") is a point sample. An animation is a three-dimensional array of point samples (tixels).

A toxel is a "time voxel" and is also a point sample. A theater is a four-dimensional array of point samples (toxels).

A scene is a (n+1)-dimensional array of point samples where one of dimensions represents the time. For n=2 it is an animation and for n=3 it is a theater.

The thickness on the T axis is the "time span" of the point. It represents how long it remains on the drawing plane. For example when the point thickness is 3 then it will remain on the screen for three time units. When the thickness is smaller than dt, then some points may "fall of the grid".

(defun inside-p (jiff dt time radius)
  ;; (= time jiff)                   ; old definition (wrong)
  ;; (<= (abs (- time jiff)) radius) ; intuitive(?), also wrong. 
  (let ((tixel-center (+ jiff (/ dt 2))))
    (and (<= (- time radius) tixel-center)
         (< tixel-center (+ time radius)))))

(defun draw-frame (stream jiff dt points)
  (draw-rectangle* stream -51 -51 51 51)
  (draw-rectangle* stream -50 -50 50 50 :ink +grey90+)
  (loop with thickness = 10
        with radius = (/ thickness 2)
        for (x y time) in points
        when (inside-p jiff dt time radius)
          do (draw-point* stream x y :ink +dark-red+ :line-thickness thickness)))

This definition is discrete and has the advantage that points that do not overlap won't appear on the drawing plane at the same jiff.

Now let's "rotate" the frame to see the time scanline in the vertical and the horizontal slices of points on the plane. To have more fun, remember to do it while the scene is running! Let's change the time to start at -50 and end at +50, so it is consistent with x/y resolutions.

(defun draw-horizontal-slice (stream jiff dt points)
  (draw-rectangle* stream -51 -51 51 51)
  (draw-rectangle* stream -50 -50 50 50 :ink +light-cyan+)
  (with-drawing-options (stream :clipping-region (make-rectangle* -50 -50 50 50))
    (draw-rectangle* stream -50 jiff  50 (+ jiff dt) :ink +green+)
    (loop with thickness = 10
          for (x y time) in points do
            (draw-point* stream x time :ink +dark-red+ :line-thickness thickness))))

(defun draw-vertical-slice (stream jiff dt points)
  (draw-rectangle* stream -51 -51 51 51)
  (draw-rectangle* stream -50 -50 50 50 :ink +light-yellow+)
  (with-drawing-options (stream :clipping-region (make-rectangle* -50 -50 50 50))
    (draw-rectangle* stream jiff -50 (+ jiff dt) 50 :ink +green+)
    (loop with thickness = 10
          for (x y time) in points do
            (draw-point* stream time y :ink +dark-red+ :line-thickness thickness))))

(defparameter *dt/s* 10)

(defun project-scene (stream end-time dt points)
  (loop for jiff from -50 upto end-time by dt do
    (handler-case
        (with-scene (stream)
          (let ((blurb (format nil "Jiff = ~6,2f / ~6,2f" jiff end-time)))
            (draw-text* stream blurb 10 250
                        :align-x :left :align-y :bottom :text-family :fix))
          (with-translation (stream 60 60)
            (draw-frame stream jiff dt points))
          (with-translation (stream 170 60)
            (draw-vertical-slice stream jiff dt points))
          (with-translation (stream 60 170)
            (draw-horizontal-slice stream jiff dt points)))
      (error (c)
        (format *debug-io* "~a. (yes, ignoring)." c)))
    (sleep (/ 1 *dt/s*))))

(defparameter *scene-time* 50)
(defparameter *dt* 10)

The green line represents the current time slice [jiff, jiff+dt]. When the center of the slice is "inside" of the point, then that point is visible.

Points are nice, but how about projecting a line? We can't go wrong with the Bresenham's line algorithm. For sake of simplicity let's assume that tixels are cubes of size [10dx,10dy,10dt] and draw a line on the plane XT by collecting appropriate points in the center of corresponding tixels.

(defun collect-line-points (x0 t0 x1 t1)
  (let* ((dx (abs (- x1 x0)))
         (dt (- (abs (- t1 t0))))
         (sx (if (< x0 x1) 1 -1))
         (st (if (< t0 t1) 1 -1))
         (err (+ dx dt))
         (er2 nil)
         (coord-seq nil))
    (loop
      (setf er2 (* 2 err))
      (push (list (+ x0 .5) .5 (+ t0 .5)) coord-seq)
      (when (>= er2 dt)
        (when (= x0 x1)
          (return))
        (incf err dt)
        (incf x0 sx))
      (when (<= er2 dx)
        (when (= t0 t1)
          (return))
        (incf err dx)
        (incf t0 st)))
    coord-seq))

(defparameter *points*
  (collect-line-points -25 -40 +25 40))

The effect is as if a point is moving. This was a short exhibition, but we could go even further:

  • add antialiasing in time (this would be visible as a "blur" effect)
  • add a perspective projection matrix as the fourth slice preview
  • show other interesting curves in time (i.e bezier curve for easying)
  • instead of rasterizing the line we could use q signed distance field
  • points could be small spheres instead of being small cubes

But all that is displaying a static scene in time. All points are predefined and the scene is simply "replayed". This is not the motion I want to see, so I'll stop at this little demo. That said it is a useful construct to think about static scenes.

Footnotes

1 Hint: it isn't.

Nicolas HafnerFeeding Backers - April Kandria Update

· 81 days ago
https://filebox.tymoon.eu//file/TWpRM01RPT0=

The steam demo's now over, and we've started polishing up the next demo. A lot left to do, but we've solidly entered the overall polishing phase of the production. As you might know, the last 10% takes 90% of the time, or something like that anyway! We also have some more news about the Kickstarter...

Kickstarter

We now have a definitive date for the Kickstarter and the new demo: June 13th! The Kickstarter will launch on the same day as the Steam Next Fest, which will offer the new demo to play.

The new demo will feature new quests, areas, challenges, characters, music, and secrets to uncover! Make sure to subscribe to the Kickstarter to get notified when it goes live!

https://filebox.tymoon.eu//file/TWpRM01BPT0=

Steam Demo Feedback

The Steam demo is now no longer available, but quite a few people checked it out and sent us feedback about their experience. Thank you very much! A lot of that feedback, such as allowing you to bind multiple buttons to the same action, has now been addressed

https://filebox.tymoon.eu//file/TWpRMk53PT0=

We've also taken some of the broader feedback into account. A few told us that they'd gotten lost, so we added a little marker that points you in the direction of the next objective when a new task starts.

https://filebox.tymoon.eu//file/TWpRMk9RPT0=

The marker is only on screen for a brief while, to avoid it being annoying. The map has also been improved a lot to help you out along the way. Aside from the movement trail that was already in, you can now also place pins on the map, and it'll show the associated quest when you hover over a target hint.

Development

We've added support for mixed font text display, which paves the road for localisation to languages such as Japanese.

https://filebox.tymoon.eu//file/TWpRMk9BPT0=

Thanks to some local playtesting for the new demo, we've also implemented a bunch of small improvements to the dialogue, platforming, and UI. The next demo's already looking pretty good, and we should be done with it well in time for the Kickstarter.

Finally, we've begun detailing things out in the lower two regions. It'll be a while before testing for that can properly begin, but we're well on schedule for everything so far!

The Bottom Line

Okey, let's look at the roadmap from march.

  • Incorporate a battle music track

  • Implement better map navigation

  • Prepare the Kickstarter data

  • Complete the new demo

  • Fix various UI problems

  • Improve usability features

  • Implement new enemy types

  • Create a new trailer and Kickstarter video

  • Finish designing the remaining main story NPCs

  • Add detail to region 2 and 3

  • Do lots of user testing on the full game content

  • Add more side quests and areas

  • Implement Steam achievements

Not a ton has moved here, since a lot of time was spent on polishing things up from the feedback we got from the demo, and completing very minor things all over the place. Generally as we near the end of the production, the roadmap is going to move more slowly, as most of the time will be spent on many very small items.

In the meantime, we could really use your help with spreading the word about Kandria's Kickstarter! If you know any friends or communities, tell them about the Kickstarter!

Quicklisp newsMarch 2022 Quicklisp dist update now available

· 86 days ago

 New projects

  • asn1 — ASN.1 encoder/decoder — BSD 2-Clause
  • auto-restart — automatically generate restart-cases for the most common use cases, and also use the restart for automatic retries — Apache License, Version 2.0
  • cl-advice — Portable advice for Common Lisp — LGPL
  • cl-gltf — A library to parse the glTF file format. — zlib
  • clgplot — A Gnuplot front-end for Common lisp — MIT Licence
  • clusters — Cluster algorithms in CL, for CL. — BSD simplified
  • generalized-reference — Generalized reference over structured data by pairwise reduction of arbitrary place identifiers for Common Lisp. — MIT
  • jsown-utils — Utilities for Common Lisp JSON library jsown — MIT
  • maidenhead — Convert coordinates between Latitude/Longitude and Maidenhead. — GPL-3
  • olc — Convert coordinates between Latitude/Longitude and Open Location Code. — GPL-3
  • one-more-re-nightmare — A regular expression compiler — BSD 2-clause
  • posix-shm — POSIX shared memory — BSD 3-Clause

Updated projects: 3b-bmfont, 3d-matrices, 3d-quaternions, 3d-transforms, a-cl-logger, access, adhoc, adopt, alexandria, april, arc-compat, architecture.builder-protocol, bp, chunga, ci, cl+ssl, cl-apertium-stream-parser, cl-capstone, cl-collider, cl-covid19, cl-data-structures, cl-dct, cl-diskspace, cl-editdistance, cl-forms, cl-fxml, cl-gamepad, cl-gopher, cl-gserver, cl-info, cl-isaac, cl-kaputt, cl-kraken, cl-lambdacalc, cl-liballegro-nuklear, cl-markless, cl-migratum, cl-mixed, cl-myriam, cl-online-learning, cl-patterns, cl-protobufs, cl-python, cl-random-forest, cl-sat, cl-sat.glucose, cl-sat.minisat, cl-sdl2, cl-smt-lib, cl-sparql, cl-str, cl-telegram-bot, cl-torrents, cl-vorbis, cl-wavelets, cl-webkit, cl-who, cl-wol, cl-yxorp, clingon, clog, closer-mop, cmd, com-on, common-lisp-jupyter, commondoc-markdown, conduit-packages, context-lite, croatoan, defmain, depot, doc, easy-routes, eazy-gnuplot, envy, esrap, event-emitter, factory-alien, file-select, fiveam, fmt, font-discovery, fresnel, functional-trees, gendl, geodesic, graph, gute, harmony, herodotus, hunchentoot-multi-acceptor, imago, jingoh, jose, journal, latter-day-paypal, let-over-lambda, lisp-binary, log4cl-extras, maiden, mcclim, media-types, mgl-pax, mgrs, mmap, mnas-package, mutility, named-readtables, neo4cl, nfiles, nibbles, nyxt, open-location-code, overlord, paren6, parsnip, pjlink, plot, polymorphic-functions, protobuf, qlot, query-repl, read-number, rove, rs-colors, sc-extensions, scriba, sel, serapeum, shasht, sly, snakes, speechless, spinneret, stumpwm, tfeb-lisp-tools, tiny-routes, trace-db, trivia, trivial-do, try, usocket, websocket-driver, xmls, yason, zippy.

Removed projects: cl-cut.

To get this update, use (ql:update-dist "quicklisp"). Enjoy!

vindarelWho's using Common Lisp ?

· 87 days ago

Everyone says "Nobody uses Lisp" and Lispers say "Yes they do, there's ITA, and, um, Autocad, and, uh, oh yeah, Paul Graham wrote Viaweb in Lisp!" Not very helpful for either side.

We now have a list: awesome-lisp-companies. It isn’t official nor exhaustive, but it’s way better than the past situaton.

Of course, see also:

In a nutshell, what’s Common Lisp good for? Let’s quote Kent Pitman’s famous answer:

But please don’t assume this is an exhaustive list, and please don’t assume Lisp is only useful for Animation and Graphics, AI, Bioinformatics, B2B and Ecommerce, Data Mining, EDA/Semiconductor applications, Expert Systems, Finance, Intelligent Agents, Knowledge Management, Mechanical CAD, Modeling and Simulation, Natural Language, Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web Authoring just because these are the only things they happened to list. Common Lisp really is a general language capable of a lot more than these few incidental application areas, even if this web page doesn’t totally bring that out.

(and this list doesn’t mention that it was used for auto-piloting the Deep Space 1 spaceship by the NASA for several days).


Here’s a shorter list of companies using CL in productionę.

Quantum computing companies use CL, especially SBCL:

Rigetti

They already sponsored a Quicklisp development. They chose Common Lisp (SBCL). Video. Their Lisp even runs 40% faster than their C code.

rigetti

D-wave systems, “quantum processor development”. “The software is implemented in Common Lisp (SBCL) and is an integral part of the quantum computing system.” lispjobs announce.

dwave

And there’s at least also HRL Laboratories in the Quantum space (I am not counting defunkt Quantum companies). They are “one of the world’s premier physical science and engineering research laboratories. [They] engage in the development of a full quantum device and computation stack, including an optimizing compiler for a quantum programming language”. Uses SBCL.

Grammarly is a famous English language writing-enhancement platform.

grammarly

Ravenpack is “the leading big data analytics provider for financial services”. reddit announce.

ravenpack

ITA Software is still Google’s leading airfaire search and scheduling platform. They use SBCL, and contribute to its development.

ita

Kina knowledge is a small company that “automates the processing of documents”: “We use Common Lisp extensively in our document processing software core for classification, extraction and other aspects of our service delivery and technology stack”. See their Lisp Interview: questions to Alex Nygren

Doremir Music Research AB develops ScoreCloud, a music notation software (a LispWorks product). It lets you play an instrument, sing or whistle into the app, and it writes the music score. Futuristic indeed.

I’ll stop here and I’ll let you discover the awesome-lisp-companies list!

Tim BradshawAvoiding circularity: a simple example

· 100 days ago

Here’s a simple example of dealing with a naturally circular function definition.

Common Lisp has a predicate called some. Here is what looks like a natural definition of a slightly more limited version of this predicate, which only works on lists, in Racket:

(define (some? predicate . lists)
  ;; Just avoid the spread/nospread problem
  (some*? predicate lists))

(define (some*? predicate lists)
  (cond
    [(null? lists)
     ;; if there are no elements the predicate is not true
     #f]
    [(some? null? lists)
     ;; if any of the lists is empty we've failed
     #f]
    [(apply predicate (map first lists))
     ;; The predicate is true on the first elements
     #t]
    [else
     (some*? predicate (map rest lists))]))

Well, that looks neat, right? Except it is very obviously doomed because some*? falls immediately into an infinite recursion.

Well, the trick to avoid this is to check whether the predicate is null? and handle that case explicitly:

(define (some*? predicate lists)
  (cond
    [(null? lists)
     ;; 
     (error 'some? "need at least one list")]
    [(eq? predicate null?)
     ;; Catch the circularity and defang it
     (match lists
       [(list (? list? l))
        (cond
          [(null? l)
           #f]
          [(null? (first l))
           #t]
          [else
           (some? null? (rest l))])]
       [_ (error 'some? "~S bogus for null?" lists)])]
    [(some? null? lists)
     ;; if any of the lists is empty we've failed
     #f]
    [(apply predicate (map first lists))
     ;; The predicate is true on the first elements
     #t]
    [else
     (some*? predicate (map rest lists))]))

And this now works fine.

Of course this is a rather inefficient version of such a predicate, but it’s nice. Well, I think it is.


Note: a previous version of this had an extremely broken version of some*? which worked, by coincidence, sometimes.

Tim BradshawTwo understandable deficiencies in Common Lisp

· 101 days ago

Common Lisp is, I think, a remarkably pleasant language, despite what some people like to say. Here are two small deficiencies, both of which are understandable in terms of the history of CL, and both of which ultimately hurt na´ve programmers working in CL.

The default floating-point type is single-float

There are two things that make this true:

  • *read-default-float-format* is initially single-float, which means that, unless it is changed, 1.0 reads as 1.0f0, a single float1;
  • The float function will convert to a single float unless it is given a prototype which is not a single float: (float 1) is 1.0f0, while to get a double float you would need (float 1 1.0d0).

In addition things like with-standard-io-syntax bind *read-default-float-format* to single-float, so you have to do a little more work to make doubles the default.

I think there are probably several historical reasons why this default was chosen:

  • a long time ago memory was very expensive and single floats take, usually, half the memory of double floats, thus pushing people towards single floats;
  • a long time ago, perhaps, on some machines, single float operations were significantly faster than double float operations even before possible float consing was taken into account;
  • Lisp hardware companies with significant influence on the standard, notably Symbolics, made hardware which allowed single (32 bit) floats to be immediate objects, while double floats were not, and had simple-minded compilers which were not capable of optimizing double float operations, thus making double float arithmetic extremely slow compared to single float arithmetic, and these companies wanted their machines to seem fast (they never, really, were) for na´ve users;
  • it was not clear that implementations would choose single-float to mean ‘single precision IEEE 754 float’ and double-float to mean ‘double precision IEEE 754 float’, for instance it’s perfectly legal to have the short-float type mean single precision IEEE 754 and all of the single-float, double-float and long-float types mean double precision IEEE 754;
  • it wasn’t even even clear that IEEE 754 would come to dominate how machines implement floating-point: VAXes didn’t, and other machines of interest at the time also did not.

So there are good historical reasons for this. However all implementations I’m aware of now translate short-float to mean single-float, single-float to mean IEEE 754 single precision, double-float to mean IEEE 754 double precision and long-float to be the same as double-float.

So what is the problem with the default float type being single-float in the modern world? The answer is

> (log (/ 1 single-float-epsilon) 10)
7.22472

In other words, single precision IEEE 754 arithmetic has about 7 significant figures of precision. For many purposes, and especially for na´vely-written code that’s at best marginal and at worst less than that. On the other hand

> (log (/ 1 double-float-epsilon) 10)
15.954589770191001D0

which is almost 16 significant figures of precision, more than twice that of single precision.

That’s why the default should have been double precision: it makes na´ve code more likely to work, and people who are writing non-na´ve code can use single precision if they need it.

The CL-USER package is defined in an implementation-dependent way

From the spec:

The COMMON-LISP-USER package is the current package when a Common Lisp system starts up. This package uses the COMMON-LISP package. The COMMON-LISP-USER package has the nickname CL-USER. The COMMON-LISP-USER package can have additional symbols interned within it; it can use other implementation-defined packages.

(My emphasis.)

What this means is that when you start a CL environment, the current package may have all sorts of implementation-dependent symbols visible in it. You can see why this happened: if you’re implementing Super-Whizz-Bang CL which has all sorts of magic extra features, you want at least some of those features to be immediately available to users, rather than requiring them to pore over boring manuals to find them.

But for users, and especially for na´ve users, it’s a terrible choice: na´ve users don’t know about packages so they write their programs in CL-USER. And they also don’t really know which symbols available in CL-USER come from CL and are thus standard parts of the language, and which come from one of Super-Whizz-Bang CL’s implementation packages, and are not standard parts of the language. So their programs turn into a mess where the portable parts are not distinct from the non-portable parts. The way the CL-USER package is defined thus makes it harder for to write programs whose non-portable parts are well-isolated, and ultimately hurts the language.

This is a direct conflict between implementors and users: implementors both want their extra features immediately available so their implementation is shinier and want to encourage users to use these extra features in a way which makes it hard to move their programs to other implementations; users, when they think about it, generally don’t want this second thing, at least.

Instead, the language should have defined CL-USER as a package which only used CL, and perhaps have defined another standard package, perhaps IMPL-USER, which was defined the way CL-USER is today.

Can these be fixed?

While both of these problems could be fixed without changing the standard, I don’t think either can realistically be fixed.

For the single-float problem there is nothing to stop implementations simply defining short-float to mean IEEE 754 single precision and all the other types to mean IEEE 754 double precision. But all the existing code which assumes otherwise will then probably break in exciting ways. So this is unlikely to happen I expect.

The CL-USER problem could be fixed if implementations agree to define CL-USER to use only CL as it is allowed to do, and perhaps to define an IMPL-USER package as above. Of course that will make implementations slightly less convenient to use, so the chances of it happening would be small, even if implementors actually talked to each other in any useful way which I suspect they no longer do. Worse than that, this change will break many programs written by na´ve users which live in CL-USER, and there are almost certainly lots of those.


A moment of convenience, a lifetime of regret, as the old saying goes.


  1. An earlier version of this article had single floats written as, for instance 1.0s0: that’s wrong, those are short floats, single floats are 1.0f0 for instance. These are almost certainly the same type on any current implementation (and I think on any implementation I have ever used, hence the mistake) but they don’t have to be. Thanks to Prem Nirved for finding this stupidity. 

Eitaro FukamachiQlot tutorial with Docker

· 107 days ago

Hi, all Common Lispers.

I've been writing about Roswell through 5 articles. I saw people trying out Roswell after reading my blog a couple of times, so I guess that helped to get interested.

I think it's time to move. Let's take Qlot for the next topic.

From the perspective of developing and operating applications over the long term, it is irreplaceable and crucial. But I consider that Qlot is still not widely accepted yet among the Common Lisp community. I'm not certain why, but it may be because of the lack of resources to learn it.

Qlot 1.0.0 was out on March 12th. I believe that this tool is now stable enough to be used in production. I hope you will take this opportunity to give it a try.

What's Qlot?

Qlot is a tool to fix versions of dependencies for each project. By fixing the versions of dependencies, it can be avoided breaking the app by version-up of dependencies unintentionally. Qlot ensures that all the same versions will be installed 5 years later.

It's important when running in other environments. Consider a project like a web application whose development environment is different from the environment it actually runs. Without Qlot, it will be cumbersome to use the same dependencies in all environments, regardless of when they are deployed. It's the same about CI/CD environments and other developers' machines.

It is not only useful for applications that connect to the Internet.

After Qlot v0.12.0 (released in November 2021), it got bundle command which allows to download all files of dependencies and make them loadable without Qlot/Quicklisp. It should be useful even for standalone applications.

Setup the project-local Quicklisp with Docker

I won't repeat the same explanations in Qlot's README, like how to use and the tutorial. Instead, I'm going to introduce how to use it with Docker.

As a starting point, create a new file "qlfile" at the project root. It is a file to write project dependencies. It's okay to be empty for now.

# Create a new empty file
$ touch qlfile

Let's install dependencies with it. The following command is equivalent to qlot install except it uses Docker:

# Equivalent to "qlot install"
$ docker run --rm -it -v $PWD:/app fukamachi/qlot install

It creates a new directory named .qlot. It is a project-local Quicklisp directory that contains dependencies written in qlfile. As a default, only a "quicklisp" dist is placed.

$ tree .qlot/dists
.qlot/dists
 quicklisp
     distinfo.txt
     enabled.txt
     preference.txt
     releases.txt
     systems.txt

1 directory, 5 files

There's another file named qlfile.lock at the same directory as qlfile. This file is generated by qlot install to keep track of versions at the time.

On my laptop, the content is like this:

$ cat qlfile.lock
("quicklisp".
 (:class qlot/source/dist:source-dist
  :initargs (:distribution "http://beta.quicklisp.org/dist/quicklisp.txt" :%version :latest)
  :version "2022-02-20"))

Some of the information is unnecessary for humans because it contains internal information for Qlot to use, but it is written here that the quicklisp dist version 2022-02-20 is used for this project.

While qlfile.lock exists, qlot install downloads quicklisp 2022-02-20 even when the newer version is released.

When you use VCS, like git, you don't want to version the .qlot directory since it contains a large number of source files of dependencies. The directory can be reproducible from qlfile.lock anytime by running qlot install.

$ echo .qlot/ >> .gitignore
$ git add qlfile qlfile.lock
$ git commit -m 'Start using Qlot.'

Using the project-local Quicklisp

There're several ways to use the project-local Quicklisp. REPL can be launched with Docker image by the following command:

# Equivalent to 'qlot exec ros run'
$ docker run --rm -it -v $PWD:/app fukamachi/qlot exec ros run
* ql:*quicklisp-home*
#P"/app/.qlot/
* (ql-dist:dist "quicklisp")
#<QL-DIST:DIST quicklisp 2022-02-20>

However, it would be inconvenient since it's inside a separated Docker container. Actually, it's possible to be loaded without Qlot, like these:

# With Roswell
$ QUICKLISP_HOME=.qlot/ ros run

# With sbcl command
# The point is loading .qlot/setup.lisp
$ sbcl --no-userinit --load .qlot/setup.lisp

It also can be applied to other implementations as long as it loads .qlot/setup.lisp on startup.

Let's see where to load the Quicklisp on the launched REPL:

* ql:*quicklisp-home*
#P"/Users/fukamachi/myproject/.qlot/"
* (ql-dist:dist "quicklisp")
#<QL-DIST:DIST quicklisp 2022-02-20>

It seems fine.

Adding a new dependency

Since here, Qlot only keeps track of the version of the Quicklisp dist.

Let's add another dependency, "clack" from GitHub. Add a line to qlfile and run qlot install.

# Run 'qlot add'
$ docker run --rm -it -v $PWD:/app fukamachi/qlot add github clack fukamachi/clack
Add 'github clack fukamachi/clack' to 'qlfile'.
Reading '/app/qlfile'...
Already have dist "quicklisp" version "2022-02-20".
Installing dist "clack" version "github-6fd0279424f7ba5fd4f92d69a1970846b0b11222".
Successfully installed.

# Same as the above
$ echo 'github clack fukamachi/clack' >> qlfile
$ docker run --rm -it -v $PWD:/app fukamachi/qlot install

After running qlot install, it applies the changes to qlfile.lock and .qlot/ directory. Now Clack of the latest GitHub version can be discovered in REPL:

$ QUICKLISP_HOME=.qlot/ ros run
* (ql:where-is-system :clack)
#P"/Users/fukamachi/myproject/.qlot/dists/clack/software/clack-6fd0279424f7ba5fd4f92d69a1970846b0b11222/"

If the added project provides Roswell scripts, Qlot adds scripts with the same names under .qlot/bin/. They are the same as the original scripts, except that they always use the version fixed in Qlot.

It refers to the default branch of GitHub (typically master or main), but it also can specify a specific branch or tag. See the README "qlfile syntax" section for detail.

Updating the version of dependencies

When you want to update a fixed version to the latest version, use qlot update.

For instance, when you find some changes of Clack on GitHub and want to use the newest version, run qlot update --project clack:

# Update a specific project (ex. Clack)
$ docker run --rm -it -v $PWD:/app fukamachi/qlot update --project clack

# Update all
$ docker run --rm -it -v $PWD:/app fukamachi/qlot update

qlot update works something like that ignores qlfile.lock, runs qlot install again, and updates the existing qlfile.lock.

Bundling dependencies

To dump all dependencies to the project root, qlot bundle is available.

Considering a project ASDF system like this:

(defsystem "myproject"
  :depends-on ("clack"
               "lack"))

qlot bundle extracts "clack" and "lack" including their dependencies into ".bundle-libs" directory.

$ docker run --rm -it -v $PWD:/app fukamachi/qlot bundle

To use it, just load .bundle-libs/bundle.lisp. It should work without Qlot or Quicklisp.

Alright, I explained through all daily operations with Qlot briefly.

In the next article, I will explain how to create your own Docker image based on this Docker container.


For older items, see the Planet Lisp Archives.


Last updated: 2022-06-30 00:00