Planet Lisp

Eric TimmonsStatic Executables with SBCL v2

· 4 days ago

It's taken me much longer than I hoped, but I finally have a second version of my patches to build static executables tested and ready to go! This set of patches vastly improves upon the first by reducing the amount of compilation needed at the cost of sacrificing a little purity. Additionally I have created a system that automates the process of building a static executable, along with other release related tasks.

At a Glance

  • The new patch set can be found on the static-executable-v2 branch of my SBCL fork or at https://www.timmons.dev/static/patches/sbcl/$VERSION/static-executable-support-v2.patch with a detached signature available at https://www.timmons.dev/static/patches/sbcl/$VERSION/static-executable-support-v2.patch.asc signed with GPG key 0x9ACF6934.
  • You'll definitely want to build SBCL with the :sb-prelink-linkage-table feature (newly added by the patch). You'll probably also want the :sb-linkable-runtime feature (already exists, but the patch also enables it on arm/arm64).
  • The new patch lets you build a static executable with less compilation of Lisp code.
  • The asdf-release-ops system automates the process of building a static executable by tying it into ASDF.

What's New?

If you need a refresher about what static executables are or what use cases they're good for, see my previous post on this topic.

With my previous patch, the only way you could create a static executable was to perform the following steps:

  1. Determine the foreign symbols needed by your code. The easiest way to do this is to compile all your Lisp code and then dump the information from the image.
  2. From that list of foreign symbols, create a C file that contains fills an array with references to those symbols.
  3. Recompile the SBCL core and runtime with this new file, additionally disabling libdl support and linking against your foreign libraries.
  4. (Re)compile all your Lisp code with the new runtime (if you made an image in step 1 it will not be compatible with the new runtime due to feature and build ID mismatches).
  5. Dump the executable.

In the most general case, this involved compiling your entire Lisp image twice. After some #lisp discussions, I realized there was a better way of doing this. While the previous process still works, the new recommended process now looks like:

  1. Build the image you would like to make into a static executable and save it.
  2. Dump the foreign symbol info from this image and write the C file that SBCL can use to prelink itself.
  3. Compile that C file and link it into an existing sbcl.o file to make a new runtime. sbcl.o is the SBCL runtime in object form, created when building with the :sb-linkable-runtime feature.
  4. Load the image from step 1 into your new runtime. It will be compatible because the build ID and feature set are the same!
  5. Dump your now static executable.

This new process can significantly reduce the amount of time needed to make an executable. Plus it lets you take more advantage of image based development. It's fairly trivial to build an image exactly like you want, dump it, and then pair it with a custom static runtime to make a static executable.

There were two primary challenges that needed to be overcome for this version of the patch set.

First, the SBCL core had to be made robust to every libdl function uncondtionally returning an error. Since we want the feature set to remain constant we can't recompile the runtime with #-os-provides-dlopen. Instead, we take advantage of the fact that Musl libc lets you link static executables against libdl, but all those functions are noops. This is the "purity" sacrifice I alluded to above.

Second, since we are reusing a image, the prelink info table (the generated C file) needed to order the symbols exactly as the image expects them to be ordered. The tricky bit here is that some libraries (like cl-plus-ssl) add symbols to the linkage table that will always be undefined. cl-plus-ssl does this in order to support a wide range of openssl versions. The previous patch set unconditionally filtered out undefined symbols, which horribly broke things in the new approach.

More Documentation

As before, after applying the patch you'll find a README.static-executable file in the root of the repo. You'll also find a Dockerfile and an example of how to use it in the README.static-executable.

You can also check out the tests and documentation in the asdf-release-ops system.

Known Issues

  • The :sb-prelink-linkage-table feature does not work on 32-bit ARM + Musl libc >= 1.2. Musl switched to 64-bit time under the hood while still mataining compatibility with everything compiled for 32-bit time.

The issue is how they maintained backwards compatibility. Every time related symbol still exists and implements everything on top of the 32-bit time interface. However, if you include the standard header file where the symbol is defined or you look up the symbol via dlsym you actually get a pointer to the 64-bit time version of the symbol. We can't use dlsym (it doesn't work in static executables). And the generated C file doesn't include any headers.

This could be fixed if someone is motiviated enough to create/find a complete, easy to use map between libc symbols and the headers that define them and integrate it into the prelink info generator.

  • The :sb-prelink-linkage-table works on Windows but causes test failures. The root issue is that mingw64 has implemented their own libm. Their trig functions are fast, but use inaccurate instructions (like FSIN) under the hood. When prelinking these inaccurate implementations are used instead of the more accurate ones (from msvcrt.dll ?) found when using dlsym to look up the symbol.

Next Steps

  1. I would love to get feedback on this approach and any ideas on how to improve it! Please drop me a line (etimmons on Freenode or daewok on Github/Gitlab) if you have suggestions.

  2. I've already incorporated static executables into CLPM and will be distributing them starting with v0.4.0! I'm going to continue rolling out static executables in my other projects.

  3. Pieces of the patch set are now solid enough that I think they can be submitted for upstream consideration. I'll start sending them after the current 2.1.2 freeze.

Max-Gerd Retzlaff"Curl/Wget for uLisp"<br />Or: An HTTP(s) get/post/put function for uLisp

· 5 days ago

Oh, I forgot to continue posting… I just published a quite comprehensive HTTP function supporting put, post, get, auth, HTTP and HTTPS, and more for uLisp at ulisp-esp-m5stack.

Activate #define enable_http and #define enable_http_keywords to get it; the keywords used by the http function are to be enabled separately as they might be used more general and not just by this function.

Note that you need to connect to the internet first. Usually with WIFI-CONNECT.

Here is the full documentation with example calls:

Syntax:
   http url &key verbose
                 (https t)
                 auth
                 (user default_username)
                 (password default_password)
                 accept
                 content-type
                 (method :get)
                 data
     => result-string

Arguments and values:
   verbose---t, or nil (the default); affects also debug output of the argument decoding itself and should be put in first position in a call for full effect.

   https---t (the default), nil, or a certificate as string; uses default certificate in C string root_ca if true; url needs to fit: "http://..." for true and and "https://..." for false.

   auth---t, or nil (the default).

   user---a string, or nil (the default); uses default value in C string default_username if nil; only used if :auth t.

   password---a string, or nil (the default); uses default value in C string default_password if nil; only used if :auth t.

   accept---nil (the default), or a string.

   content-type---nil (the default), or a string.

   method---:get (the default), :put, or :post.

   data---nil (the default), or a string; only necessary in case of :method :put or :method :post; error for :method :get.

Examples:
   ;; HTTP GET:
   (http "http://192.168.179.41:2342" :https nil)
 
   ;; HTTP PUT:
   (http "http://192.168.179.41:2342"
         :https nil
         :accept "application/n-quads"
         :content-type "application/n-quads"
         :auth t :user "foo" :password "bar"
         :method :put
         :data (format nil "<http://example.com/button> <http://example.com/pressed> \"~a\" .~%"
                           (get-time)))

It can be tested with an minimal HTTP server simulation using bash and netcat:

while true; do echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l -p 2342 -q 1; done
(To test with HTTPS in a similar fashion you can use openssl s_server, as explained, for example, in the article Create a simple HTTPS server with OPENSSL S_SERVER by Joris Visscher on July 22, 2015, but then you need to use certificates.)


See also Again more features for uLisp on M5Stack (ESP32):
time via NTP, lispstring without escaping and more space
, More features for uLisp on M5Stack (ESP32):
flash support, muting of the speaker and backlight control
and uLisp on M5Stack (ESP32).

Read the whole article.

Max-Gerd RetzlaffAgain more features for uLisp on M5Stack (ESP32):<br />time via NTP, lispstring without escaping and more space

· 12 days ago

I just pushed three small things to ulisp-esp-m5stack: Get time via NTP, add optional escape parameter to function lispstring, increased WORKSPACESIZE and SYMBOLTABLESIZE for M5Stack.

Getting time via NTP

Enable #define enable_ntptime to get time via NTP. New functions INIT-NTP and GET-TIME. Note that you need to connect to the internet first. Usually with WIFI-CONNECT.

Syntax: INIT-NTP -> nil

Initializes and configures NTP.

Syntax: GET-TIME -> timestamp

Returns a timestamp in the format of xsd:dateTime.

Add optional escape parameter to function lispstring

I have changed the function lispstring to have an optional escape parameter to switch off the default behavior of handling the backslash escape character. The default behavior is not changed.

The C function lispstring takes a C char* string and return a uLisp string object. When parsing data in the n-triples format retrieved via HTTP I noticed that the data got modified already by lispstring which broke my parser implemented in uLisp.

As lispstring might be used in other contexts that expect this behavior, I just added the option to switch the un-escaping off.

Increased WORKSPACESIZE and SYMBOLTABLESIZEfor M5Stack

The M5Stack ESP32 has 320 kB of usable DRAM in total. Although with a lot of restrictions (see next section),

I increased WORKSPACESIZE to 9000 cells, which equals 72,000 bytes, and SYMBOLTABLESIZE to 2048 bytes. These sizes seem to work still safely even with bigger applications and a lot of consing.

Warning: You cannot load images created with different settings!

The SRAM of the M5Stack ESP32

In total the M5Stack ESP32 comes with 520 kB of SRAM. The catch is that the ESP32 is based on the Harvard architecture and 192 kB is in the SRAM0 block intended(!) for instructions (IRAM). There is another 128 kB block in block SRAM1 which can be used either for instructions or data (DRAM). The third block SRAM2 has got a size of 200 kB and is for data only. But 8 kB of SRAM2 is lost for ROM mappings.

The ESP-IDF and thus also the Arduino environment use only SRAM0 for instructions and SRAM1 and SRAM2 for data, which is fine for uLisp as it is an interpreter and therefore more RAM for data is perfect. SRAM0 will just hold the machine code of the uLisp implementation but no code written in the language uLisp.

Of the remaining 320 kB another 54 kB will be dedicated for Bluetooth if Bluetooth is enabled in ESP-IDF (which it is by default, #define CONFIG_BT_RESERVE_DRAM 0xdb5c) in the SRAM2 block. And if trace memory is enabled, another 32 kB of SRAM1 are reserved (by default it is disabled, #define CONFIG_TRACEMEM_RESERVE_DRAM 0x0).

So, by default with Bluetooth enabled and trace memory disabled, 266 kB are left. At the bottom of SRAM2 right after the 62 kB used for Bluetooth and ROM are the application's data and BSS segments. Sadly, at around the border between SRAM1 and SRAM2 there seem to be two small reserved regions again of a bit more the 1 kB each, limiting statically allocated memory.

Thus, the "shared data RAM" segment dram0_0_seg in the linker script memory layout is configured to have a default length of 0x2c200 -; CONFIG_BT_RESERVE_DRAM. That is, 176.5 kB (= 180,736 bytes) without Bluetooth and 121.66 kB (= 124,580 bytes) with Bluetooth enabled.

But actually I have already written more than I have intended for this blog post and the rest of my notes, calculations and experiments will have to wait for a future article. For now, I just increased the size of the statically allocated uLisp workspace to make more use of the available memory of the ESP32 in the M5Stack.


See also More features for uLisp on M5Stack (ESP32):
flash support, muting of the speaker and backlight control
and uLisp on M5Stack (ESP32).

References

Espressif Systems, ESP32 Technical Reference Manual, Shanghai, 2020, section 2.3.2 Embedded Memory.

Read the whole article.

Tycho Garen Programming in the Common Lisp Ecosystem

· 12 days ago

I've been writing more and more Common Lips recently and while I reflected a bunch on the experience in a recent post that I recently followed up on .

Why Ecosystems Matter

Most of my thinking and analysis of CL comes down to the ecosystem: the language has some really compelling (and fun!) features, so the question really comes down to the ecosystem. There are two main reasons to care about ecosystems in programming languages:

  • a vibrant ecosystem cuts down the time that an individual developer or team has to spend doing infrastructural work, to get started. Ecosystems provide everything from libraries for common tasks as well as conventions and established patterns for the big fundamental application choices, not to mention things like easily discoverable answers to common problems.

    The more time between "I have an idea" to "I have running (proof-of-concept quality) code running," matters so much. Everything is possible to a point, but the more friction between "idea" and "working prototype" can be a big problem.

  • a bigger and more vibrant ecosystem makes it more tenable for companies/sponsors (of all sizes) to choose to use Common Lisp for various projects, and there's a little bit of a chicken and egg problem here, admittedly. Companies and sponsors want to be confidence that they'll be able to efficiently replace engineers if needed, integrate or lisp components into larger ecosystems, or be able to get support problems. These are all kind of intangible (and reasonable!) and the larger and more vibrant the ecosystem the less risk there is.

    In many ways, recent developments in technology more broadly make lisp slightly more viable, as a result of making it easier to build applications that use multiple languages and tools. Things like microservices, better generic deployment orchestration tools, greater adoption of IDLs (including swagger, thrift and GRPC,) all make language choice less monolithic at the organization level.

Great Things

I've really enjoyed working with a few projects and tools. I'll probably write more about these individually in the near future, but in brief:

  • chanl provides. As a current/recovering Go programmer, this library is very familiar and great to have. In some ways, the API provides a bit more introspection, and flexibility that I've always wanted in Go.
  • lake is a buildsystem tool, in the tradition of make, but with a few additional great features, like target namespacing, a clear definition between "file targets" and "task targets," as well as support for SSH operations, which makes it a reasonable replacement for things like fabric, and other basic deployment tools.
  • cl-docutils provides the basis for a document processing system. I'm particularly partial because I've been using the python (reference) implementation for years, but the implementation is really quite good and quite easy to extend.
  • roswell is really great for getting started with CL, and also for making it possible to test library code against different implementations and versions of the language. I'm a touch iffy on using it to install packages into it's own directory, but it's pretty great.
  • ASDF is the "buildsystem" component of CL, comparable to setuptools in python, and it (particularly the latest versions,) is really great. I like the ability to produce binaries directly from asdf, and the "package-inferred" is a great addition (basically, giving python-style automatic package discovery.)
  • There's a full Apache Thrift implementation. While I'm not presently working on anything that would require a legit RPC protocol, being able to integrate CL components into larger ecosystem, having the option is useful.
  • Hunchensocket adds websockets! Web sockets are a weird little corner of any stack, but it's nice to be able to have the option of being able to do this kind of programming. Also CL seems like a really good platform to do
  • make-hash makes constructing hash tables easier, which is sort of needlessly gawky otherwise.
  • ceramic provides bridges between CL and Electron for delivering desktop applications based on web technologies in CL.

I kept thinking that there wouldn't be good examples of various things, (there's a Kafka driver! there's support for various other Apache ecosystem components,) but there are, and that's great. There's gaps, of course, but fewer, I think, than you'd expect.

The Dark Underbelly

The biggest problem in CL is probably discoverability: lots of folks are building great tools and it's hard to really know about the projects.

I thought about phrasing this as a kind of list of things that would be good for supporting bounties or something of the like. Also if I've missed something, please let me know! I've tried to look for a lot of things, but discovery is hard.

Quibbles

  • rove doesn't seem to work when multi-threaded results effectively. It's listed in the readme, but I was able to write really trivial tests that crashed the test harness.
  • Chanl would be super lovely with some kind of concept of cancellation (like contexts in Go,) and while it's nice to have a bit more thread introspection, given that the threads are somewhat heavier weight, being able to avoid resource leaks seems like a good plan.
  • There doesn't seem to be any library capable of producing YAML formated data. I don't have a specific need, but it'd be nice.
  • it would be nice to have some way of configuring the quicklisp client to be able to prefer quicklisp (stable) but also using ultralisp (or another source) if that's available.
  • Putting the capacity in asdf to produce binaries easily is great, and the only thing missing from buildapp/cl-launch is multi-entry binaries. That'd be swell. It might also be easier as an alternative to have support for some git-style sub-commands in a commandline parser (which doesn't easily exist at the moment'), but one-command-per-binary, seems difficult to manage.
  • there are no available implementations of a multi-reader single-writer mutex, which seems like an oversite, and yet, here we are.

Bigger Projects

  • There are no encoders/decoders for data formats like Apache Parquet, and the protocol buffers implementation don't support proto3. Neither of these are particular deal breakers, but having good tools dealing with common developments, lowers to cost and risk of using CL in more applications.
  • No support for http/2 and therefore gRPC. Having the ability to write software in CL with the knowledge that it'll be able to integrate with other components, is good for the ecosystem.
  • There is no great modern MongoDB driver. There were a couple of early implementations, but there are important changes to the MongoDB protocol. A clearer interface for producing BSON might be useful too.
  • I've looked for libraries and tools to integrate and manage aspects of things like systemd, docker, and k8s. k8s seems easiest to close, as things like cube can be generated from updated swagger definitions, but there's less for the others.
  • Application delievery remains a bit of an open. I'm particularly interested in being able to produce binaries that target other platforms/systems (cross compilation,) but there are a class of problems related to being able to ship tools once built.
  • I'm eagerly waiting and concerned about the plight of the current implementations around the move of ARM to Darwin, in the intermediate term. My sense is that the transition won't be super difficult, but it seems like a thing.

Max-Gerd RetzlaffMore features for uLisp on M5Stack (ESP32):<br />flash support, muting of the speaker and backlight control

· 12 days ago

I finished the IoT sensor device prototype and shipped it last Thursday. It just has a stub bootstrap system in the flash via uLisp's Lisp Library and downloads the actual application in a second boot phase via HTTPS. More on that later.

To make it happen I've added a bunch of things to ulisp-esp-m5stack: flash support, fixes for some quirks of the M5Stack, time via NTP, an HTTP function supporting methods PUT, POST, GET, Auth, HTTP and HTTP, temperature sensors via one wire, and more. I plan to publish all these features in the next days.

Today you get: flash support, muting of the builtin speaker and control of the LED backlight of the builtin display.

Read the whole article.

Quicklisp newsNewer Quicklisp client available

· 14 days ago

 I had to revert the change that allows slashes in dist names for Ultralisp. If your Quicklisp directory has a lot of files and subdirectories (which is normal), the wild-inferiors file search for dist info is unacceptably slow. 

You can get an updated client with the feature reverted with (ql:update-client).

Quicklisp newsNew Quicklisp client available

· 16 days ago

 I've just published a new version of the Quicklisp client. You can get it with (ql:update-client).

This version updates the fallback ASDF from 2.26 to 3.2.1. (This will not have any effect on any implementation except CLISP, which does not come with ASDF of any version.)

It also includes support for dists with slashes in the name, as published by Ultralisp.

Thanks to those who contributed pull requests incorporated in this update.

Nicolas HafnerSetting Up Camp - February Kandria Update

· 20 days ago

header
I hope you've all started well into the new year! We're well into production now, with the vertical slice slowly taking shape. Much of the work in January has been on concept and background work, which is now done, so we are moving forward on the implementation of the new features, assets, and writing. This entry will have a lot of pictures and videos to gander at, so be warned!

The vertical slice will include three areas - the central camp, or hub location, the first underground area, and the desert ruins. We're now mostly done implementing the central camp. Doing so was a lot of work, since it requires a lot of unique assets. It still requires a good amount of polish before it can be called well done, but for the vertical slice I think we're good at the point we are now.

camp-1 camp-3

The camp is where all the main cast are (Fi, Jack, Catherine, and Alex), and where you'll return to after most missions. As such, it's important that it looks nice, since this is where you'll spend a lot of your time. It also has to look believable and reasonable for the cast to try and live here, so we spent a good amount of time thinking about what buildings there would be, what purpose they should fulfil, and so forth.

We also spent a good deal of time figuring out the visual look. Since Kandria is set quite far into the future, with that future also having undergone a calamity, the buildings both have to look suitably modern for a future society to have built, but at the same time ruined and destroyed, to fit the calamity event.

camp-2 camp-4

I also finished the character redesign for Fi. Her previous design no longer really fit with her current character, so I really wanted to get that done.

fi-draft fi

On the gameplay side the movement AI has been revised to be able to deal with far more complicated scenarios. Characters can now follow you along, move to various points on the map independently, or lead the player to a destination.

Quests now also automatically track your time to completion, which allows us both to do some nice tracking for score and speedrun purposes, but also to implement a 'race' quest. We have a few ideas on those, and it should serve as a nice challenge to try and traverse the various areas as quickly as possible.

We're also thinking of setting up leaderboards or replays for this, but that's gonna have to wait until after the vertical slice.

For look and feel there's also been a bunch of changes. First, there's now a dedicated particle system for effects like explosions, sparks, and so forth. Adding such details really enhances the feel of the combat, and gives a nice, crunchy, oomph to your actions. I still have a few more ideas for additional effects to pile on top, and I'll see that I can get to those in due time.

particles

Also on the combat side, there's now a quick-use menu so you can access your healing items and so forth easily during combat. It even has a nice slow-mo effect!

Since we're not making a procedural game, we do have to have a way of gating off far areas in a way that feels at least somewhat natural. To do this I've implemented a shader effect that renders a sandstorm on top of everything. The strength of the effect can be fine-tuned, so we could also use it for certain setpieces or events.

The effect looks a lot better in-game. Video compression does not take kindly to very noisy and detailed effects like this. Having the sand howl around really adds a lot to the feel of the game. In a similar vein, there's also grass and other foliage that can be placed now, which reacts to the wind and characters stepping on it. You can see that in action in this quick run-down of the camp area:

There's a bunch of other things we can't show off quite yet, especially a bunch of excellent animations by Fred. I haven't had the time to integrate all of those yet!

We've also been thinking more about how to handle the marketing side of things. I'm now doing a weekly screenshotsaturday thing on Twitter, and semi-regularly post quick progress gifs and images as well. Give me a follow if you haven't yet!

Then I took advantage of Rami Ismail's excellent consulting service and had a talk with him about what we should do to improve the first impressions for Kandria and how to handle the general strategy. He gave some really excellent advice, though I wish I had had more time to ask other questions, too! I'll probably schedule a consultancy hour with him later this year to catch up with all of that.

Anyway, I think a lot of the advice he gave us isn't necessarily specific to Kandria, so I thought it would be good to share it here, in case you're a fellow developer, or just interested in marketing in general:

  • Make sure to keep a consistent tone throughout your paragraph or trailer. This means that you want to avoid going back and forth between advertising game features or narrative elements, for instance. In Kandria's case we had a lot of back and forth in our press kit and steam page texts, which we've now gone over and revised to be more consistent.
  • Marketing is as much about attracting as many people as possible as it is about pushing people away. You want to be as efficient as possible at advertising to your target group. This also means being as up-front as possible about what your game is and who it is for, so you immediately pull in the people that would care about it, and push away the people that would not.
  • You need to figure out which part of your game best appeals to your core audience, and how you need to put it to make it attractive. Having an advertisement platform that gives you plenty of statistics and targeting features is tremendously helpful for this. Rami specifically suggested using short Facebook ads, since those can be targeted towards very specific groups. Do many small ads using different copy texts and trailers to see which work the best at attracting people to your Steam page.
  • Always use a call to action at the end of your top of the funnel (exposure) marketing. In fact, don't just use one link, use one for every way people have to interact with your game, if you have several. For us in specific this means I'll now include a link to our mailing list, our discord, and our steam page in our material.
  • Only use community/marketing platforms that you're actually comfortable with engaging with yourself. This means don't force yourself to make a Discord or whatever if you're not going to really engage with it. I'm fairly comfortable with where we are now, though I'm considering also branching out to imgur for more top of the funnel marketing. We'll see.
  • Two years is plenty of time to get marketing going. Generally you want to really up the hype train about three months before release. The wishlist peak about one month before release should give you a rough idea of whether the game is going to be successful or not - 5-10k is good, 15-20k should be very good.
  • Three weeks before release is when you want to start contacting press - write emails to people that have reviewed the games that inspired yours and seem to generally fit the niche you're targeting. Let them know you'll send a final build a week before release.
  • Actually do that exactly a week before release. Ideally your game will be done and you won't fudge with it until after release.
  • On the day before release, log onto gamespress.com and submit your game. Actual journalists don't tend to look there it seems, since they already get way more than enough mail, but third parties and independent people might!

And that's about what we managed to discuss in the 20 minutes we had. As mentioned, I'll probably schedule another consultancy later in the year. I'll be sure to let you know how it went!

Alright, I've run my mouth for long enough now, here's some words from Tim about his experience for January:

It's been a documentation-heavy month for me: designing the vertical slice quests on paper (which will become the first act of the narrative), making some tweaks to the characters and plots to fit the game's pillars, and also tweaking the press kit and marketing copy from Rami's feedback.

The last two weeks I've also started implementing the first quest, reminding myself how to use the scripting language and editor (it's amazing how much you forget after a couple of weeks away from it). This has also involved familiarising myself with the "proper" quest structure, using the hierarchy of quest > task > trigger (for the demo quest it was more like task > trigger, trigger, trigger, etc. you get the idea). What's been most fun though is getting into the headspace for Jack and Catherine, writing their initial dialogues, and threading in some player choice. Catherine is quickly becoming my favourite character.

It's also been great to see the level design and art coming along - Nick's sketched layouts, and now the pixel art for the ruined buildings which he and Fred have been working on. Oh, and seeing the AI in action, with Catherine bounding along after The Stranger blew my mind.

Well, that's about it for this month. It's been exciting to finally see a change in the visuals, and I'm excited to start tackling the first underground area. I see a lot more pixel work ahead of us...

Anyway, in the meantime until the next monthly update, do consider checking out the mailing list if you want more in-depth, weekly updates on things. We cover a lot of stuff there that never makes it into the monthlies, too! If you want to get involved in discussions and feedback around the game, hop onto the discord. We're slowly building a community of fans there, and are trying to post more actively about the process. For a more casual thing, there's also my twitter with plenty of gifs and images. Finally, please do wishlist Kandria on Steam! It might seem like it isn't much, but it really does help out a lot!

Thanks for reading, and see you next time!

Vsevolod Dyomkin"Programming Algorithms in Lisp" Is Out!

· 20 days ago

The updated version of my book "Programming Algorithms" has been released by Apress recently. It has undergone a number of changes that I want to elaborate on in this post.

But first, I'd like to thank all the people who contributed to the book or supported my work on it in other ways. It was an honor for me to be invited to Apress as "Practical Common Lisp" published by them a decade ago was my one-way ticket to the wonderful world of Lisp. Writing "Programming Algorithms" was, in a way, an attempt to give something back. Also, I was very curious to see how the cooperation with the publisher would go. And I can say that they have done a very professional job and helped significantly improve the book through the review process. That 5-10% change that is contributed by the editors, although it may seem insignificant, is very important to bring any book to the high standard that allows not to annoy many people. Unfortunately, I am not a person who can produce a flawless result at once, so helping with correcting those flaws is very valuable. Part of gratitude for that also, surely, goes to many of the readers who have sent their suggestions.

I was very pleased that Michał "phoe" Herda has agreed to become the technical reviewer. He has found a number of bugs and suggested lots of improvements, of which I could implement, maybe, just a third. Perhaps, the rest will go into the second edition :)

Now, let's speak about some of those additions to Programming Algorithms in Lisp.

Curious Fixes

First of all, all the executable code from the book was published in a github repo (and also republished to the oficial Apress repo). As suggested by Michał, I have added automated tests to ensure (for now, partially, but we plan to make the test suite all-encompassing) that everything compiles and runs correctly. Needless to say that some typos and other issues were found in the process. Especially, connected with handling different corner cases. So, if you have trouble running some code from the book, you can use the github version. Funny enough, I got into a similar situation recently, when I tried to utilize the dynamic programming example in writing a small tool for aligning outputs of different ASR systems and found a bug in it. The bug was is in the matrix initialization code:


-    (dotimes (k (1+ (length s1))) (setf (aref ld k 0) 0))
-    (dotimes (k (1+ (length s2))) (setf (aref ld 0 k) 0)))
+    (dotimes (k (1+ (length s1))) (setf (aref ld k 0) k))
+    (dotimes (k (1+ (length s2))) (setf (aref ld 0 k) k)))

Another important fix that originated from the review process touched not only the book but also the implementation of the slice function in RUTILS! It turned out that I was naively assuming that displaced arrays will automatically recursively point into the original array, and thus, inadvertently, created a possibility for O(n) slice performance instead of O(1). It explains the strange performance of array sorting algorithms at the end of Chapter 5. After fixing slice, the measurements started to perfectly resemble the theoretical expectations! And, also the performance has improved an order of magnitude :D


CL-USER> (let ((vec (random-vec 10000)))
           (print-sort-timings "Insertion " 'insertion-sort vec)
           (print-sort-timings "Quick" 'quicksort vec)
           (print-sort-timings "Prod" 'prod-sort vec))
= Insertion sort of random vector (length=10000) =
Evaluation took:
  0.632 seconds of real time
...
= Insertion sort of sorted vector (length=10000) =
Evaluation took:
  0.000 seconds of real time
...
= Insertion sort of reverse sorted vector (length=10000) =
Evaluation took:
  1.300 seconds of real time
...
= Quicksort of random vector (length=10000) =
Evaluation took:
  0.039 seconds of real time
...
= Quicksort of sorted vector (length=10000) =
Evaluation took:
  1.328 seconds of real time
...
= Quicksort of reverse sorted vector (length=10000) =
Evaluation took:
  1.128 seconds of real time
...
= Prodsort of random vector (length=10000) =
Evaluation took:
  0.011 seconds of real time
...
= Prodsort of sorted vector (length=10000) =
Evaluation took:
  0.011 seconds of real time
...
= Prodsort of reverse sorted vector (length=10000) =
Evaluation took:
  0.021 seconds of real time
...

Also, there were some missing or excess closing parens in a few code blocks. This, probably, resulted from incorrectly copying the code from the REPL after finishing experimenting with it. :)

New Additions

I have also added more code to complete the full picture, so to say, in several parts where it was lacking, from the reviewers' point of view. Most new additions went into expanding "In Action" sections where it was possible. Still, unfortunately, some parts remain on the level of general explanation of the solution as it was not possible to include whole libraries of code into the book. You can see a couple of snippets below:

Binary Search in Action: a Fast Specialized In-Memory DB

We can outline the operation of such a datastore with the following key structures and functions.

A dictionary *dict* will be used to map words to numeric codes. (We'll discuss hash-tables that are employed for such dictionaries several chapters later. For now, it will be sufficient to say that we can get the index of a word in our dictionary with (rtl:? *dict* word)). The number of entries in the dictionary will be around 1 million.

All the ngrams will be stored alphabetically sorted in 2-gigabyte files with the following naming scheme: ngram-rank-i.bin. rank is the ngram word count (we were specifically using ngrams of ranks from 1 to 5) and i is the sequence number of the file. The contents of the files will constitute the alternating ngram indices and their frequencies. The index for each ngram will be a vector of 32-bit integers with the length equal to the rank of an ngram. Each element of this vector will represent the index of the word in *dict*. The frequency will also be a 32-bit integer.

All these files will be read into memory. As the structure of the file is regular — each ngram corresponds to a block of (1+ rank) 32-bit integers — it can be treated as a large vector.

For each file, we know the codes of the first and last ngrams. Based on this, the top-level index will be created to facilitate efficiently locating the file that contains a particular ngram.

Next, binary search will be performed directly on the contents of the selected file. The only difference with regular binary search is that the comparisons need to be performed rank times: for each 32-bit code.

A simplified version of the main function get-freq intended to retrieve the ngram frequency for ranks 2-5 will look something like this:


(defun get-freq (ngram)
  (rt:with ((rank (length ngram))
            (codes (ngram-codes ngram))
            (vec index found?
                 (bin-search codes
                             (ngrams-vec rank codes)
                             :less 'codes<
                             :test 'ngram=)))
     (if found?
         (aref vec rank)
         0)))

where


(defun ngram-codes (ngram)
  (map-vec (lambda (word) (rtl:? *dict* word))
           ngram))

(defun ngrams-vec (rank codes)
  (loop :for ((codes1 codes2) ngrams-vec) :across *ngrams-index*
        :when (and (<= (aref codes1 0) (aref codes 0))
                   (codes< codes codes2 :when= t))
        :do (return ngrams-vec)))
             
(defun codes< (codes1 codes2 &key when=)
  (dotimes (i (length codes1)
              ;; this will be returned when all
              ;; corresponding elements of codes are equal
              when=)
    (cond ((< (aref codes1 i)
              (aref codes2 i))
           (return t))
          ((> (aref codes1 i)
              (aref codes2 i))
           (return nil)))))

(defun ngram= (block1 block2)
  (let ((rank (1- (length block1))))
    (every '= (rtl:slice block1 0 rank)
              (rtl:slice block2 0 rank)))

We assume that the *ngrams-index* array containing a pair of pairs of codes for the first and last ngram in the file and the ngrams data from the file itself was already initialized. This array should be sorted by the codes of the first ngram in the pair. A significant drawback of the original version of this program was that it took quite some time to read all the files (tens of gigabytes) from disk. During this operation, which measured in several dozens of minutes, the application was not responsive. This created a serious bottleneck in the system as a whole and complicated updates, as well as put normal operation at additional risk. The solution we utilized to counteract this issue was a common one for such cases: switching to lazy loading using the Unix mmap facility. With this approach, the bounding ngram codes for each file should be precalculated and stored as metadata, to initialize the *ngrams-index* before loading the data itself.

Pagerank MapReduce Explanation


;; this function will be executed by mapper workers
(defun pr1 (node n p &key (d 0.85))
  (let ((pr (make-arrray n :initial-element 0))
        (m (hash-table-count (node-children node))))
    (rtl:dokv (j child (node-children node))
      (setf (aref pr j) (* d (/ p m))))
    pr))

(defun pagerank-mr (g &key (d 0.85) (repeat 100))
  (rtl:with ((n (length (nodes g)))
             (pr (make-arrray n :initial-element (/ 1 n))))
    (loop :repeat repeat :do
      (setf pr (map 'vector (lambda (x)
                              (- 1 (/ d n)))
                    (reduce 'vec+ (map 'vector (lambda (node p)
                                                 (pr1 node n p :d d))
                                       (nodes g)
                                       pr)))))
    pr))

Here, we have used the standard Lisp map and reduce functions, but a map-reduce framework will provide replacement functions which, behind-the-scenes, will orchestrate parallel execution of the provided code. We will talk a bit more about map-reduce and see such a framework in the last chapter of this book.

One more thing to note is that the latter approach differs from the original version in that each mapper operates independently on an isolated version of the pr vector, and thus the execution of Pagerank on the subsequent nodes during a single iteration will see an older input value p. However, since the algorithm is stochastic and the order of calculations is not deterministic, this is acceptable: it may impact only the speed of convergence (and hence the number of iterations needed) but not the final result.

Other Significant Changes

My decision to heavily rely on syntactic utilities from my RUTILS library was a controversial one, from the start. And, surely, I understood it. But my motivation, in this regard, always was and still remains not self-promotion but a desire to present Lisp code so that it didn't seem cumbersome, old-fashioned, or cryptic (and, thankfully, the language provides all possibilities to tune its surface look to your preferences). However, as it bugged so many people, including the reviewers, for the new edition, we have come to a compromise to use all RUTILS code only qualified with the rtl prefix so that it was apparent. Besides, I have changed some of the minor purely convenience abbreviations to their standard counterparts (like returning funcall instead of call).

Finally, the change that I regret the most, but understand that it was inevitable, is the change of title and the new cover, which is in standard Apress style. However, they have preserved the Draco tree in the top right corner. And it's like a window through which you can glance at the original book :)  


So, that is an update on the status of the book.

For those who were waiting for the Apress release to come out, it's your chance to get it. The price is quite affordable. Basically, the same as the one I asked for (individual shipping via post is a huge expense).

And for those who have already gotten the original version of the book, all the major changes and fixes are listed in the post. Please, take notice if you had any issues.

I hope the book turns out to be useful to the Lisp community and serves both Lisp old-timers and newcomers.

Max-Gerd RetzlaffStumpWM: vsplit-three

· 21 days ago

A good ten month ago I switched away from a full desktop environment being finally tired enough that user software gets more and ever more features and tries to anticipate more and more what I might want but in the end my own computer never actually does what I want and only that. PulseAudio being the most dreaded example of a piece of code that gets more, more and more magic and complexity and in the end it never does what you actually want while at the same time telling it to do so got completely impossible because of many layers of abstractions and magic. PulseAudio has a lot of "rules", "profiles", "device intended roles", "autodetecting", "automatic setup and routing" and "other housekeeping actions". Look at this article PulseAudio under the hood by Victor Gaydov (which is also the source of the terms I just quoted): It has 174 occurrences of words starting with "auto-": automatically – 106, automatic – 27, autoload – 16, autospawn – 14, autodetect – 4, autoexit – 2, automate – 2, auto timing – 1, auto switch – 1, and once "magically" when it is even too much for the author.

So, more control and less clutter instead. After years again I use just a good old window manager, individual programs, and got rid of PulseAudio.

I switched to StumpWM which is written in Common Lisp. It is easy to modify and try stuff. While it's running. I have it run Slime so that I can connect to it from Emacs and hack stuff that is missing. From time to time I got StumpWM hanging while hacking, so I added a signal handler for POSIX signal SIGHUP to force a hard stumpwm restart. (There is a new version of that signal handler without the CFFI dependency but that pull request is not merged yet.) When I did something stupid I switch to a console, fire a killall -HUP stumpwm to have it reset hard. Since then I haven't lost a X11 session even while changing quite a bit.

Read the whole article.

Jonathan GodboutProto Cache: Flags and Hooks

· 25 days ago

Today’s Updates

Last week we made our Pub/Sub application use protocol buffer objects for most of its internal state. This week we'll take advantage of that change by setting startup and shutdown hooks to load state and save state respectively. We will add flags so someone starting up our application can set the load and save files on the command line. We will then package our application into an executable with a new asdf command.

Code Changes

Proto-cache.lisp

Defpackage Updates:

We will use ace.core.hook to implement our load and exit hooks. We will show how to make methods that will run at load and exit time when we use this library in the code below. In the defpackage we use the nickname hook. The library is available in the ace.core repository.

We use ace.flag as our command line flag parsing library. This is a command line flag library used extensively at Google for our lisp executables. The library can be found in the ace.flag repository.

Flag definitions:

We define three command line flags:

  • flag::*load-file*
  • flag::*save-file*
  • flag::*new-subscriber* 
    • This flag is used for testing purposes. It should be removed in the future.
  • flag::*help*

The definitions all look the same, we will look at flag::*load-file* as an example:

(flag:define flag::*load-file* ""
  "Specifies the file from which to load the PROTO-CACHE on start up."
  :type string)
  • We use the flag:define macro to define a flag. Please see the code for complete documentation of this macro (REAME.md update coming). We only use a small subset of the ace.flag package.
  • flag::*load-file*: This is the global where the parsed command line flag will be stored.
  • The documentation string to document the flag. If flag:print-help is called this documentation will be printed:

    --load-file (Determines the file to load PROTO-CACHE from on startup)

     Type: STRING

  • :type : The type of the flag. Here we have a string.

We use the symbol-name string of the global in lowercase as the command line input. 

For example:

  1. flag::*load-file* becomes --load-file
  2. flag::*load_file* becomes –load_file

The :name or :names key in the flag:define macro will let users select their own names for the command line input instead of this default.

Main definition:

We want to create a binary for our application. Since we have no way to add publishers and subscribers outside of the repl we define a dummy main that adds publishers and subscribers for us:

(defun main ()
  (register-publisher "pika" "chu")
  (register-subscriber "pika" flag::*new-subscriber*)
  (update-publisher-any
    "pika" "chu"
    (google:make-any :type-url "a"))
  ;; Sleep to make sure running threads exit.
  (sleep 2))

After running the application we can check for a new subscriber URL in the saved proto-cache application state file. I will show this shortly.

Load/Exit hooks:

We have several pre-made hooks defined in ace.core.hook. Two useful functions are ace.core.hook:at-restart and ace.core.hook:at-exit. As one can imagine, at-restart runs when the lisp image starts up, and at-exit runs when the lisp image is about to exit.

The first thing we do when we start our application is parse our command line:

(defmethod hook::at-restart parse-command-line ()
  "Parse the command line flags."
  (flag:parse-command-line)
  (when flag::*help*
    (flag:print-help)))

You MUST call flag:parse-command-line for the defined command line flags to have non default values.

We also print a help menu  if --help was passed in.

Then we can load our proto if the load-file flag was passed in:

(defmethod hook::at-restart load-proto-cache :after parse-command-line  ()
  "Load the command line specified file at startup."
  (when (string/= flag::*load-file* "")
    (load-state-from-file :filename flag::*load-file*)))                                                                                            

We see an :after clause in our defmethod. We want the load-proto-cache method called during start-up but after we have parsed the command line so flag::*load-file* has been properly set. 

Note: The defmethod here uses a special defmethod syntax added in ace.core.hook. Please see the hook-method documentation for complete details.

Finally we save our image state at exit:

(defmethod hook::at-exit save-proto-cache ()
  "Save the command line specified file at exit."
  (when (string/= flag::*save-file* "")
    (save-state-to-file :filename flag::*save-file*)))

The attentive reader will notice our main function never explicitly called any of these hook functions...

Proto-cache.asd:

We add code to build an executable using asdf:

(defpackage :proto-cache ...
  :build-operation "program-op"
  :build-pathname "proto-cache"
  :entry-point "proto-cache:main")

This is a program-op. The executable pathname is relative, we save the binary as "proto-cache" in the same directory as our proto-cache code. The entry point function is proto-cache:main.

We may then call: 

sbcl --eval "(asdf:operate :build-op :proto-cache)" 

at the command line to create our binary.

Running our binary:

With our binary built we can call:

./proto-cache  --save-file /tmp/first.proto --new-subscriber http://www.google.com

Trying cat /tmp/first.pb:

pika'
http://www.google.com
a?pika"chujg

These are serialized values so one shouldn't try to understand the output so much. We can see "http://www.google.com", "pika", and "chu" are all saved.

Calling

./proto-cache   --load-file /tmp/first.pb --save-file /tmp/first.pb --new-subscriber http://www.altavista.com

And then cat /tmp/first.pb:

I
pikaA
?http://www.altavista.com
http://www.google.com
a?pika"chujg
"

Finally calling  ./proto-cache  --help

We get:

Flags from ace.flag:

    --lisp-global-flags
     (When provided, allows specifying global and special variables as a flag on the command line.
       The values are NIL - for none, :external - for package external, and T - for all flags.)
     Type: ACE.FLAG::GLOBAL-FLAGS

    --help (Whether to print help) Type: BOOLEAN Value: T

    --load-file (Determines the file to load PROTO-CACHE from on startup)
     Type: STRING
     Value: ""

    --new-subscriber (URL for a new subscriber, just for testing)
     Type: STRING
     Value: ""

    --lisp-normalize-flags
     (When non-nil the parsed flags will be transformed into a normalized form.
       The normalized form contains hyphens in place of underscores, trims '*' characters,
       and puts the name into lower case for flags names longer than one character.)
     Type: BOOLEAN

    --save-file (Determines the file to save PROTO-CACHE from on shutdown)
     Type: STRING
     Value: ""

This shows our provided documentation of the command line flags as expected.

Conclusions:

Today we added command line flags, load and exit hooks, and made our application buildable as an executable. We can build our executable and distribute it as we see fit. We can direct it to load and save the application state to user specified files without updating the code. There is still much to do before it’s done but this is slowly becoming a usable application.

There are a few additions I would like to make, but I have a second child coming soon. This may (or may not) be my last technical blog post for quite some time. I hope this sequence of Proto Cache posts has been useful thus far, and I hope to have more in the future.

Thanks to Ron Gut and Carl Gay for copious edits and comments.

ECL NewsECL 21.2.1 release

· 27 days ago

Dear Community,

We are announcing a new stable ECL release which fixes a number of bugs from the previous release. Changes made include amongst others

  • working generational and precise garbage collector modes
  • support for using precompiled headers to improve compilation speed
  • the bytecompiler correctly implements the ANSI specification for load time forms of literal objects in compiled files
  • fixes for encoding issues when reading in the output of the MSVC compiler
  • issues preventing ECL from compiling on Xcode 12 and running on ARM64 versions of Mac OS have been rectified

More detailed information can be obtained from the CHANGELOG file and git commit logs. We'd like to thank all people who contributed to this release. Some of them are listed here (without any particular order): Paul Ruetz, Karsten Poeck, Eric Timmons, Vladimir Sedach, Dima Pasechnik, Matthias Köppe, Yuri Lensky, Tobias Hansen, Pritam Baral, Marius Gerbershagen and Daniel Kochmański.

This release is available for download in a form of a source code archive (we do not ship prebuilt binaries):

Happy Hacking,
The ECL Developers

Pascal CostanzaThe Slick programming language

· 30 days ago

I'm happy to announce the release of the Slick programming language, an s-expression surface syntax for Go, with some extensions inspired by Common Lisp and Scheme. See the Slick programming language repository for more details.

Max-Gerd RetzlaffuLisp on M5Stack (ESP32)

· 31 days ago

About two years I ago I bought a couple of M5Stack ESP32 computers on a Maker Fair. I made little use of them so far but now I started to put uLisp on them and suddenly it is much more fun.

“Hello uLisp!” on the M5Stack; click for a larger version (323 kB).

Read the whole article.

Jonathan GodboutProto Cache: Saving State

· 33 days ago

Todays Updates:

In our last post we implemented a basic Pub Sub application that stores an Any protocol buffer message and a list of subscribers. When the Any protocol buffer message gets updated we send the new Any message in the body of an http request to all of the subscribers in the subscribe-list. 

Today we will update our service to save all of the state in a protocol buffer message. We will also add functionality to save and load the state of the Proto Cache application. 

Note: Viewing the previous post is highly suggested!

Code Updates:

Note: We use red to denote removed code and green to denote added code.

pub-sub-details.proto

`syntax = proto3`

We will use proto3 syntax. I've yet to find a great reason to choose proto3 over proto2, but I've also yet to find a great reason to choose proto2 over proto3. The biggest reason to choose proto3 over proto2 is that most people use proto3, but the Any proto will store proto2 or proto3 messages regardless.

import “any.proto”

Our users are publishing Any messages to their clients, so we must store them in our application state. This requires us to include the any.proto file in our proto file.

message PubSubDetails

This contains (almost) all of the state needed for the publish subscribe service for one user:

  • repeated string subscriber_list
  • google.protobuf.Any current_message
    • This is the latest Any message that the publisher has stored in the Proto Cache.
  • string username
  • string password
    • For any kind of production use this should be salted and hashed. 

message PubSubDetailsCache

This message contains one entry, a map from a string (which will be a username for a publisher) to a PubSubDetails instance. The attentive reader will notice that we save the username twice, once in the PubSubDetails message and once in the PubSubDetailsCache map as the key. This will be explained when we discuss changes to the proto-cache.lisp file.

proto-cache.asd

The only difference in proto-cache.asd from all of the other asd files we've seen using protocol buffers is the use of a protocol buffer message in a package different from our current package. That is, any.proto resides in the cl-protobufs package but we are including it in the pub-sub-details.proto file in proto-cache.

To allow the protoc compiler to find the any.proto file we give it a :proto-search-path containing the path to the any.proto file. 


...
    :components
    ((:protobuf-source-file "pub-sub-details"
      :proto-pathname "pub-sub-details.proto"
      :proto-search-path ("../cl-protobufs/google/protobuf/"))
...

Note: We use a relative path: “../cl-protobufs/google/protobuf/”, which may not work for you. Please adjust to reflect your set-up.

We don't need a component in our defsystem to load the any.proto file into our lisp image since it's already loaded by cl-protobufs. We might want to just to recognize the direct dependency of the any.proto file. 

proto-cache.lisp

Defpackage updates:

We are adding new user invokable functionality so we export:

  • save-state-to-file
  • load-state-from-file

local-nicknames:

  • cl-protobufs.pub-sub-details as psd
    • This is merely to save typing. The cl-protobufs.pub-sub-details is the package that contains the functionality derived from pub-sub-details.proto.

Globals:

*cache*: This will be a protocol buffer message containing a hash table with string keys and pub-sub-details messages. 

(defvar *cache* (make-hash-table :test 'equal))
(defvar *cache* (psd:make-pub-sub-details-cache))

*mutex-for-pub-sub-details*: Protocol buffer messages can't store lisp mutexes. Instead, we store the mutex for a pub-sub-details in a new hash-table with string (username) keys.

make-pub-sub-details:

This function makes a psd:pub-sub-details protocol buffer message. It's almost the same as the previous iteration of pub-sub-details except for the addition of username.


...
  (make-instance 'pub-sub-details :password password))
  (psd:make-pub-sub-details :username username
                            :password password
                            :current-any (google:make-any))
...

(defmethod (setf psd:current-any) (new-value (psd psd:pub-sub-details))

This is really a family of functions:

  • :around: When someone tries to set the current-message value on a pub-sub-details struct we want to write-protect the pub-sub-details entry. We use an around method which activates before any call to the psd:current-any setter. Here we take the username from the pub-sub-details message and write-hold the corresponding mutex in the *mutex-for-pub-sub-details* global hash-table. Then we call call-next-method which will call the main (setf current-any) method.
(defmethod (setf current-any) (new-value (psd pub-sub-details))
(defmethod (setf psd:current-any) :around (new-value (psd psd:pub-sub-details))
  • (setf psd:current-any): This is the actual defmethod defined in cl-protobufs.pub-sub-details. It sets the current-messaeg slot on the message struct.
  • :after: This occurs after the current-any setter was called. We send an http call to all of the subscribers on the pub-sub-details subscriber list. Minus the addition of the psd package prefix to accessor functions of pub-sub-details this function wasn't changed.

 register-publisher:

The main differences between the last iteration of proto-cache and this one are:

  1. This *-gethash method is exported by cl-protobufs.pub-sub-details so the user can call gethash on the hash-table in a map field of a protocol buffer message.
    • (gethash username *cache*)
    • (psd:pub-sub-cache-gethash username *cache*)
  2. We add a mutex to the *mutex-for-pub-sub-details* hash-table with the key being the username string sent to register-publisher.
  3. We return t if the new user was registered successfully, nil otherwise.

register-subscriber and update-publisher-any:

  1. The main difference here is:
    1. (gethash publisher *cache*)
    2. (psd:pub-sub-cache-gethash publisher *cache*)
  2. We have to use the psd package prefix to all of the accessors to pub-sub-details

save-state-to-file:

(defun save-state-to-file (&key (filename "/tmp/proto-cache.txt"))
  "Save the current state of the proto cache to *cache* global
   to FILENAME as a serialized protocol buffer message."
  (act:with-frmutex-read (*cache-mutex*)
    (with-open-file (stream filename :direction :output
                                     :element-type '(unsigned-byte 8))
      (cl-protobufs:serialize-to-stream stream *cache*))))

This is a function that accepts a filename as a string, opens the file for output, and calls cl-protobufs:serialize-to-stream. This is all we need to do to save the state of our applications!

load-state-from-file:

We need to do three things:

  1. Open a file for reading and deserialize the Proto Cache state saved by save-sate-to-file
  2. Create a new map containing the mutexes for each username.
  3. Set the new state into the *cache* global and the new mutex hash-table in *mutex-for-pub-sub-details*.
    1. We do write-hold the *cache-mutex* but I would suggest only loading the saved state when Proto Cache is started.
(defun load-state-from-file (&key (filename "/tmp/proto-cache.txt"))                                                                                   
  "Load the saved *cache* globals from FILENAME. Also creates                                                                                          
   all of the fr-mutexes that should be in *mutex-for-pub-sub-details*."
  (let ((new-cache
          (with-open-file (stream filename :element-type '(unsigned-byte 8))
            (cl-protobufs:deserialize-from-stream
              'psd:pub-sub-details-cache :stream stream)))
        (new-mutex-for-pub-sub-details (make-hash-table :test 'equal)))
    (loop for key being the hash-keys of (psd:pub-sub-cache new-cache)
          do
             (setf (gethash key new-mutex-for-pub-sub-details)
                   (act:make-frmutex)))
    (act:with-frmutex-write (*cache-mutex*)
      (setf *mutex-for-pub-sub-details* new-mutex-for-pub-sub-details
            *cache* new-cache))))

Conclusion:

The main update we made today was defining pub-sub-details in a .proto file instead of a Common Lisp defclass form. The biggest downside is the requirement to save the pub-sub-details mutex in a separate hash-table. For this cost, we:

  1. Gained the ability to save our application state with one call to cl-protobufs:serialize-to-stream.
  2. Gained the ability to load our application with little more then one call to cl-protobufs:deserialize-from-stream.

We were also able to utilize the setf methods defined in cl-protobufs to create :around and :after methods.

Note: Nearly all services will be amenable to storing their state in protocol buffer messages.

I hope the reader has gained some insight into how they can use cl-protobufs in their application even if their application doesn't make http-requests. Being able to save the state of a running program and load it for later use is very important in most applications, and protocol buffers make this task simple.

Thank you for reading!

Thanks to Ron, Carl, and Ben for edits!

Quicklisp newsJanuary 2021 Quicklisp dist update now available

· 34 days ago

 New projects

  • astonish — A small library for querying and manipulating Lisp ASTs — GPLv3
  • cl-dejavu — Repack of DejaVu fonts for Common Lisp — CC0-1.0 (fonts have a separate license)
  • cl-fxml — cl-fxml: Common Lisp - Finally eXtended Markup Language. — MIT
  • cl-zstd — Zstandard (de)compression using bindings to libzstd — GPL-3
  • clog — The Common Lisp Omnificent GUI — BSD
  • core — Make Interactive-Server-Side-Rendered web pages with declaritive and recursive programming. This is the core functionality is reusable for all server modules. — LLGPL
  • definer — A DEF macro for Common Lisp. The DEFINER library adds a simple macro DEF to Common Lisp that replaces the various 'def*' forms in the language. It is a simple hack, but it adds some elegance to the language. Of course, it comes with its own way to be extended. — BSD
  • hunchenissr — Make Interactive-Server-Side-Rendered web pages with declaritive and recursive programming. — LLGPL
  • inheriting-readers — Provides a simple yet powerful value inheritance scheme. — Unlicense
  • mailgun — A thin wrapper to post HTML emails through mailgun.com — Unlicense
  • portal — Portable websockets. — LLGPL
  • quicklisp-stats — Fetches and operates on Quicklisp download statistics. — MIT
  • shared-preferences — Notably allows flexible specification of package-local preferences. — Unlicense
  • tclcs-code — Companion code for "The Common Lisp Condition System" — MIT
  • unicly — UUID Generation per RFC 4122 — MIT
  • wallstreetflets — Wall Street FLETs: A library for calculating Options Greeks — GPL v3

Updated projects: access, algae, anaphora, anypool, april, architecture.builder-protocol, atomics, bp, cepl, chanl, cl+ssl, cl-ansi-text, cl-collider, cl-data-structures, cl-fad, cl-fastcgi, cl-gobject-introspection, cl-gserver, cl-ipfs-api2, cl-kraken, cl-liballegro-nuklear, cl-marklogic, cl-mixed, cl-mssql, cl-ssdb, cl-str, cl-telegram-bot, cl-unicode, cl-utils, cl-wave-file-writer, cl-webkit, clack-pretend, clast, clath, closer-mop, cmd, coleslaw, common-lisp-jupyter, concrete-syntax-tree, conium, croatoan, cytoscape-clj, damn-fast-priority-queue, data-lens, djula, easy-audio, fast-generic-functions, flac-metadata, functional-trees, fuzzy-match, garbage-pools, gendl, golden-utils, graph, gtwiwtg, harmony, helambdap, house, ironclad, kekule-clj, lichat-protocol, local-time, markup, math, mcclim, mgl-pax, mutility, named-readtables, nibbles, numpy-file-format, ook, opticl, origin, parachute, paren6, parsley, patchwork, petalisp, phoe-toolbox, picl, plump, pngload, portable-condition-system, postmodern, qlot, read-number, rpcq, sb-cga, sb-fastcgi, sel, select, serapeum, shadow, sheeple, sly, stripe, terminfo, trace-db, trivia, trivial-gray-streams, trivial-mmap, trucler, uax-15, ucons, umbra, uncursed, unix-opts, varjo, vgplot, with-contexts, xhtmlambda, zpb-exif, zpb-ttf.

Removed projects: flac-parser, gamebox-dgen, gamebox-ecs, gamebox-frame-manager, genie, shorty, simple-logger.

To get this update, use (ql:update-dist "quicklisp"). Enjoy!

Max-Gerd Retzlaff20 years' anniversary for SBCL

· 35 days ago

Max-Gerd RetzlaffMotto: Our LISP is COMMON LISP

· 35 days ago

Our LISP is COMMON LISP, from:  Lisp 2nd edition by Winston and Horn, Addison Wesley, 1984; click for a larger version (198 kB).

Excerpt from: Lisp 2nd edition by Winston and Horn, Addison Wesley, 1984.


(Moving content from Twitter: originally posted on Dec 13, 2019, 4:04 PM.)

Read the whole article.

Eric TimmonsCommon Lisp Docker Images

· 35 days ago

As alluded to in my previous post, I've been working on a set of Docker images for Common Lisp. The latest version of that effort is now finally live! Check it out at https://common-lisp.net/project/cl-docker-images/. Many thanks are also due to the Common Lisp Foundation for hosting the images on their Docker Hub org!

I've been building Docker images for Common Lisp going on five years now. They were originally hosted on my personal account (daewok) on both Github and Docker Hub. If you use the daewok images, please migrate over to these new ones at your earliest convenience. I plan to stop updating the daewok images later this year.

My original images were... not super good. The images were too big, used the entrypoint incorrectly, had out of date pieces, etc. But over the years I finally grokked Docker best practices and (IMO) they are now a set of high quality images.

I use them regularly for CI/CD purposes. They work very well with Gitlab CI and I also use them locally to make repeatable builds of CL software. I also know that they're in use by others in the community (largely for CI from what I can tell).

There are several other CL related Docker images out there. But, as far as I can tell, none of them are updated as regularly as these, support as many implementations, nor run on as many CPU architectures.

As part of this project, there are currently two classes of images published: "implementation specific" and "development".

The "implementation specific" images are meant to include only a single CL implementation. The default variant for these images is "fat" and has many commonly used OS packages and programs. They are mostly built off the buildpack-deps image (a common base image for programming lanugagues). There are also "slim" variants that include only the CL implementation.

The "development" image is geared toward local interactive development. It is a kitchen-sink image that includes every implementation the project builds an implementation specific image for, as well as all the OS packages needed to load ~every system in Quicklisp, the latest version of ASDF, etc.

The last piece of this project is slime-docker, an Emacs package that automates the starting of CL processes in Docker containers for use with SLIME.

If you'd like to see improvements or additions, would like another implementation to be supported, or have an idea for another category of image, please join in on this project!

Max-Gerd RetzlaffOn the different versions of the<br /> ANSI Common Lisp Standard X3.226-1994

· 37 days ago

The Common Lisp standard has had different names and annual details suffixes. The standard was developed by the Technical Committee X3J13 of the “Accredited Standards Committee X3, Information Technology” founded and accredited by the American National Standards Institute (ANSI) in 1961.

In one of my professional lives I am working in the “Accredited Standards Committee X12” (also written as “ASC X12” or just “X12”; founded in 1979) since just shy of four years now and was voted “TAS Representative” of “Subcommittee X12C – Communications and Controls” (local copy) in January 2018, that is, I represent X12C in “Subcommittee X12J – Technical Assessment” (local copy), short “TAS”, since then. Having that background, I might be able to shed a light on the different versions of the Common Lisp standard and on how standardization committees' work looks in general.

I have to admit that one of the reasons I liked to start working in X12 is that I knew that the Common Lisp standard was developed by X3. I did now know how standards were developed and how such a committee worked, and I was curious to change that.

When you search in the ANSI Webstore you'll find different versions:

The standard was originally published (or, more precisely, approved for publication) in 1994, so the first version is just called “ANSI X3.226-1994”. The date of approval was December 8, 1994.

Read the whole article.

Max-Gerd RetzlaffOriginal IBM 3-ring binders

· 37 days ago

Follow-up to Physical dimensions of ANSI Common Lisp: It just took two days of searching and I got these three beautiful 25+ year old IBM 3-ring binders. How nice to live in Berlin!

ANSI Common Lisp in IBM 3-ring binders; click for a larger version (228 kB).

Ah, yes, much better:

ANSI Common Lisp in IBM 3-ring binders; click for a larger version (247 kB).

And yes, if you don't want them full to the brim you need three binders for the ANSI Common Lisp standard.

Read the whole article.

Jonathan GodboutProto Cache: Implementing Basic Pub Sub

· 39 days ago

Today's Updates

In our last post we saw some of the features of the ace.core.defun and ace.core.thread libraries by creating a thread-safe cache of the Any protocol buffer object. Today we are going to update the proto-cache repository to implement publisher/subscriber features. This will allow a publisher to publish a feed of Any messages and a subscriber to subscribe to such a  feed. 

It is expected (but not required) that the reader has read the previous post Proto Cache: A Caching Story. That post details some of the functions and objects you will see in today’s code.

Note: This is a basic implementation, not one ready for production use. This will serve as our working project going forward.

Code Updates

Proto-cache.asd

We want subscribers to be able to get new versions of an Any protocol buffer message. On the web, the usual way to receive messages is over HTTP. We use the Drakma HTTP client. You can see we added :drakma to the depends-on list in the defsystem.

Proto-cache.lisp

There are three major regions to this code. The first region is the global objects that make up the cache. The second is the definition of a new class, pub-sub-details. Finally the actual publisher-subscriber functions are at the bottom of the page.

Global objects:

The global objects section looks much like it did in our previous post. We update the *cache* hash-table to use equal as its test function and we are going to make the keys to this cache be username strings.

Pub-sub-details class:

The global objects section looks much like it did in our previous post. We update the *cache* hash-table to use equal as its test function and we are going to make the keys to this cache be username strings.

The pub-sub-details class contains the data we need to keep track of the publisher and subscriber features:

  • subscriber-list: This will be a list of the HTTP endpoints to send the Any messages to after the Any message is updated. Currently, we only allow for an HTTP message string. Future implementations should allow for security functionality on those endpoints.
  • current-any: The current Any message that the publisher has supplied.
  • mutex: A fr-mutex to protect the current-any slot. This should be read-held to get the current-any and it should be read-held to set a new current-any message.
  • password: The password for the subscriber held as a string. 

We shouldn't be saving the password being as a string in the pub-sub-details class. At a minimum we should be salting and hashing this value. In the future we should implement an account system for readers and subscribers giving access to reading and updating the pub-sub-details. As this is only instructional and not production-ready code, I feel okay leaving it as is for the moment.

We create a make-pub-sub-details function that will create a pub-sub-details object with a given password. The register function doesn't allow the user to set an Any message at creation time, and none of the other slots are useful to the publisher.

We create an accessor method to set the any-message value slot. We also create an :after method to send the Any message to any listening subscribers by iterating through the subscriber list and calling a drakma:http-request. We wrap this in unwind-protect so an IO failure doesn't stop other subscribers from getting the message.

Finally we add a setter function for the subscriber list.

Function definitions:

Register-publisher:

This function is the registration point for a new publisher. It is almost the same as set-in-cache from our previous post except it checks that an entry in the cache for the soon-to-be-registered publisher doesn't already exist. It would be bad to let a new publisher overwrite an existing publisher.

Register-subscriber:

Here we use a new macro, ace.core.etc:clet from the etc package in ace.core.

(defun register-subscriber (publisher address)
  "Register a new subscriber to a publisher."
  (ace:clet ((ps-struct
               (act:with-frmutex-read (*cache-mutex*)
                 (gethash publisher *cache*)))
             (ps-mutex (mutex ps-struct)))
    (act:with-frmutex-write (ps-mutex)
      (push address (subscriber-list ps-struct)))))

In the code below we search the cache for a user entry, if the entry is found then ps-struct will be non-nil and we can evaluate the body adding the subscriber to the list. If the subscriber is not found we return nil.

Update-publisher-any:

(defun update-publisher-any (username password any)
  "Updates the google:any message for a publisher
   with a specified username and password.
   The actual subscriber calls happen in a separate thread
   but 'T is returned to the user to indicate the any
   was truly updated."
  (ace:clet ((ps-class
              (act:with-frmutex-read (*cache-mutex*)
                (gethash username *cache*)))
             (correct-password (string= (password ps-class)
                                        password)))
    (declare (ignore correct-password))
    (act:make-thread
     (lambda (ps-class)
       (setf (current-any ps-class) any))
     :arguments (list ps-class))
    t))

In the update-publisher-any code we use clet to verify that the publisher exists and the password is found. We ignore the correct-password entry though.

We don't want the publisher to be thread-blocked while we send the new message to all of the subscribers so we update the current-any in a separate thread. To do this we use the ace.core.thread function make-thread. A keen reader will see for SBCL this calls the sbcl make-thread function, otherwise it calls bordeaux-threads make-thread function.

If we are able to find a publisher with the correct password we return T to show success.

Conclusion

In today’s post we have made a basic publisher-subscriber library that will send an Any protocol buffer message to a list of subscribers. We have detailed some new functions that we used in ace.core. We have also listed some of the problems with this library. The code has evolved substantially from the previous post but it still has a long way to go before being production-ready.

Thank you for your reading!


Ron Gut, Carl Gay, and Ben Kuehnert gave comments and edits to this post.

Tycho Garen Learning Common Lisp Again

· 41 days ago

In a recent post I spoke about abandoning a previous project that had gone off the rails, and I've been doing more work in Common Lisp, and I wanted to report a bit more, with some recent developments. There's a lot of writing about learning to program for the first time, and a fair amount of writing about lisp itself, neither are particularly relevant to me, and I suspect there may be others who might find themselves in a similar position in the future.

My Starting Point

I already know how to program, and have a decent understanding of how to build and connect software components. I've been writing a lot of Go (Lang) for the last 4 years, and wrote rather a lot of Python before that. I'm an emacs user, and I use a Common Lisp window manager, so I've always found myself writing little bits of lisp here and there, but it never quite felt like I could do anything of consequence in Lisp, despite thinking that Lisp is really cool and that I wanted to write more.

My goals and rational are reasonably simple:

  • I'm always building little tools to support the way that I use computers, nothing is particularly complex, but it'd enjoy being able to do this in CL rather than in other languages, mostly because I think it'd be nice to not do that in the same languages that I work in professionally. [1]
  • Common Lisp is really cool, and I think it'd be good if it were more widely used, and I think by writing more of it and writing posts like this is probably the best way to make that happen.
  • Learning new things is always good, and I think having a personal project to learn something new will be a good way of stretching my self as a developer. Most of my development as a programmer has focused on
  • Common Lisp has a bunch of features that I really like in a programming language: real threads, easy to run/produce static binaries, (almost) reasonable encapsulation/isolation features.

On Learning

Knowing how to program makes learning how to program easier: broadly speaking programming languages are similar to each other, and if you have a good model for the kinds of constructs and abstractions that are common in software, then learning a new language is just about learning the new syntax and learning a bit more about new idioms and figuring out how different language features can make it easier to solve problems that have been difficult in other languages.

In a lot of ways, if you already feel confident and fluent in a programming language, learning a second language, is really about teaching yourself how to learn a new language, which you can then apply to all future languages as needed.

Except realistically, "third languages" aren't super common: it's hard to get to the same level of fluency that you have with earlier languages, and often we learn "third-and-later" languages are learned in the context of some existing code base or project4, so it's hard to generalize our familiarity outside of that context.

It's also the case that it's often pretty easy to learn a language enough to be able to perform common or familiar tasks, but fluency is hard, particularly in different idioms. Using CL as an excuse to do kinds of programming that I have more limited experience with: web programming, GUI programming, using different kinds of databases.

My usual method for learning a new programming language is to write a program of moderate complexity and size but in a problem space that I know pretty well. This makes it possible to gain familiarity, and map concepts that I understand to new concepts, while working on a well understood project. In short, I'm left to focus exclusively on "how do I do this?" type-problems and not "is this possible," or "what should I do?" type-problems.

Conclusion

The more I think about it, the more I realize that when we talk about "knowing a programming language," inevitably linked to a specific kind of programming: the kind of Lisp that I've been writing has skewed toward the object oriented end of the lisp spectrum with less functional bits than perhaps average. I'm also still a bit green when it comes to macros.

There are kinds of programs that I don't really have much experience writing:

  • GUI things,
  • the front-half of the web stack, [2]
  • processing/working with ASTs, (lint tools, etc.)
  • lower-level kind of runtime implementation.

There's lots of new things to learn, and new areas to explore!

Notes

[1]There are a few reasons for this. Mostly, I think in a lot of cases, it's right to choose programming languages that are well known (Python, Java+JVM friends, and JavaScript), easy to learn (Go), and fit in with existing ecosystems (which vary a bit by domain,) so while it might the be right choice it's a bit limiting. It's also the case that putting some boundaries/context switching between personal projects and work projects could be helpful in improving quality of life.
[2]Because it's 2020, I've done a lot of work on "web apps," but most of my work has been focused on areas of applications including including data layer, application architecture, and core business logic, and reliability/observability areas, and less with anything material to rendering web-pages. Most projects have a lot of work to be done, and I have no real regrets, but it does mean there's plenty to learn. I wrote an earlier post about the problems of the concept of "full-stack engineering" which feels relevant.

Jonathan GodboutProto Cache: A Caching Story

· 46 days ago

What is Proto-Cache?

I've been working internally at Google to open source several libraries including cl-protobufs and a series of utility libraries we call "ace". I wrote several blog posts making an HTTP server that takes in either protocol buffers or JSON strings and responds in kind. I think I have worked enough on Mortgage Server and wish to work on a different project.

Proto-cache will grow up to be a pub-sub system that takes in google.protobuf:any protos and send them to users over http requests. I'm developing it to showcase the ace.core library and the Any proto well-known-type. In this post we create a cache system which stores google.protobuf.any messages in a hash-table keyed off of a symbol.

The current incarnation of Proto Cache:

The code can be found here: https://github.com/Slids/proto-cache

Proto-cache.asd:

This is remarkable in-as-much as cl-protobufs isn't required for the defsystem! It's not required at all, but we do require the cl-protobufs.google.protobuf:any protocol buffer message object. Right now we are only adding and getting it from the cache. This allows us to store a protocol buffer message object that any user system can parse by calling unpack-any. We never have to understand the message inside.

Proto-cache.lisp:

The actual implementation. We give three different functions:

  • get-from-cache
  • set-in-cache
  • remove-from-cache

We also have a:

  • fast-read mutex
  • hash-table

Note: The ace.core library can be found at: https://github.com/cybersurf/ace.core

Fast-read mutex (fr-mutex):

The first interesting thing to note is the fast-read mutex. This can be found in the ace.core.thread package included in the ace.core utility library. This allows for mutex free reads of a protected region of code. One has to call:

  • (with-frmutex-read (fr-mutex) body)
  • (with-frmutex-write (fr-mutex) body)

If the body of with-frmutex-read is finished with nobody calling with-frmutex-write then the value is returned. If someone calls with-frmutex-write while another thread is in with-frmutex-read then the body of with-frmutex-read has to be re-run. One should be careful to not modify state in the with-frmutex-read body.

Discussion About the Individual Functions

get-from-cache:

(acd:defun* get-from-cache (key)
  "Get the any message from cache with KEY."
  (declare (acd:self (symbol) google:any))
  (act:with-frmutex-read (cache-mutex)
    (gethash key cache)))


This function uses the defun* form from ace.core.defun. It looks the same as a standard defun except has a new declare statement. The declare statement takes the form

(declare (acd:self (lambda-list-type-declarations) output-declaration))

In this function we state that the input KEY must be a symbol and the return value is going to be a google:any protobuf message. The output declaration is optional. For all of the options please see the macro definition for ace.core.defun:defun*.

The with-fr-mutex-read macro is also being used.

Note in the macro's body we only do a simple accessor call into a hash-table. Safety is not guaranteed, only consistency.

set-in-cache:

(acd:defun* set-in-cache (key any)
  "Set the ANY message in cache with KEY."
  (declare (acd:self (symbol google:any) google:any))
  (act:with-frmutex-write (cache-mutex)
    (setf (gethash key cache) any)))

We see that the new defun* call is used. In this case we have two inputs, KEY will be a symbol ANY will be a google:any proto message. We also see that we will return a google:any proto message.

The with-frmutex-write macro is being used. The only thing that is done in the body is setting a cache value. If we try to get a message from the cache and set a message into the cache, it is possible a reader will have to read multiple times. In systems where readers are more common than writers fr-mutexes and spinlocking are much faster than having readers lock a mutex for every read..

remove-from-cache:

We omit this function in this write-up for brevity.

Conclusion:

Fast-read mutexes like the one found in ace.core.thread are incredibly useful tools. Having to access a mutex can be slow even in cases where that mutex is never locked. I believe this is one of the more useful additions in the ace.core library.

The new defun* macro found in ace.core.defun for creating function definitions is more mixed. I find a lack of clarity in mapping the lambda list s-expression in the defun statement to the s-expression in the declaration. Others may find it provides nicer syntax and the clarity is more obvious.

Future posts will show the use of the any protocol buffer message.

As usual Carl Gay gave copious edits and suggestions.

Eric TimmonsStatic Executables with SBCL

· 54 days ago

UPDATE: Make sure to see my follow up post to this).

Common Lisp is an amazing language with many great implementations. The image based development paradigm vastly increases developer productivity and enjoyment. However, there frequently comes a time in a program's life cycle where development pauses and a version must be delivered for use by non-developers. There are many tools available to build an executable in Common Lisp, most of which follow the theme of "construct a Lisp image in memory, then dump it to disk for later reloading". That being said, none of the existing methods fit 100% of my use cases, so this post is dedicated to documenting how I filled the gap by convincing SBCL to generate completely static executables.

Background

There are a variety of reasons to want static executables, but the most common ones I run into personally are:

  1. I want to archive my executables. I want to have a version of my executables saved that I can dig up at any point in the future, long after I've upgraded my OS (multiple times), and run for benchmarking purposes, to test if old versions exhibited specific behavior, etc. without needing to recompile.
  2. I want to enable someone to reproduce my results exactly. This is important for reproducibility in academic contexts. Also, some computing contests that conferences organize prefer static executables so they can run tests on their hardware without needing to set up a complicated run time environment.
  3. I want to make it trivial for someone to install my software. With a static executable, all anyone running on Linux needs to do is download a single file, chmod +x it, and copy it onto their path (preferably after verifying its integrity, but, let's be honest, fewer people do that than should).

There certainly are issues with static executables/linking in general. If you are unaware of what they are, I highly encourage you to read up on the subject before deciding that static executables are the be-all-end-all of application delivery. Static executables are just another tool in a developer's toolbox, to be pulled out only when the time is right.

I'll pause at the moment for a clarification: when I say static executable I mean a truly static executable. As in I want to be able to run ldd on it and have it output not a dynamic executable and I do not want it to call any libdl functions (such as dlopen or dlsym at runtime). While some existing methods claim or imply that they make static executables with SBCL (such as CFFI's static-program-op or manually linking external libraries into the SBCL runtime while building it), they by and large mean they statically link foreign code into the runtime, but the runtime itself is not a static executable.

I have yet to find a publicly documented method of creating a fully static executable with SBCL and it's not too hard to understand why. Creating a static executable requires statically linking in libc and the most common libc implementation for Linux (glibc) does a half-assed job at statically linking itself. While it is possible, many functions will cause your "static" executable to dynamically load pieces of glibc behind your back. Except now you have the requirement that the runtime version must match the compiled version exactly. That defeats the whole point of having a static executable!

For that reason, musl libc is commonly used when creating a truly static executable is important. Unfortunately, musl is not 100% compatible with glibc and for a while SBCL would not work with it. There have been various efforts at patching SBCL to run with musl libc throughout the years, but the assorted (minor!) changes finally got merged upstream in SBCL 2.0.5. This laid the groundwork necessary for truly static executables with SBCL.

Patches

Enough with the blabber, show me the code!

I am maintaining a fork of SBCL that contains the necessary patches. There is a static-executable branch which will always contain the latest version. I plan to rebase this branch on new SBCL releases or on top of upstream's master branch if it looks like I'm going to need to do some extra legwork for an upcoming release. There will also be a series of branches named static-executable-$VERSION which have my patches applied on top of the named version, starting with SBCL 2.1.0.

The patch for any SBCL release is also located at https://www.timmons.dev/static/patches/sbcl/$VERSION/static-executable-support.patch. There is a detached signature available at https://www.timmons.dev/static/patches/sbcl/$VERSION/static-executable-support.patch.asc signed with GPG key 0x9ACF6934.

I would love to get these patches upstreamed, but they didn't get much traction the last time I submitted them to sbcl-devel. Admittedly, they were an early, less elegant version that hadn't seen much use in the real-world. My hope is that other people who desire this capability from SBCL will collaborate to test and refine these patches over time for eventual upstreaming.

Quickstart

Given that most people aren't using musl libc on their development computer, the quickest, easiest way to get a static executable is to build one with Docker. After getting the patchset, simply run the following set of commands in the root of the SBCL repo. This will use the clfoundation/sbcl:alpine3.12 Docker image (another project of mine for a future post) to build a static executable and then copy it out of the image to your host's file system.

docker build -t sbcl-static-executable -f tools-for-build/Dockerfile.static-executable-example .
docker create --name sbcl-static-executable-extractor sbcl-static-executable
docker cp sbcl-static-executable-extractor:/tmp/sb-gmp-tester /tmp/sb-gmp-tester
docker rm sbcl-static-executable-extractor

You should now be able to examine /tmp/sb-gmp-tester to see that it is a static executable:

$ ldd /tmp/sb-gmp-tester
     not a dynamic executable

If all goes well, you should also be able to run it, see the sb-gmp contrib tests all pass (fingers crossed), and realize that this worked because libc, the SBCL runtime, and libgmp were all statically linked!

The file README.static-executable (after applying the patchset) has an example of building locally and a set of docker commands that doesn't require tagging images and naming containers.

How does it work??

This approach requires that the target image be be built twice: once to record the necessary foreign symbols, and then again with the newly built static runtime. I can, however, envision ways around this for a sufficiently motivated person.

One way could be to modify the (already in-tree) shrinkwrapping recipe to handle libdl not being available at runtime. I abandoned this approach largely because the shrinkwrapping code is written for x86-64 and does a lot of things with assembly (which I do not know). It is important for me to have static executables on ARM as well. A second way could be to patch out or otherwise improve the check that the runtime version used to build the core matches the runtime version used to run it. I didn't go this approach as it would certainly lead to difficult to debug issues if used incorrectly, plus the Lisp code in the core would need to check the presence/usefulness of libdl functions at runtime.

So, how does this patchset work and why does it require two passes? Apologies to the SBCL devs if I completely butcher the explanation of SBCL internals, but here it goes anyways!

Lisp code routinely calls into C code, whether it is to a runtime provided function, a libc function, or another library the user has linked and defined using the sb-alien package or the portable counterparts in CFFI. In order to mediate these calls from the Lisp side, SBCL maintains a linkage table. This table has two components. First is a Lisp-side hash table that maps foreign names (and an indicator of if it is data or a function) to an integer. The second is a C-side vector that contains either the address of the symbol (in the case of data) or the opcodes necessary to call the function (e.g., by JMPing to its address).

The C-side vector is populated by looking up the symbol's address using dlsym. This lookup generally happens under two possible scenarios. First, when the Lisp code defines a foreign symbol it wants to be able to call or read. Second, every time the runtime starts, it populates the C-side entries for every symbol contained in the core's hash-table. This second case is how SBCL handles the dynamic linker changing the address of symbols in between core dumps.

This reliance on dlopen and dlsym is so baked into SBCL at this point that, even though the code is nominally conditioned on the internal feature :os-provides-dlopen, I was unable to build a working SBCL without it (before these patches, of course).

With these patches, you first build your Lisp image that you want to deliver like normal. Then, you load the file tools-for-build/dump-linkage-info.lisp into it. Next, you call sb-dump-linkage-info:dump-to-file to extract the Lisp side linkage table entries into a separate file (filtered to remove functions from libdl). Once you have this file, you rebuild SBCL, this time with the intention of creating a static runtime. To do this, you should provide the following:

  • The environment variable LINKFLAGS should contain -no-pie -static in order to build the static runtime.
  • Any additional libraries you need should be specified using the environment variable LDLIBS.
  • You probably want to set the environment variable IGNORE_CONTRIB_FAILURES to yes.
  • You need to pass the file containing the linkage table entries to make.sh using the --extra-linkage-table-entries argument.
  • Build without the :os-provides-dlopen and :os-provides-dladdr features. One way of doing this is to pass --without-os-provides-dlopen and --without-os-provides-dladdr to make.sh.

During the build process, the contents of the --extra-linkage-table-entries file are inserted into the cold SBCL core during second genesis and a C file is autogenerated containing a single function that populates the C side of the linkage table using the address of every symbol. This C file is the built into the runtime and called while the runtime boots, before it starts executing the core. This means that, if the runtime is a dynamic executable, the system linker will patch up all the references we need at runtime without SBCL needing to call dlsym explicitly. If the runtime is a static executable, then the symbols are statically linked for us and nothing needs to be done at runtime.

Issues

Given how new this approach is, you will certainly run into issues. Many systems that load foreign code will blindly assume that libraries can be linked in at runtime and will fail to work (silently or loudly) if that assumption is not met. Some libraries already have their own homebrew ways of dealing with this. For instance, if the feature :cl+ssl-foreign-libs-already-loaded is present, the cl+ssl system will not attempt to load the libraries. To deal with this issue in a more principled way, I strongly recommend patching systems to use CFFI's (relatively) new canary argument to define-foreign-library.

CFFI itself also has some issues with this arrangement because it dives into some sb-alien internals that simply aren't present on #-os-provides-dlopen. I currently fix this in a kludgy way by commenting out most of %close-foreign-library in src/cffi-sbcl.lisp, but if more people start building static executables, we'll need to come up with a better way of handling it.

Next Steps

I would love to get feedback on this approach and any ideas on how to improve it! I strongly believe that better support for building static executables with SBCL should be upstreamed and I doubt I am alone in that belief. Please drop me a line (etimmons on Freenode or daewok on Github/Gitlab) if you have suggestions.

Personally, I have used earlier iterations of these patches to build static executables for some of my grad school work. My next real deployment of these patches will likely be to build CLPM with them and providing static executables starting with v0.4.

Michał HerdaTIL that Common Lisp dynamic variables can be made locally unbound

· 55 days ago
            ;;; let's first define a global variable...
CL-USER> (defvar *foo* 42)
*FOO*

;;; ...and then make a binding without a value using PROGV
CL-USER> (progv '(*foo*) '() (print *foo*))

debugger invoked on a UNBOUND-VARIABLE in thread
#<THREAD "main thread" RUNNING {1004A684B3}>:
  The variable *FOO* is unbound.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE   ] Retry using *FOO*.
  1: [USE-VALUE  ] Use specified value.
  2: [STORE-VALUE] Set specified value and use it.
  3: [ABORT      ] Exit debugger, returning to top level.

((LAMBDA ()))
   source: (PRINT *FOO*)
0] ; look ma, locally unbound!

          

Timofei ShatrovIchiran@home 2021: the ultimate guide

· 55 days ago

Recently I've been contacted by several people who wanted to use my Japanese text segmenter Ichiran in their own projects. This is not surprising since it's vastly superior to Mecab and similar software, and is occassionally updated with new vocabulary unlike many other segmenters. Ichiran powers ichi.moe which is a very cool webapp that helped literally dozens of people learn Japanese.

A big obstacle towards the adoption of Ichiran is the fact that it's written in Common Lisp and people who want to use it are often unfamiliar with this language. To fix this issue, I'm now providing a way to build Ichiran as a command line utility, which could then be called as a subprocess by scripts in other languages.

This is a master post how to get Ichiran installed and how to use it for people who don't know any Common Lisp at all. I'm providing instructions for Linux (Ubuntu) and Windows, I haven't tested whether it works on other operating systems but it probably should.

PostgreSQL

Ichiran uses a PostgreSQL database as a source for its vocabulary and other things. On Linux install postgresql using your preferred package manager. On Windows use the official installer. You should remember the password for the postgres user, or create a new user if you know how to do it.

Download the latest release of Ichiran database. On the release page there are commands needed to restore the dump. On Windows they don’t really work, instead try to create database and restore the dump using pgAdmin (which is usually installed together with Postgres). Right-click on PostgreSQL/Databases/postgres and select “Query tool…”. Paste the following into Query editor and hit the Execute button.

CREATE DATABASE [database_name]
    WITH TEMPLATE = template0
    OWNER = postgres
    ENCODING = 'UTF8'
    LC_COLLATE = 'Japanese_Japan.932'
    LC_CTYPE = 'Japanese_Japan.932'
    TABLESPACE = pg_default
    CONNECTION LIMIT = -1;

Then refresh the Databases folder and you should see your new database. Right-click on it then select “Restore”, then choose the file that you downloaded (it wants “.backup” extension by default so choose “Format: All files” if you can’t find the file).

You might get a bunch of errors when restoring the dump saying that “user ichiran doesn’t exist”. Just ignore them.

SBCL

Ichiran uses SBCL to run its Common Lisp code. You can download Windows binaries for SBCL 2.0.0 from the official site, and on Linux you can use the package manager, or also use binaries from the official site although they might be incompatible with your operating system.

However you really want the latest version 2.1.0, especially on Windows for uh… reasons. There’s a workaround for Windows 10 though, so if you don’t mind turning on that option, you can stick with SBCL 2.0.0 really.

After installing some version of SBCL (SBCL requires SBCL to compile itself), download the source code of the latest version and let’s get to business.

On Linux it should be easy, just run

sh make.sh --fancy
sudo sh install.sh

in the source directory.

On Windows it’s somewhat harder. Install MSYS2, then run “MSYS2 MinGW 64-bit”.

pacman -S mingw-w64-x86_64-toolchain make

# for paths in MSYS2 replace drive prefix C:/ by /c/ and so on

cd [path_to_sbcl_source]
export PATH="$PATH:[directory_where_sbcl.exe_is_currently]"

# check that you can run sbcl from command line now
# type (sb-ext:quit) to quit sbcl

sh make.sh --fancy

unset SBCL_HOME
INSTALL_ROOT=/c/sbcl sh install.sh

Then edit Windows environment variables so that PATH contains c:\sbcl\bin and SBCL_HOME is c:\sbcl\lib\sbcl (replace c:\sbcl here and in INSTALL_ROOT with another directory if applicable). Check that you can run a normal Windows shell (cmd) and run sbcl from it.

Quicklisp

Quicklisp is a library manager for Common Lisp. You’ll need it to install the dependencies of Ichiran. Download quicklisp.lisp from the official site and run the following command:

sbcl --load /path/to/quicklisp.lisp

In SBCL shell execute the following commands:

(quicklisp-quickstart:install)
(ql:add-to-init-file)
(sb-ext:quit)

This will ensure quicklisp is loaded every time SBCL starts.

Ichiran

Find the directory ~/quicklisp/local-projects (%USERPROFILE%\quicklisp\local-projects on Windows) and git clone Ichiran source code into it. It is possible to place it into an arbitrary directory, but that requires configuring ASDF, while ~/quicklisp/local-projects/ should work out of the box, as should ~/common-lisp/ but I’m not sure about Windows equivalent for this one.

Ichiran wouldn’t load without settings.lisp file which you might notice is absent from the repository. Instead, there’s a settings.lisp.template file. Copy settings.lisp.template to settings.lisp and edit the following values in settings.lisp:

  • *connection* this is the main database connection. It is a list of at least 4 elements: database name, database user (usually “postgres”), database password and database host (“localhost”). It can be followed by options like :port 5434 if the database is running on a non-standard port.
  • *connections* is an optional parameter, if you want to switch between several databases. You can probably ignore it.
  • *jmdict-data* this should be a path to these files from JMdict project. They contain descriptions of parts of speech etc.
  • ignore all the other parameters, they’re only needed for creating the database from scratch

Run sbcl. You should now be able to load Ichiran with

(ql:quickload :ichiran)

On the first run, run the following command. It should also be run after downloading a new database dump and updating Ichiran code, as it fixes various issues with the original JMdict data.

(ichiran/mnt:add-errata)

Run the test suite with

(ichiran/test:run-all-tests)

If not all tests pass, you did something wrong! If none of the tests pass, check that you configured the database connection correctly. If all tests pass, you have a working installation of Ichiran. Congratulations!

Some commands that can be used in Ichiran:

  • (ichiran:romanize "一覧は最高だぞ" :with-info t) this is basically a text-only equivalent of ichi.moe, everyone’s favorite webapp based on Ichiran.
  • (ichiran/dict:simple-segment "一覧は最高だぞ") returns a list of WORD-INFO objects which contain a lot of interesting data which is available through “accessor functions”. For example (mapcar 'ichiran/dict:word-info-text (ichiran/dict:simple-segment "一覧は最高だぞ") will return a list of separate words in a sentence.
  • (ichiran/dict:dict-segment "一覧は最高だぞ" :limit 5) like simple-segment but returns top 5 segmentations.
  • (ichiran/dict:word-info-from-text "一覧") gets a WORD-INFO object for a specific word.
  • ichiran/dict:word-info-str converts a WORD-INFO object to a human-readable string.
  • ichiran/dict:word-info-gloss-json converts a WORD-INFO object into a “json” “object” containing dictionary information about a word, which is not really JSON but an equivalent Lisp representation of it. But, it can be converted into a real JSON string with jsown:to-json function. Putting it all together, the following code will convert the word 一覧 into a JSON string:
(jsown:to-json
 (ichiran/dict:word-info-json
  (ichiran/dict:word-info-from-text "一覧")))

Now, if you’re not familiar with Common Lisp all this stuff might seem confusing. Which is where ichiran-cli comes in, a brand new Command Line Interface to Ichiran.

ichiran-cli

ichiran-cli is just a simple command-line application that can be called by scripts just like mecab and its ilk. The main difference is that it must be built by the user, who has already did the previous steps of the Ichiran installation process. It needs to access the postgres database and the connection settings from settings.lisp are currently “baked in” during the build. It also contains a cache of some database references, so modifying the database (i.e. updating to a newer database dump) without also rebuilding ichiran-cli is highly inadvisable.

The build process is very easy. Just run sbcl and execute the following commands:

(ql:quickload :ichiran/cli)
(ichiran/cli:build)

sbcl should exit at this point, and you’ll have a new ichiran-cli (ichiran-cli.exe on Windows) executable in ichiran source directory. If sbcl didn’t exit, try deleting the old ichiran-cli and do it again, it seems that on Linux sbcl sometimes can’t overwrite this file for some reason.

Use -h option to show how to use this tool. There will be more options in the future but at the time of this post, it prints out the following:

>ichiran-cli -h
Command line interface for Ichiran

Usage: ichiran-cli [-h|--help] [-e|--eval] [-i|--with-info] [-f|--full] [input]

Available options:
  -h, --help      print this help text
  -e, --eval      evaluate arbitrary expression and print the result
  -i, --with-info print dictionary info
  -f, --full      full split info (as JSON)

By default calls ichiran:romanize, other options change this behavior

Here’s the example usage of these switches

  • ichiran-cli "一覧は最高だぞ" just prints out the romanization
  • ichiran-cli -i "一覧は最高だぞ" - equivalent of ichiran:romanize :with-info t above
  • ichiran-cli -f "一覧は最高だぞ" - outputs the full result of segmentation as JSON. This is the one you’ll probably want to use in scripts etc.
  • ichiran-cli -e "(+ 1 2 3)" - execute arbitrary Common Lisp code… yup that’s right. Since this is a new feature, I don’t know yet which commands people really want, so this option can be used to execute any command such as those listed in the previous section.

By the way, as I mentioned before, on Windows SBCL prior to 2.1.0 doesn’t parse non-ascii command line arguments correctly. Which is why I had to include a section about building a newer version of SBCL. However if you use Windows 10, there’s a workaround that avoids having to build SBCL 2.1.0. Open “Language Settings”, find a link to “Administrative language settings”, click on “Change system locale…”, and turn on “Beta: Use Unicode UTF-8 for worldwide language support”. Then reboot your computer. Voila, everything will work now. At least in regards to SBCL. I can’t guarantee that other command line apps which use locales will work after that.

That’s it for now, hope you enjoy playing around with Ichiran in this new year. よろしくおねがいします!

Nicolas Hafner2020 for Kandria in Review - Gamedev

· 59 days ago

header
Well, 2020 has certainly been a year. Given the amount of stuff that's happened, and especially the big changes in my life around Kandria, I thought it would be interesting to write up a review on the entire year. I'm not going to go month by month, but rather just give an overview on the many things that happened and how I feel about it all, so don't be surprised if I jump between things a little bit.

With that said, I want to start this out by thanking everyone for their support throughout the year. It's been really nice to see people interested in the project! I really hope that we can deliver on a good game, though it is going to take a long time still to get there. I hope you can wait for a couple more years!

A year ago Kandria still had its prototype name "Leaf", and I had just gotten done with a redesign of the main character, The Stranger. Much of the visual style of the game had already been defined by then, though, including the shadows. Most of the UI toolkit, Alloy, was also standing at that point. I think it was also then that I decided to do public monthly updates on the project.

I'm glad that I started on that pretty early, as I got a few eyes on the project pretty soon after I had posted things on Gamedev.net. There's a lot more that needs to be done in terms of outreach and marketing, though. Since the Steam launch we've been thinking a lot about how to get a bigger community together and foster active discussion surrounding the project. For now I'll keep doing the monthly summaries and weekly updates on the mailing list. I'll also try to be more active on Twitter and the Discord, but other than that we don't have a solid strategy yet.

The Steam launch and everything with Pro Helvetia leading up to that was a pretty stressful time all in all, when I was already running on fumes from everything else that had been going on. I'm really glad that I decided to afford myself these two weeks of holidays just to get away from it all. I didn't succeed entirely - I've been thinking about Kandria every day in at least some fashion - but I have been working on other projects at least, and been spending a lot of time just playing games, too, so I think I'm at least getting my mind cleared up enough to start fresh into the year next week.

On the topic of Pro Helvetia, the story there began in February, when the Swiss Game Hub had a little presentation on the organisation and its grant programme. With a little push from fellow local devs I decided to take the step and try to apply. This in turn forced a lot of changes as I decided to finally "properly go public". This meant finding a real name, creating a website and trailer, as well as a publicly playable demo, and mailing list to manage the marketing. And of course, polishing everything to actually run on other systems. I also got the Steam app at that point, with the idea of using it for testing distribution, but I only really got that sorted out after the grant submission deadline.

When I applied at Pro Helvetia I didn't expect to get the grant - and as expected, I didn't get it either. However, when we applied for the Swiss Games showcase in November, I did think we had a pretty good shot at it. Getting the message that we were, once again, rejected just two weeks before Christmas was pretty crushing, especially after all the work and rush that went into squeezing out a new trailer, new demo, Steam page, and press kit in time for it. Worst of all though, we weren't given any reason as to why others were selected over Kandria. I've tried contacting them the day after to ask for feedback, but have not heard back from them.

I've never been a confident person, so getting these rejections has been wearing down my already feeble remaining amounts of confidence, which hasn't been great for morale. While I'm not a confident person, I am however a very stubborn person, so despite everything I'm still determined to see this through to the end. Worst comes to worst I'll have to finish it on my own, but even if that came to pass I'd still do it. This is the best shot I've ever had at getting a real game made, and I'm not going to give up on it.

Moving on from these more rough sides of development, there has been a lot of progress this year, though a lot of it was in the innards of the game, and not necessarily on the visible side. That pains me a bit, since the screenshots from a year ago look very similar to the ones from today. I have to keep in mind that even without this, the progress made is necessary and valuable. Anyway, on to what I did do.

I reworked the SteamWorks library to work properly again. I rewrote the sound system stack almost entirely from scratch to allow for more complex effects and to work properly on all platforms. Large parts of the engine had to be rewritten to fix some big issues in how resources and rendering used to be organised. Not directly part of the game, but still important, I made custom mailing list and feedback systems. Hopefully there will be less things like that that I need to do next year, so there's more time for the actual game.

On the side of visible progress, most of it has been surrounding the combat system, and starting on upping the pizzazz by introducing fancy effects and post processing. There's still a lot more to do in that department though. Especially combat needs to have a lot more flare to it - explosions should kick and spray particles around, slashes need to connect visibly, getting hit has to really impact. I've looked at some other games and how they handle combat, and it really does seem like a much larger part than one might think of how the combat feels depends entirely on how many effects there are piled on. Sparks, flashes, particles, and especially crunchy sound effects make an enormous difference.

Don't get me wrong though, the animations of the characters themselves are also very important. They have to be fluid and have visible weight that is being thrown around. I struggled tremendously with that when I started out with the combat in Spring and had to do the first animations myself. I'm very glad that I've recruited Fred to take care of that part, as he's done an amazing job at it. The new animations feel a lot more fun, fluid, and real.

Speaking of Fred, one of the biggest changes this year was that I finally decided to put not only my time, but also my money on the line and actually hire some people to expand the team. This is something that was a long time coming. I always knew when I started out that I'd have to eventually expand the team, simply because the scale would require it to get it done in a reasonable amount of time, and because I simply don't trust my own skills well enough to get a great product out of them. That's where the confidence thing comes in again.

The hiring process took an entire month of my time, mostly because there were way more applications than I ever thought there would be, and I wanted to do my due diligence and investigate everyone to a good degree. Ultimately finalising the selection was also difficult for me, and took me over a week of deliberation. I'm happy with the choices I made, but I still wish I had the funds to just hire more people.

Since the game is almost entirely built on a custom stack of software, engine and all, there's a lot of rough edges and corner case bugs that hinder development and cost us a lot of time. I really wish I had the funds to hire another skilled programmer to take care of those so I can focus more on directing the story, art, and general features and level design. Still, we're already on a tight budget that isn't going to last for the entire duration of development unless we can procure additional funding somehow. We've been talking about that a fair bit, too, but there's no clear decision yet.

So far the plan is still to complete a vertical slice in the coming months and then do another planning session to see how things hash out once we have a better idea of the development costs involved and how the overall plot and world will pan out. Then comes another application for the Pro Helvetia grant in September. If we get that, we'll have extended funds for another year, which should hopefully bridge the gap well enough to pull through to the end. If not... well, there's other possibilities that I don't want to really discuss yet as it's all still too uncertain.

As you may know, during most of the development of Kandria so far I was a Master's student at ETH. I've been a student for a long time, since my Bachelor's took me a long time to complete, largely in part to not being able to take the stress of taking on too many subjects at once. Most of the classes I either didn't care for, or outright loathed having to work on, so it was not a very merry time. Still, I managed to persevere. Now, in the Master's programme for Computer Science at ETH there's a requirement to complete two of three "interdisciplinary laboratories". You have to complete these regardless of the focus you take, and so regardless of your interests or target skillset. I tried all three, and failed all three, the last two of which I failed this Summer. All three were very hard courses that required a ton of time investment. I did not expect to fail them all. Whatever the case, this, in addition with the strict term limits at ETH, meant that it was not guaranteed I'd be able to complete my Master's even if I did decide to try them again in a year. It would mean spending at least one and a half more years to complete my Master's, if I managed to pass these classes the second time.

I decided that these odds were no longer worth it. University made me miserable, and I was not sure how big of a benefit the degree would be anyway. So I made the big decision to work full time on Kandria, which I have now been doing since September.

Doing this also shifted the project quite a bit though, as now it is no longer a game project I just want to complete on the side, it's now something that has to prove not only possible, but also financially viable, in order to be able to keep doing this. Naturally this places a huge burden on me, and even if I don't want to think about it much, my subconscious still does anyway. This has lead to a somewhat unhealthy work/life balance, where I couldn't justify working on other side projects like I used to all this time before, as the thought of "but shouldn't you be working on the game, instead?" always came creeping around the corner.

This has especially been a problem in November and the beginning of December, and is why I've run so badly out of steam. These two weeks of holidays have really been great to get away from that. Still, I'm going to have to figure out some better balance to make this sustainable in the long run. I can't be going on holidays every two months or so after all. At this point I don't yet know how exactly to do this, except that I know I need to weave different projects into my schedule somehow. That's something to figure out in the new year.

Tim and I have already been making some good progress discussing the characters, setting, world, and overall story in December, and I'm really eager to dive back into that and get started on planning out the first section of Kandria for the vertical slice. I also have a bunch of cool ideas for new features and effects to implement. I'm looking forward to diving back into all of that next week, but I'm also cautious about all the challenges we already know about. I really don't want to rush it and end up with something we have to throw away in the end.

This entry has gone on for long enough already, even if there's a lot of details and smaller developments I skipped, so I'll try to bring this to a close. As always, if you want to be kept up to date on the development, sign up for the mailing list!

Tim also wanted to write a little bit about his experience working on Kandria the past two months, so here goes:

It's been a whirlwind two months working on Kandria! I've already gotten heavily involved in writing marketing text, developing the lore, and making a demo quest to learn the dev tools. I'm looking forward to coming back after Christmas and keeping the momentum going for the vertical slice. I expect I'll be getting more hands on with the tools in particular, to write multiple quests for a hub-like area; now I've learned the basics and will have more time, I'll be looking to structure it better as well, using the quest system to its fullest, rather than brute-forcing it with task interactions alone. :)

With that, I think I'll call the yearly round-up done. I hope next year will be better than this one, and am currently being cautiously optimistic about that. I wish everyone out there, and especially you reading this, all the best in 2021!

Leo ZovicProfiling `house`. Again.

· 69 days ago

So I've plowed some of my vacation time into polishing up/hacking on some old projects. Including house, the web server I complained was garbage, but still had one distinct advantage over other Common Lisp webservers. Namely; because it's the only natively implemented one, it will work out-of-the-box, without issue, anywhere you can install quicklisp and a LISP it runs on.

This hacking attempt was aimed at addressing the complaint. Most of the major-overhaul branch was aimed at making the code more readable and sensical. Making handlers and http-types much simpler, both implementationally and conceptually. But I want to throw at least a little effort at performance. With that in mind, I wanted a preliminary benchmark. I'm following fukamachis' procedure for woo. Note that, since house is a single-threaded server (for now), I'm only doing single-threaded benchmarks.

; SLIME 2.26
CL-USER> (ql:quickload :house)
To load "house":
  Load 1 ASDF system:
    house
; Loading "house"
.....
(:HOUSE)
CL-USER> (in-package :house)
#<PACKAGE "HOUSE">
HOUSE> (define-handler (root) () "Hello world!")
#<HANDLER-TABLE {1004593CF3}>
HOUSE> (house:start 5000)
inaimathi@this:~$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.01ms    5.85ms 204.63ms   98.73%
    Req/Sec     2.64k     0.89k    7.22k    62.16%
  104779 requests in 10.10s, 30.58MB read
  Socket errors: connect 0, read 104775, write 0, timeout 0
Requests/sec:  10374.93
Transfer/sec:      3.03MB
inaimathi@this:~$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.74ms   19.05ms 408.54ms   98.18%
    Req/Sec     2.58k     0.85k    4.64k    57.39%
  102543 requests in 10.10s, 29.92MB read
  Socket errors: connect 0, read 102539, write 0, timeout 0
Requests/sec:  10152.79
Transfer/sec:      2.96MB
inaimathi@this:~$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.56ms   59.54ms   1.66s    99.27%
    Req/Sec     3.10k     1.83k    9.56k    76.72%
  103979 requests in 10.01s, 30.34MB read
  Socket errors: connect 0, read 103979, write 0, timeout 4
Requests/sec:  10392.46
Transfer/sec:      3.03MB
inaimathi@this:~$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.49ms   85.22ms   1.66s    98.81%
    Req/Sec     3.23k     2.16k   11.90k    81.01%
  102236 requests in 10.01s, 29.83MB read
  Socket errors: connect 0, read 102232, write 0, timeout 4
Requests/sec:  10215.87
Transfer/sec:      2.98MB
inaimathi@this:~$

So that puts house comfortably in the same league as Tornado on PyPy or the node.js server. This is not a bad league to be in, but I want to see if I can do better.

Step 1 - Kill Methods

defmethod is a thing I was seemingly obsessed with when I wrote house. This isn't necessarily a bad thing from the legibility perspective; because they have type annotations, it's clearer what an expected input is from a reading of the code. However, there's two disadvantages to using methods where you don't have to.

  1. You'll often get a no-defined-method error on weird input, rather than something more descriptive and specific the way you probably would when using a normal function
  2. Your performance will sometimes irredeemably suck.

The first point is a nit, but the second one is worth dealing with in the context of a library that should probably perform reasonably well at least some of the time. The cause of that problem is that methods can't be inlined. Because the point of them is to dispatch on a type-table of their arguments at runtime, they can't do their work at compile-time to inline the result without some serious trickery1. Today, I'm avoiding trickery and just re-writing every method in house that I can into a function, usually by using etypecase.

Some of these are trivial conversions

;;; house.lisp
...
-(defmethod start ((port integer) &optional (host usocket:*wildcard-host*))
+(defun start (port &optional (host usocket:*wildcard-host*))
+  (assert (integerp port))
...
-(defmethod process-ready ((ready stream-server-usocket) (conns hash-table))
-  (setf (gethash (socket-accept ready :element-type 'octet) conns) nil))
-
-(defmethod process-ready ((ready stream-usocket) (conns hash-table))
+(defun process-ready (ready conns)
+  (assert (hash-table-p conn))
+  (etypecase ready
+    (stream-server-usocket (setf (gethash (socket-accept ready :element-type 'octet) conns) nil))
+    (stream-usocket
...
-(defmethod parse-cookies ((cookie string))
+(defun parse-cookies (cookie)
+  (assert (stringp cookie))
...
-(defmethod handle-request! ((sock usocket) (req request))
+(defun handle-request! (sock req)
...
-(defmethod error! ((err response) (sock usocket) &optional instance)
-  (declare (ignorable instance))
+(defun error! (err sock)
,,,
;;; session.lisp
...
-(defmethod new-session-hook! ((callback function))
+(defun new-session-hook! (callback)
...
-(defmethod poke! ((sess session))
+(defun poke! (sess)
...
;;; util.lisp
...
-(defmethod path->uri ((path pathname) &key stem-from)
+(defun path->uri (path &key stem-from)
...
-(defmethod path->mimetype ((path pathname))
+(defun path->mimetype (path)
...

Some are slightly more complicated. In particular, parse looks like it would conflate two entirely separate functions, but on inspection, we know the type of its argument at every call site.

./house.lisp:46:		      (setf (parameters (request buf)) (nconc (parse buf) (parameters (request buf)))))
./house.lisp:68:	   do (multiple-value-bind (parsed expecting) (parse buffer)
./house.lisp:92:(defmethod parse ((str string))
./house.lisp:110:(defmethod parse ((buf buffer))
./house.lisp:116:	(parse str))))

So, we can convert parse to two separate, named functions. write! is basically the same situation.

;;; house.lisp
...
-(defmethod parse ((str string))
+(defun parse-request-string (str)
...
-(defmethod parse ((buf buffer))
+(defun parse-buffer (buf)
...
-(defmethod write! ((res response) (stream stream))
+(defun write-response! (res stream)
...
-(defmethod write! ((res sse) (stream stream))
+(defun write-sse! (res stream)
...

Not pictured; changes at each call-site to call the correct one.

The parse-params method is a bit harder to tease out. Because it looks like it genuinely is one polymorphic function. Again, though, on closer inspection of the fully internal to house call-sites makes it clear that we almost always know what we're passing as arguments at compile-time.

./house.lisp:78:(defmethod parse-params (content-type (params null)) nil)
./house.lisp:79:(defmethod parse-params (content-type (params string))
./house.lisp:83:(defmethod parse-params ((content-type (eql :application/json)) (params string))
./house.lisp:107:	(setf (parameters req) (parse-params nil parameters))
./house.lisp:113:	(parse-params
					 (->keyword (cdr (assoc :content-type (headers (request buf)))))
					 str)

That "almost" is going to be a slight pain though; we need to do a runtime dispatch inside of parse-buffer to figure out whether we're parsing JSON or a param-encoded string.

...
-(defmethod parse-params (content-type (params null)) nil)
-(defmethod parse-params (content-type (params string))
+(defun parse-param-string (params)
   (loop for pair in (split "&" params)
-     for (name val) = (split "=" pair)
-     collect (cons (->keyword name) (or val ""))))
-
-(defmethod parse-params ((content-type (eql :application/json)) (params string))
-  (cl-json:decode-json-from-string params))
+	for (name val) = (split "=" pair)
+	collect (cons (->keyword name) (or val ""))))
...
-	(parse-params
-	 (->keyword (cdr (assoc :content-type (headers (request buf)))))
-	 str)
-	(parse str))))
+	(if (eq :application/json (->keyword (cdr (assoc :content-type (headers (request buf))))))
+	    (cl-json:decode-json-from-string str)
+	    (parse-param-string str))
+	(parse-request-string str))))
...

The last one is going to be a headache. The lookup method is meant to be a general accessor, and has a setf method defined. I'm not going that way right now; lets see if we gained anything with our current efforts.

Second verse same as the first.

; SLIME 2.26
CL-USER> (ql:quickload :house)
To load "house":
  Load 1 ASDF system:
    house
; Loading "house"
.....
(:HOUSE)
CL-USER> (in-package :house)
#<PACKAGE "HOUSE">
HOUSE> (define-handler (root) () "Hello world!")
#<HANDLER-TABLE {1004593CF3}>
HOUSE> (house:start 5000)
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.96ms    4.02ms  76.87ms   98.43%
    Req/Sec     2.70k     0.98k    7.57k    73.83%
  103951 requests in 10.10s, 30.34MB read
  Socket errors: connect 0, read 103947, write 0, timeout 0
Requests/sec:  10292.48
Transfer/sec:      3.00MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   846.32us    2.63ms  58.29ms   98.26%
    Req/Sec     2.64k     0.94k   11.13k    72.89%
  102661 requests in 10.10s, 29.96MB read
  Socket errors: connect 0, read 102658, write 0, timeout 0
Requests/sec:  10165.46
Transfer/sec:      2.97MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.57ms   90.07ms   1.66s    98.96%
    Req/Sec     3.71k     2.87k   11.73k    74.30%
  105162 requests in 10.10s, 30.69MB read
  Socket errors: connect 0, read 105159, write 0, timeout 2
Requests/sec:  10412.91
Transfer/sec:      3.04MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.69ms   70.32ms   1.66s    99.25%
    Req/Sec     3.06k     1.82k    9.46k    74.40%
  101302 requests in 10.10s, 29.56MB read
  Socket errors: connect 0, read 101299, write 0, timeout 3
Requests/sec:  10030.14
Transfer/sec:      2.93MB
inaimathi@this:~/quicklisp/local-projects/house$

Aaand it looks like the effect was neglegible. Oh well. I honestly think that the untangling we've done so far makes the parts of the codebase that its' touched more readable, so I'm keeping them, but there's no great improvement yet. Perhaps if we inline some things?

;;; package.lisp
-(declaim (inline crlf write-ln idling? flex-stream))
+(declaim (inline crlf write-ln idling? flex-stream write-response! write-sse! process-ready parse-param-string parse-request-string))
wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.71ms   15.37ms 412.51ms   98.91%
    Req/Sec     2.69k     0.91k    6.28k    65.37%
  103607 requests in 10.10s, 30.24MB read
  Socket errors: connect 0, read 103603, write 0, timeout 0
Requests/sec:  10258.44
Transfer/sec:      2.99MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   837.49us    2.66ms  58.36ms   98.36%
    Req/Sec     2.63k   836.52     3.81k    49.37%
  103449 requests in 10.10s, 30.19MB read
  Socket errors: connect 0, read 103446, write 0, timeout 0
Requests/sec:  10242.91
Transfer/sec:      2.99MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.23ms   74.76ms   1.89s    99.08%
    Req/Sec     4.01k     2.20k   10.23k    58.89%
  101524 requests in 10.10s, 29.63MB read
  Socket errors: connect 0, read 101522, write 0, timeout 4
Requests/sec:  10052.56
Transfer/sec:      2.93MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.75ms   70.98ms   1.67s    99.27%
    Req/Sec     3.19k     2.11k   10.26k    81.39%
  100944 requests in 10.01s, 29.46MB read
  Socket errors: connect 0, read 100941, write 0, timeout 1
Requests/sec:  10088.23
Transfer/sec:      2.94MB

Again, no huge difference. On closer inspection, lookup is only used in one place internally, and it's easy to replace with gethash so I'm just going to do that and re-check real quick.

;;; channel.lisp
...
-  (push sock (lookup channel *channels*))
+  (push sock (gethash channel *channels*))
...
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.95ms    3.72ms  72.70ms   98.43%
    Req/Sec     2.66k     1.00k   11.52k    73.45%
  102839 requests in 10.10s, 30.01MB read
  Socket errors: connect 0, read 102835, write 0, timeout 0
Requests/sec:  10183.46
Transfer/sec:      2.97MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.87ms    2.85ms  59.32ms   98.19%
    Req/Sec     2.62k     0.86k    3.87k    54.82%
  102818 requests in 10.10s, 30.00MB read
  Socket errors: connect 0, read 102814, write 0, timeout 0
Requests/sec:  10180.62
Transfer/sec:      2.97MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.96ms   80.03ms   1.68s    99.10%
    Req/Sec     3.11k     2.12k   11.72k    78.40%
  105460 requests in 10.10s, 30.78MB read
  Socket errors: connect 0, read 105456, write 0, timeout 5
Requests/sec:  10441.77
Transfer/sec:      3.05MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.22ms   83.95ms   1.66s    98.84%
    Req/Sec     3.19k     2.07k   11.66k    73.23%
  103933 requests in 10.10s, 30.33MB read
  Socket errors: connect 0, read 103930, write 0, timeout 5
Requests/sec:  10290.43
Transfer/sec:      3.00MB

To no ones' great surprise, still not much of a difference. I'm going to let the lookup issue dangle for the moment, because it has to do with a trick I want to pull a bit later on, but before we get to that...

Step 2 - Kill Classes

The second step is to kill class definitions entirely. Their accessor functions are also generic, and therefore rely on method dispatch. structs are a bit clumsier, but probably faster in the end. Now, we can't really mess with session, request and response, because those are part of houses' external interface, but there's three places where we can replace defclass with defstruct.

Re-writing buffer, sse and handler-entry ...

;;; model.lisp
...
-(defclass sse ()
-  ((id :reader id :initarg :id :initform nil)
-   (event :reader event :initarg :event :initform nil)
-   (retry :reader retry :initarg :retry :initform nil)
-   (data :reader data :initarg :data)))
...
-(defclass buffer ()
-  ((tries :accessor tries :initform 0)
-   (contents :accessor contents :initform nil)
-   (bi-stream :reader bi-stream :initarg :bi-stream)
-   (total-buffered :accessor total-buffered :initform 0)
-   (started :reader started :initform (get-universal-time))
-   (request :accessor request :initform nil)
-   (expecting :accessor expecting :initform 0)))
...
-(defclass handler-entry ()
-  ((fn :reader fn :initarg :fn :initform nil)
-   (closing? :reader closing? :initarg :closing? :initform t)))
...
;;; house.lisp
...
-(defun write-sse! (res stream)
-  (format stream "~@[id: ~a~%~]~@[event: ~a~%~]~@[retry: ~a~%~]data: ~a~%~%"
-	  (id res) (event res) (retry res) (data res)))
...
-(defun buffer! (buffer)
-  (handler-case
-      (let ((stream (bi-stream buffer)))
-	(incf (tries buffer))
-	(loop for char = (read-char-no-hang stream)
-	   until (or (null char) (eql :eof char))
-	   do (push char (contents buffer))
-	   do (incf (total-buffered buffer))
-	   when (request buffer) do (decf (expecting buffer))
-	   when (and #-windows(char= char #\linefeed)
-		     #+windows(char= char #\newline)
-		 (line-terminated? (contents buffer)))
-	   do (multiple-value-bind (parsed expecting) (parse-buffer buffer)
-		(setf (request buffer) parsed
-		      (expecting buffer) expecting
-		      (contents buffer) nil)
-		(return char))
-	   when (> (total-buffered buffer) +max-request-size+) return char
-	   finally (return char)))
-    (error () :eof)))
...
-(defun parse-buffer (buf)
-  (let ((str (coerce (reverse (contents buf)) 'string)))
-    (if (request buf)
-	(if (eq :application/json (->keyword (cdr (assoc :content-type (headers (request buf))))))
-	    (cl-json:decode-json-from-string str)
-	    (parse-param-string str))
-	(parse-request-string str))))
...
;;; define-handler.lisp
+(defstruct handler-entry
+  (fn nil)
+  (closing? t))
...
-    (make-instance
-     'handler-entry
+    (make-handler-entry
;;; channel.lisp
...
+(defstruct (sse (:constructor make-sse (data &key id event retry)))
+  (id nil) (event nil) (retry nil)
+  (data (error "an SSE must have :data") :type string))
...
-(defun make-sse (data &key id event retry)
-  (make-instance 'sse :data data :id id :event event :retry retry))
+(defun write-sse! (res stream)
+  (format stream "~@[id: ~a~%~]~@[event: ~a~%~]~@[retry: ~a~%~]data: ~a~%~%"
+	  (ss-id res) (sse-event res) (sse-retry res) (sse-data res)))
...
;;; buffer.lisp
+(in-package :house)
+
+(defstruct (buffer (:constructor make-buffer (bi-stream)))
+  (tries 0 :type integer)
+  (contents nil)
+  (bi-stream nil)
+  (total-buffered 0 :type integer)
+  (started (get-universal-time))
+  (request nil)
+  (expecting 0 :type integer))
+
+(defun buffer! (buffer)
+  (handler-case
+      (let ((stream (buffer-bi-stream buffer)))
+	(incf (buffer-tries buffer))
+	(loop for char = (read-char-no-hang stream)
+	   until (or (null char) (eql :eof char))
+	   do (push char (buffer-contents buffer))
+	   do (incf (buffer-total-buffered buffer))
+	   when (buffer-request buffer) do (decf (buffer-expecting buffer))
+	   when (and #-windows(char= char #\linefeed)
+		     #+windows(char= char #\newline)
+		 (line-terminated? (buffer-contents buffer)))
+	   do (multiple-value-bind (parsed expecting) (parse-buffer buffer)
+		(setf (buffer-request buffer) parsed
+		      (buffer-expecting buffer) expecting
+		      (buffer-contents buffer) nil)
+		(return char))
+	   when (> (buffer-total-buffered buffer) +max-request-size+) return char
+	   finally (return char)))
+    (error () :eof)))
+
+(defun parse-buffer (buf)
+  (let ((str (coerce (reverse (buffer-contents buf)) 'string)))
+    (if (buffer-request buf)
+	(if (eq :application/json (->keyword (cdr (assoc :content-type (headers (buffer-request buf))))))
+	    (cl-json:decode-json-from-string str)
+	    (parse-param-string str))
+	(parse-request-string str))))

... should get us _something. Right?

inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.09ms    6.18ms 202.73ms   98.55%
    Req/Sec     2.69k     0.89k    4.02k    56.74%
  105108 requests in 10.10s, 30.67MB read
  Socket errors: connect 0, read 105105, write 0, timeout 0
Requests/sec:  10406.92
Transfer/sec:      3.04MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 10 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.98ms    5.78ms 204.47ms   98.86%
    Req/Sec     2.67k   848.77     3.98k    54.71%
  104242 requests in 10.10s, 30.42MB read
  Socket errors: connect 0, read 104242, write 0, timeout 0
Requests/sec:  10321.40
Transfer/sec:      3.01MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.93ms   79.75ms   1.66s    99.10%
    Req/Sec     3.33k     2.46k   11.95k    79.87%
  105920 requests in 10.10s, 30.91MB read
  Socket errors: connect 0, read 105918, write 0, timeout 2
Requests/sec:  10487.59
Transfer/sec:      3.06MB
inaimathi@this:~/quicklisp/local-projects/house$ wrk -c 100 -t 4 -d 10 http://127.0.0.1:5000
Running 10s test @ http://127.0.0.1:5000
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.78ms   61.11ms   1.68s    99.30%
    Req/Sec     2.83k     1.26k    7.01k    70.22%
  103381 requests in 10.10s, 30.17MB read
  Socket errors: connect 0, read 103378, write 0, timeout 0
Requests/sec:  10235.14
Transfer/sec:      2.99MB

Very little noticeable gain, I'm afraid. Ok, there's one more thing I'm tempted to try. There were hints earlier that this was coming, including this, but if you don't follow my github you might still be surprised.

Step 3 - Musing on CLJ

Now that we have what I think is a reasonably fast implementation of house, I want to see whether2 [clj](https://github.com/inaimathi/clj) does performance damage to the implementation. I want to see this because, the clj datastructures and syntax really improve readability and REPL development; there's a bunch of situations in which I missed having that level of visibility into my structures before I even began this benchmark article. There's even probably a few places where it saves some performance by referencing other partial structures. The problem is that I'm guessing it's a net negative in terms of performance, so I want to see what a conversion would do to my benchmark before I go through with it.

This is going to be especially useful for houses' external interface. And given that I've already had to break compatibility to write this overhaul, this is probably the best possible time to test the theory. The trouble is that I'm not entirely sure what the real interface looks like quite yet, so I'm not going to be implementing it today. These are just some musings.

The current house model for handler/response interaction is that a handler returns either a response (in the event of a redirect!) or a string (in any other event). This makes a few things kind of difficult. Firstly, it means that session and header manipulation has to happen by effect. That is, they're not included as part of the return value; they have to be exposed in some other way. In the case of headers, it's via an alist bound to the invisible symbol headers inside of the handler body. This ... is less than ideal.

If we take the http-kit approach, we'd expect our handlers to always return a map. And if that map had slots for headers/session, those things would be set as appropriate in the outgoing response and/or server state. Our input would also be a map. And it would naturally contain method/headers/path/parameters/session/etc slots that a handler writer would want to make use of. I'm not entirely clear on whether we'd want to make this the primary internal and external representation, or if we're just looking for an easily manipulated layer for the users. I'm leaning towards the first of those options.

This ... actually doesn't sound too hard if cut at the right level. Lets give it a shot, I guess.

It wasn't.

There's enough weird shit happening here that I need a fresh brain for it. That was enough for now. The main roadblock I hit is that it turns out that a lot more of the internal interface here depends on mutation than I thought. This is bad for readability and coceptual simplicity, but good in the sense that I can move away from these models first, then see about integrating clj later.

I'll probably take another run up this hill later, but for now, I think I'm moving on to other issues.

  1. Wait, why use methods then? They're good specifically in the situation where you want to establish an interface for a set of datastructures that you expect to have to extend outside of your library. If all the extension is going to happen inside, you can still make the argument that etypecase is the right way to go. But if you want the callers of your code to be able to define new behaviors for datastructures they specify themselves, then absolutely reach for defmethod.
  2. More realistically, "how much" rather than "whether"

Quicklisp newsDecember 2020 Quicklisp dist update now available

· 69 days ago

 New projects

  • aether — A DSL for emulating an actor-based distributed system, housed on a family of emulated devices. — MIT (See LICENSE.md)
  • binding-arrows — An implementation of threading macros based on binding anonymous variables — MIT
  • bitfield — Efficiently represent several finite sets or small integers as a single non-negative integer. — MIT
  • cl-bloggy — A simple extendable blogging system to use with Hunchentoot — MIT
  • cl-data-structures — Data structures, ranges, ranges algorithms. — BSD simplified
  • cl-html-readme — A HTML Documentation Generator for Common Lisp projects. — MIT
  • cl-ini — INI file parser — MIT
  • cl-notebook — A notebook-style in-browser editor for Common Lisp — AGPL3
  • cl-unix-sockets — UNIX Domain socket — Apache License, Version 2.0
  • cmd — A utility for running external programs — MIT
  • cytoscape-clj — A cytoscape widget for Common Lisp Jupyter. — MIT
  • damn-fast-priority-queue — A heap-based priority queue whose first and foremost priority is speed. — MIT
  • dataloader — A universal loader library for various data formats for images/audio — LLGPL
  • ecclesia — Utilities for parsing Lisp code. — MIT
  • fuzzy-match — From a string input and a list of candidates, return the most relevant candidates first. — MIT
  • geco — GECO: Genetic Evolution through Combination of Objects A CLOS-based Framework for Prototyping Genetic Algorithms — GPL 2.0
  • gtwiwtg — Lazy-ish iterators — GPLv3
  • gute — Gene's personal kitchen sink library. — MIT
  • lense — Racket style lenses for the Common Lisp. — BSD-2
  • linear-programming-glpk — A backend for linear-programming using GLPK — GPL 3.0
  • mgrs — Convert coordinates between Latitude/Longitude and MGRS. — GPL-3
  • monomyth — A distributed data processing library for CL — MPL 2.0
  • neural-classifier — Classification of samples based on neural network. — 2-clause BSD
  • roan — A library to support change ringing applications — MIT
  • simple-neural-network — Simple neural network — GPL-3
  • stefil- — Unspecified — Unspecified
  • tree-search — Search recursively through trees of nested lists — ISC
  • ttt — A language for transparent modifications of s-expression based trees. — GPLv3
  • utm-ups — Convert coordinates between Latitude/Longitude and UTM or UPS. — GPL-3
  • with-contexts — The WITH-CONTEXT System. A system providing a WITH macro and 'context'ualized objects handled by a ENTER/HANDLE/EXIT protocol in the spirit of Python's WITH macro. Only better, or, at a minimum different, of course. — BSD

Updated projects: 3bmd, 3bz, 3d-matrices, 3d-vectors, adopt, algae, april, arc-compat, architecture.builder-protocol, array-utils, arrow-macros, aws-sign4, bdef, binpack, check-bnf, cl-ana, cl-ansi-text, cl-bunny, cl-catmull-rom-spline, cl-cffi-gtk, cl-collider, cl-conllu, cl-covid19, cl-custom-hash-table, cl-digraph, cl-environments, cl-gamepad, cl-gd, cl-glfw3, cl-gserver, cl-interpol, cl-kraken, cl-liballegro, cl-liballegro-nuklear, cl-libyaml, cl-lzlib, cl-markless, cl-maxminddb, cl-mime, cl-mixed, cl-mongo-id, cl-naive-store, cl-octet-streams, cl-pass, cl-patterns, cl-pdf, cl-portaudio, cl-prevalence, cl-randist, cl-rdkafka, cl-sdl2, cl-sdl2-mixer, cl-semver, cl-sendgrid, cl-setlocale, cl-skkserv, cl-steamworks, cl-str, cl-tcod, cl-telegram-bot, cl-unicode, cl-utils, cl-wavelets, cl-webkit, cl-yaml, clesh, clj, clml, closer-mop, clsql, clweb, colored, common-lisp-jupyter, concrete-syntax-tree, conduit-packages, consix, corona, croatoan, curry-compose-reader-macros, dartscltools, dartscluuid, data-lens, defclass-std, deploy, dexador, djula, docparser, doplus, easy-audio, easy-routes, eazy-documentation, eclector, esrap, file-select, flexichain, float-features, floating-point-contractions, functional-trees, gadgets, gendl, generic-cl, glacier, golden-utils, gtirb-capstone, harmony, helambdap, house, hunchentoot-multi-acceptor, hyperluminal-mem, imago, ironclad, jingoh, jpeg-turbo, jsonrpc, kekule-clj, linear-programming, linux-packaging, lisp-chat, lisp-critic, lisp-gflags, literate-lisp, lmdb, local-package-aliases, local-time, lquery, markup, math, mcclim, millet, mito, mmap, mutility, named-readtables, neo4cl, nibbles, num-utils, origin, orizuru-orm, parachute, pathname-utils, perceptual-hashes, petalisp, phoe-toolbox, physical-quantities, picl, pjlink, portable-condition-system, postmodern, prometheus.cl, protest, protobuf, py4cl, py4cl2, qt-libs, quilc, quri, rcl, read-number, reader, rpcq, rutils, s-graphviz, sc-extensions, secret-values, sel, select, serapeum, shadow, simple-parallel-tasks, slime, sly, snooze, static-dispatch, stmx, stumpwm, swank-client, swank-protocol, sxql, tesseract-capi, textery, tooter, trace-db, trivial-compress, trivial-do, trivial-pooled-database, trivial-string-template, uax-15, uncursed, verbose, vp-trees, weblocks-examples, weblocks-prototype-js.

Removed projects: cl-arrows, cl-generic-arithmetic, clcs-code, dyna, osmpbf, sanity-clause, unicly.

To get this update, use (ql:update-dist "quicklisp")

Enjoy!


For older items, see the Planet Lisp Archives.


Last updated: 2021-02-24 12:50