# Content tagged Lisp

Migrating Pigeons
posted on 2018-04-02 11:00

About a month ago, I converted Podcastinet from using mito to postmodern as the database interface. One of the things I wanted was to have finer-grained control over the database. I was relying on mito's ability to automtaically migrate based on database introspection, which was useful, but also didn't allow migrating data between columns or other, more manual sorts of migrations. To enable that sort of thing, I wrote Postmodern Passenger Pigeon, which I still need to document/write a README for, but is basically functional.

It creates a table in your database named pigeon_migrations that stores a bunch of revisions and it works based on a linear idea of your database's state. That is, there are no branches in the revisions, it's a list, not a tree.

It can be configured with a pigeon.toml file that mostly just takes a migration-directory (but can also take a database-url, though it will look at the environment variable DATABASE_URL, too). And then you have a bunch of files in that directory named like [universal-time]-[hyphenized-name].lisp that house the actual revisions.

For example:

3727984190-create-people-table.lisp:

"Creates the people table."

(defun up ()
(query (:create-table people
((id :type integer :primary-key t)
(name :type (string 20))))))

(defun down ()
(query (:drop-table 'people)))

The file can have a docstring at the top to describe the revision in more detail, and then expects an up and a down function to be defined. (The revisions are each evaluated in their own sandbox package, so you can refer to other packages by their full name, though I don't recommend it.) Postmodern is used by default.

## Creating migrations

If you want to create a new revision, you can call (ppp.migration:new "Short description of my migration") and it will create the new file for you. If you decide that you don't need a migration you're working on, just revert the migration and delete the file and you can go ahead and make new ones. They're applied in numerical order of the timestamp.

## Running migrations

To apply or revert migrations, use the ppp:migrate function. The first argument is the direction to migrate in: :up will apply new migrations, :down will revert existing migrations. You can give a number as the second argument to say how many you'd like to apply or revert, i.e. (ppp:migrate :up 1) will apply the next migration that hasn't been applied, and no more, (ppp:migrate :down 3) will revert 3 migrations, starting will the most recently applied.

There are also the special values :head and :base which mean "all the migrations applied" and "no migrations applied" (which should mean an empty db schema). In my site, I run (ppp:migrate :up :head) when it starts up to apply any migrations that came with the latest code push.

Horrible Bug
posted on 2017-06-10 16:01:00

I just figured out the most horrible bug! For nearly two months, I've been watching my deploys of a lisp webapp say

Woo server is going to start.
Listening on localhost:5000.
{1005C85A43}>:
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<WOO.EV.CONDITION:OS-ERROR {1007BD7533}> #<unused argument>)

over and over and over.

Since I only find a little time to work on this stuff, around my day job, it was really pissing me off.

When I finally figured out what the problem was, I was even more pissed off.

It took me forever to try to figure out what that -8 error code was. At least I'd seen C code, before, to know that that was what it was. I must have miscounted, looking at strange webpages that listed the possible errors for getaddrinfo(), and that threw me off for a month alone. Today, I finally decided there wasn't some quick way to solve this, and I had to dig into the problem with due dilligence. I wrote a quick C program to get the description of the error from the horse's mouth:

#include <netdb.h>
#include <stdio.h>

int main(int argc, char** argv[]) {
const char *explanation = gai_strerror(-8);
printf("Error -8 is %s", explanation);
return 0;
}

And it informed me: Error -8 is Servname not supported for ai_socktype. A quick search led me to this stackoverflow, which had a comment I include here for your convenience:

I had this problem with a Tornado/Python app. Apparently, this can be caused by the port being interpreted as a string instead of an integer. So in my case, I needed to change my startup script to force it to be interpreted as an integer.

This turned out to be precisely my problem.

I had written my development configuration as a p-list, which, being readable could contain numbers. In production, I was reading configuration out of environment variables, which are all strings. So, pop in an if statement with a (parse-integer port) and blam.

Two months of hobby time, down the toilet.

I'm gonna have to see if I can make a patch for woo or something to make sure this doesn't happen, again.

An afternoon hack
posted on 2016-12-05 14:42:00

A few months ago, I decided that lucerne had enough things missing that I was rewriting for different projects, that I would bite the bullet and try writing my own framework. I'm generally a fan of Fernando Borretti's work, and I definitely suggest looking at the different libraries he's created for common lisp webapps, but these days he's been more of an inspiration for writing my own versions than anything else.

So, I've been writing nest for a while, now, which doesn't have much documentation and very spotty test coverage, at the moment, because it is still very much in the process of spilling out of my forehead, like some halting Athena, and I wouldn't want to break anyone else's webapps while I work on it. (I am trying to version and tag it, though, so at least you'll get the same sort of broken if you stick to a tag.)

I was looking through fossdroid.com, yesterday, and noticed the shopping list app, which boasted an open source php and mysql backend. It comes with some script to set it up under apache, but the list of technologies I wasn't very familiar with was too long, so I decided to test out my framework to get it running under lisp (and whatever database you want that's supported by cl-dbi).

Happily, I didn't bite off more than I could chew on this one, and I had it all working by midnight so I could sleep, too!

I found the api very strange, because it's all implemented at one endpoint. The android app also adds a trailing slash to whatever you configure as the host, which was odd, because I would get requests to /api.php/ or // depending on how I configured it. The app sends what function it wants to run as a post parameter, and then, hopefully, all the parameters appropriate for that function are along for the ride, too. It was a good excuse to pull out my validation library for a ride, which I generally use with a macro like

(defmacro validated (&body body)
"Handle validation errors (v:<validation-error>) and respond with an error for the client."

(handler-case
(progn ,@body)
(v:<validation-error> (e)
(declare (ignore e))
(api-response :type +api-error-unknown+
:content "Invalid parameter"))))


I've also started using validate in Podcastinet, with a similar macro that sends back a 400 error in the event validation fails. Right now, there isn't a good way to collect failed validations for a meaningful error for the API consumer, but I suppose that will come in time.

In any event, it was a fun project and made me feel pretty good about nest, since I was able to use it to build an app in an afternoon. Now I need to make a docker image so I can throw it up on a server and keep track of my groceries! Check out the project here and let me know what you think. I'd also really appreciate feedback on validate. Maybe I'll post it to the lisp subreddit, soon, for some of that.

Clack Static Asset Middleware
posted on 2016-05-12 02:01:00

As part of my glacial march to get Knttl running the way I'd like, I wrote a middleware for the clack web framework to help me out!

When you're serving javascript and css files for a website, you want to inform the browser to keep a local copy as long as possible, because then it won't bother your webserver to download it again and the page will render faster for your users, too. There's a popular quote about programming

There are only two hard things in Computer Science: cache invalidation and naming things.
It seems this was said by a Phil Karlton.

So, like many things related to the it-was-never-meant-to-be-used-this-way world of web technology, there are some interesting best practices that evolved around this idea that, ideally, only one copy of a static file should ever need to be sent to a user, if their browser works well.

There are ways to tell a browser how long to hold onto a file before checking if there's a new version of it, but that only gets you so far. Presumably, your javascript is going to change, some day, and then you'll feel silly when all of your users are still running a three week old version of it that's still cached by their browsers.

The next bit is the hack that makes it all shine.

"Why not just change the filename when the file's contents change?" asked some bright engineer, somewhere. So, we do. We just attach a hash of the file to the filename and serve that. When the file changes, the filename we show the web browser changes, and we can tell the browser to just cache the file forever, or as close to forever as we can!

It's kind of a pain to generate a copy of all of your files with hashes attached, so what clack-static-asset-middleware does is store calculate some hashes of your files and just pretends those files exist and serves up whatever you tell it about.

I also wrote some helpers you can load to use this stuff easily with Djula, like so:

<a href="/" class="logo u-pull-left" title="Home">  <img src="{% asset-path "small-logo.png" %}"       width="54" height="60"       alt="Knttl" /></a>
So, to summarize:
1. Have style.css
2. Template asks for hashed filename, gets something like style_2867f3f83a6a91ad4a19a6cd45536152.css
3. Browser asks for style_2867f3f83a6a91ad4a19a6cd45536152.css
4. Server says "Aha! I have fooled you! Here's style.css! Keep it as long as you like!"
5. Browser says "Golly gee! I will, mister Server! I'll never ask for style_2867f3f83a6a91ad4a19a6cd45536152.css again!"
6. Hooray!
So check out clack-static-asset-middleware, which will hopefully be in quicklisp soon!

This was kind of fun to write!
posted on 2016-01-16 20:43:00

(defun zip (&rest sequences)  (apply #'map 'list (lambda (&rest args) (,@args)) sequences))

(zip '(1 2 3) '(2 3 4));; => ((1 2) (2 3) (3 4))
Making a knitting tool
posted on 2015-11-28 19:01:47
Hey there, friends!  Over the years, I've been aspiring to making interseting web apps in Common Lisp.  At the same time, Sarah's been kniting more and more and I've gotten to see some of that world.  Lots of people use Ravelry for posting their work and it's got a strong community.  I thought it might be interesting to try to make something that formatted knitting patterns in an easy to follow way.  Maybe eventually work up to some sort of markdown extension.  We'll see where it goes.

A few weeks ago, I stuck an early version up at knttl.co!  I'm sure there are all sorts of little bugs in it.  But it's fairly pretty and quick and runs on top of PostgreSQL and Common Lisp!  It's running Eitaro Fukamachi's woo web server, which is nice and snappy.  It's ona very low tier DigitalOcean VM and I haven't configured everything that well for the environment, so it crashes if it gets too much traffic.  While it's working, though, it's pretty darn quick.  I wish I had more time/effort available to work on it, but it's hard to find the motivation after full time programming work during the week.  If you find knttl interesting or useful, though, please let me know!
I have become Programmer
posted on 2015-07-11 23:54:00
As I mentioned last time, I've taken a new job in a new field. I've left academia, which I'd like to write more about, because it was important and difficult. Now, I'm working as a Software Engineer for Octopart, where I've been working in Python and Javascript, mostly doing fronted work.

It's been exciting doing something I've done for so long professionally. Octopart is a small team and it means I have a lot of freedom to work on things I think need work, but some of that is a bit disorienting, especially as someone working in the field for the first time. It's hard to know what expectations are in any situation, but people are trusting me to do what needs to be done and I'm working the confidence to believe my decisions about that are correct.

I also haven't really worked in Python before. It's always nice to learn a new language – especially with such a defined project – so that's been a plus. I am surprised again and again by how different languages handle importing files/packages/modules and how often it feels magic, weirdly difficult or some other sort of uncomfortable. Lisp has the strange duality of packages and ASDF systems, which aren't the same thing but often effectively are. Python has it's own difficulties, but in many ways is simpler. You pretty much just use paths with dots replacing slashes for folders. I'm not sure I'm a huge fan, but that's probably more stylistic than anything else. Even though python has a command line REPL and more sophisticated tools like IPython, I miss SLIME when I'm working in it.

Really, I miss lisp, generally. One downside of working in programming is that I have little enthusiasm left over after work for looking after my own projects. Hopefully, I'll get better at that in time. In any event, go check out Octopart if you're into electronics and let us know how we can make it more useful. It's made by a lot of nice people who care about what they're doing.
Cooking with websockets
posted on 2014-12-05 17:57:00
I finally got Chirp working with websockets! Well, it had worked before, but just sent down all the chirps that existed. Now it updates in real-time when someone else chirps. Though, it sends it to everyone, so don't say anything you'd want kept secret!

Eventually, I'd want to send messages only to followers and have them show up in the nav bar as notifications, but this was mostly meant as a proof of concept, anyway. I'd much rather go write a new sql library for lisp, now, that try to finish everything. At some point, paging chirps so you can scroll to fetch more would be good.

This was sort of my minimum product for this project, but I might polish it more as I have time.
Playing with the MOP
posted on 2014-12-01 20:30:00
I was poking around at my project, Chirp, today and finally decided to deal with a strange bug I was having. When I sent JSON down to the AngularJS controller that renders the page, dates were always coming back as the time the request was made at. This was strange, since the database records of the chirps store when they were created, and the time should have been that. The one I was testing was over a week old.

Eventually, I realized that in the slot definition

(created-at :type clsql:wall-time            :initform (clsql:get-time))

the time stamp would always be the current if the time wasn't set on the instance when the database was read.

I'd recently started using clsql-helper to get instances of classes back from complicated SQL queries. By 'complicated' I mean 'contains a basic join.' Apparently, CLSQL doesn't feel it needs to do that. I wish I knew why.

In any event, when I used clsql-helper, the dates were wrong, when I used clsql's built-in #'select, the dates were correct. So it slowly dawned on me that clsql-helper was doing something wrong. Sure enough, it wasn't converting the string that the time stamp returned into the wall-time instance that clsql expected.

So I took the chance to play around with the MOP a bit and wrote code to convert the values to the right type, looking at their slot definitions in the class.

(defun get-slot-by-name (name class)  (find name (closer-mop:class-direct-slots (find-class class))      :key (lambda (slot)             (intern (symbol-name (closer-mop:slot-definition-name slot))                     *package*))))(defun coerce-value-to-column-type (class column value)  (let ((type (clsql-sys::specified-type (get-slot-by-name column class))))    (coerce-value-to-db-type value type)))

With that code, you just do something like

(coerce-value-to-column-type class coulmn data)

and things come out the right way.
posted on 2013-07-08 10:33:06
Anyone who has bothered reading here will have some idea that I've been learning about making games lately.  I haven't written in a bit, because it's a lot of learning and I have very little that's finished enough to show off.  Yesterday, I got bored of working on my networking code, and decided to learn what these things called shaders were.  Luckily, I already knew about the CLinch framework, which had some shader support built in, so I could dive in without having to worry about linking and compiling them myself.

I played around a bit and made a spinning cube with some shading working, but man, having to work with C syntax again was frustrating.  Even more so was having to define the variables for the shader programs both in their explicit code and twice in the lisp that managed them as well.  That sort of work duplication is the sort of thing I think Lisp is well-suited to avoid, so I decided there needed to be yet another library for parsing lisp syntax into a lower-level language, so I started cl-glsl.

It's 5:30 am now, and it can already turn this:
(cl-glsl:translate  (defun main ()    (let ((intensity :float)	  (color     :vec4)	  (tex-color :vec4 (texture2D texture01 v-texture-coord)))      (setf intensity (dot light-dir (normalize normal)))      (cond	((> intensity 0.95)	 (setf color (* tex-color (vec4 1.0 1.0 1.0 1.0))))	((> intensity 0.5)	 (setf color (* tex-color (vec4 0.6 0.6 0.6 0.6))))	((> intensity 0.25)	 (setf color (* tex-color (vec4 0.4 0.4 0.4 0.4))))	(t	 (setf color (* tex-color (vec4 0.2 0.2 0.2 0.2)))))      (setf gl_frag-color color))))

into this:
void main () {    float intensity;    vec4 color;    vec4 texColor = texture2d(texture01, vTextureCoord);      intensity = dot(lightDir, normalize(normal));      if (intensity > 0.95) {        color = (texColor * vec4(1.0, 1.0, 1.0, 1.0));      } else if (intensity > 0.5) {        color = (texColor * vec4(0.6, 0.6, 0.6, 0.6));      } else if (intensity > 0.25) {        color = (texColor * vec4(0.4, 0.4, 0.4, 0.4));      } else {        color = (texColor * vec4(0.2, 0.2, 0.2, 0.2));      }      gl_FragColor = color;}

Which I'm going to call a resounding success and maybe go lie down now. Links to cl-glsl will appear here when and if it proves useful at all.

Good night!
CL-TGA
posted on 2013-04-15 20:27:06
Following up on my post with the oddly rendered marbles, I've reached a working version of cl-tga. See the README for instructions on using it with cl-opengl.  It's still lacking the ability to load certain types of TGA files, but is very usable as is!
Working on OpenGL in Lisp
posted on 2013-04-15 04:18:48
Recently, I've started working on a 3D game.  Something I've dreamed about for a while, and then saw a glimmer of in Artemis. I'll write more about that if I manage to get anywhere with it.

In support of making the game, I've been using the waveform file format (.obj) for things I'm working on in Blender. I'm not entirely happy with the obj loader, which is currently living on github, but still subject to tremendous revision.  To work with that, I've just begun working on cl-tga, which loads tga images.  It seems that obj files use mtl files to describe surfaces, and they in turn use tga files to hold the textures.  I haven't written anything to read a binary file format before, so it's been interesting.

It only took me a day to get things almost right, but they way in which it's currently wrong is amusing.

You'll notice that it's the same image repeated three times, but in each one the marbles are different shades.  For some reason, it's drawing the red, green, and blue as separate images.  Here's the file it's loading from:

As much as it's frustrating to write all of this infrastructure, I'm hoping it'll make the barrier lower for other people who want to use lisp for games like this.

Map Take 2: The Slicening
posted on 2012-11-30 22:05:50

It's been a while since I wrote about my scientific computing library for lisp: map.  Map is a great exercise for me -- in learning Lisp and learning algorithms and optimization.  Since I'm between jobs and not yet in grad school, the time has been ripe to work on the library with some toy cases and try to make it as usable as possible by the time I get back in a position where I could use it for real work.

Because of my copious free time, a lot has been changing lately, including the addition of matrix slices, which were important, and such important features as matrix inversion.  I'm also looking at adding parallelization at some point, as well as a matrix-defining macro in the style of the #v() one for vectors.

In any event, here's what's new:

Common Lisp Puzzle
posted on 2012-09-19 15:35:27
In the last week or so, a puzzle has shown up in one or two different places about a certain strange piece of Lisp code.  The code is as follows:
(let (#'1) (+ function 2))==> 3

It took me a bit to figure out, but the trick is what reader macros expand to.  Also, where certain macros are in the code.

Don't have it yet?  I'll give you a hint:
(let ('1) (+ quote 2))==> 3
Map
posted on 2012-08-22 21:26:44

For a while now, I've wanted to make some of the great functionality of things like Octave (and it's parent Matlab) to Common Lisp.  Mostly, I want things like the slice assignments (arrays with ranges in the indices) and matrix math.  Also, it never hurts to make higher mathematical functions available in simple ways.  So, I started writing Map, which I hope will do just that.

Lately, I've been working in singular optics, largely with Spatial Light Modulators, which are like very tiny LCD screens in front of a mirror. They're cool because they can transform the wavefront of a laser beam into any sort of beam you'd like!  We use a fair bit of math to pump out the right images to put onto the SLM, and I'd been wanting to make a more general interface for that for a while.  I'm not there yet, but I'd like to share the few things I've done so far.

## Vectors

In Octave, you can type

[1:10]
and it will automatically be expanded to
[1 2 3 4 5 6 7 8 9 10]
.  This is great notation and can be used both for generating a vector or setting a range.  In map, I've implemented those with
#r(1 10)
, which creates a 'range' class that just keeps track of the numbers, and #v(1 10), which will expand to
#(1 2 3 4 5 6 7 8 9 10)
, like the Octave code.  This means you can do things like
(reduce #'+ #v(1 100))

To add up the numbers 1 to 100.  That, of course, isn't hard, especially when Gauss came up with a much easier way, but you get the picture.

One further thing that's special, is that you can apply a function to the vector you're generating by passing another argument (you can also pass a step size by entering three numbers).

(reduce #' #v(1 .1 10 #'sin))

will give you the sum of the sine of the numbers 1, 1.1, ..., 9.9, 10!  Moving on...

## Differential Equations

This is a much more meaty section, and was really fun for me, since I hadn't really implemented a ODE solver before (that I remember).  Map now contains a Runge-Kutta Feldberg 45 solver, which is supposed to have an adaptive step size, and currently doesn't seem to, so I didn't do everything right, but it works to a certain accuracy.

Here's a demonstration of finding the angular position and velocity ($\theta, \omega$) of a damped, driven pendulum, taken from some of my old computational physics notes.

(defun damped-pendulum (ti yi)        (declare (type double-float ti)                 (type (array double-float 2))                 (ignore ti))        (vector (+ (* -1d0 (sin (elt yi 1))) (* -1d0 .5d0 (elt yi 0)))                (elt yi 0)))
(map::mute (multiple-value-setq (ts ys)         (map:rkf45 #'damped-driven-pendulum                #r(0d0 1d-7 4d1)            #(0d0 1d0) 1d-10)))

Now, ts holds the time values, and ys is a 2x(length ts) array of position and velocity values!  Once I implement array slices, it'll be easy to graph them, but for now, you need to manually extract one of the rows of ys to plot against ts. There's still a lot of work to be done. Anyway, the resulting graphs are:

Thanks for reading, I'll probably post more once I've got slices sorted out.

I should have known
posted on 2012-02-07 15:46:44
I was just checking out Shuffletron, which I found by way of this awesome article about writing an assembler for NES programs in lisp.  Apparently, Suffletron is a console-based music player written in Lisp!  Well, I love things written in Lisp, so I obviously needed to try this out.  I cloned the github repo, and was happy to see a familiar Makefile sitting in the new directory, chanted the required "make" incantation, and...
sbcl --noinform --no-userinit --disable-debugger \ --eval "(require :asdf)" \ --eval "(load \"build-sbcl.lisp\")" unhandled ASDF:MISSING-DEPENDENCY in thread #<SB-THREAD:THREAD "initial thread" RUNNING {AB80B69}>: component :MIXALOT-FLAC not found, required by #<SYSTEM "shuffletron">

De Novibus Locibus
posted on 2009-04-04 01:48:00

I'm proud to say that cl-facebook has a new life at common-list.net and I've set up a svn repository that's listed on that page. I hope someone finds it and starts using it. Maybe I'll have to get the word out.

De Libri Oris
posted on 2009-04-02 02:00:00

I have finally found a project for learning Common Lisp better. I've been implementing a lisp interface to facebook's REST API. I found some old code at common-lisp.net, which had lots of useful hashing/request forming code, but was using an out of date API and had no documentation.

I'm happy to say it is now much more useable. I applied for hosting on common-lisp.net. So, maybe I'll have a link to that for the interested soon

Unless otherwise credited all material by Matt Novenstern