Quantum Mechanic

Content tagged Lisp

Horrible Bug

posted on 2017-06-10 16:01:00

I just figured out the most horrible bug! For nearly two months, I've been watching my deploys of a lisp webapp say

Woo server is going to start.
Listening on localhost:5000.
Unhandled WOO.EV.CONDITION:OS-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
  getaddrinfo() failed (Code: -8)
Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1005C85A43}>

over and over and over.

Since I only find a little time to work on this stuff, around my day job, it was really pissing me off.

When I finally figured out what the problem was, I was even more pissed off.

It took me forever to try to figure out what that -8 error code was. At least I'd seen C code, before, to know that that was what it was. I must have miscounted, looking at strange webpages that listed the possible errors for getaddrinfo(), and that threw me off for a month alone. Today, I finally decided there wasn't some quick way to solve this, and I had to dig into the problem with due dilligence. I wrote a quick C program to get the description of the error from the horse's mouth:

#include <netdb.h>
#include <stdio.h>

int main(int argc, char** argv[]) {
  const char *explanation = gai_strerror(-8);
  printf("Error -8 is %s", explanation);
  return 0;

And it informed me: Error -8 is Servname not supported for ai_socktype. A quick search led me to this stackoverflow, which had a comment I include here for your convenience:

I had this problem with a Tornado/Python app. Apparently, this can be caused by the port being interpreted as a string instead of an integer. So in my case, I needed to change my startup script to force it to be interpreted as an integer.

This turned out to be precisely my problem.

I had written my development configuration as a p-list, which, being readable could contain numbers. In production, I was reading configuration out of environment variables, which are all strings. So, pop in an if statement with a (parse-integer port) and blam.

Two months of hobby time, down the toilet.

I'm gonna have to see if I can make a patch for woo or something to make sure this doesn't happen, again.

An afternoon hack

posted on 2016-12-05 14:42:00

A few months ago, I decided that lucerne had enough things missing that I was rewriting for different projects, that I would bite the bullet and try writing my own framework. I'm generally a fan of Fernando Borretti's work, and I definitely suggest looking at the different libraries he's created for common lisp webapps, but these days he's been more of an inspiration for writing my own versions than anything else.

So, I've been writing nest for a while, now, which doesn't have much documentation and very spotty test coverage, at the moment, because it is still very much in the process of spilling out of my forehead, like some halting Athena, and I wouldn't want to break anyone else's webapps while I work on it. (I am trying to version and tag it, though, so at least you'll get the same sort of broken if you stick to a tag.)

I was looking through fossdroid.com, yesterday, and noticed the shopping list app, which boasted an open source php and mysql backend. It comes with some script to set it up under apache, but the list of technologies I wasn't very familiar with was too long, so I decided to test out my framework to get it running under lisp (and whatever database you want that's supported by cl-dbi).

Happily, I didn't bite off more than I could chew on this one, and I had it all working by midnight so I could sleep, too!

I found the api very strange, because it's all implemented at one endpoint. The android app also adds a trailing slash to whatever you configure as the host, which was odd, because I would get requests to /api.php/ or // depending on how I configured it. The app sends what function it wants to run as a post parameter, and then, hopefully, all the parameters appropriate for that function are along for the ride, too. It was a good excuse to pull out my validation library for a ride, which I generally use with a macro like

(defmacro validated (&body body)
  "Handle validation errors (`v:<validation-error>`) and respond with an error for the client."

       (progn ,@body)
     (v:<validation-error> (e)
       (declare (ignore e))
       (api-response :type +api-error-unknown+
                     :content "Invalid parameter"))))

I've also started using validate in Podcastinet, with a similar macro that sends back a 400 error in the event validation fails. Right now, there isn't a good way to collect failed validations for a meaningful error for the API consumer, but I suppose that will come in time.

In any event, it was a fun project and made me feel pretty good about nest, since I was able to use it to build an app in an afternoon. Now I need to make a docker image so I can throw it up on a server and keep track of my groceries! Check out the project here and let me know what you think. I'd also really appreciate feedback on validate. Maybe I'll post it to the lisp subreddit, soon, for some of that.

Clack Static Asset Middleware

posted on 2016-05-12 02:01:00

As part of my glacial march to get Knttl running the way I'd like, I wrote a middleware for the clack web framework to help me out!

When you're serving javascript and css files for a website, you want to inform the browser to keep a local copy as long as possible, because then it won't bother your webserver to download it again and the page will render faster for your users, too. There's a popular quote about programming

There are only two hard things in Computer Science: cache invalidation and naming things.
It seems this was said by a Phil Karlton.

So, like many things related to the it-was-never-meant-to-be-used-this-way world of web technology, there are some interesting best practices that evolved around this idea that, ideally, only one copy of a static file should ever need to be sent to a user, if their browser works well.

There are ways to tell a browser how long to hold onto a file before checking if there's a new version of it, but that only gets you so far. Presumably, your javascript is going to change, some day, and then you'll feel silly when all of your users are still running a three week old version of it that's still cached by their browsers.

The next bit is the hack that makes it all shine.

"Why not just change the filename when the file's contents change?" asked some bright engineer, somewhere. So, we do. We just attach a hash of the file to the filename and serve that. When the file changes, the filename we show the web browser changes, and we can tell the browser to just cache the file forever, or as close to forever as we can!

It's kind of a pain to generate a copy of all of your files with hashes attached, so what clack-static-asset-middleware does is store calculate some hashes of your files and just pretends those files exist and serves up whatever you tell it about.

I also wrote some helpers you can load to use this stuff easily with Djula, like so:

<a href="/" class="logo u-pull-left" title="Home">
<img src="{% asset-path "small-logo.png" %}"
width="54" height="60"
alt="Knttl" />
So, to summarize:
  1. Have style.css
  2. Template asks for hashed filename, gets something like style_2867f3f83a6a91ad4a19a6cd45536152.css
  3. Browser asks for style_2867f3f83a6a91ad4a19a6cd45536152.css
  4. Server says "Aha! I have fooled you! Here's style.css! Keep it as long as you like!"
  5. Browser says "Golly gee! I will, mister Server! I'll never ask for style_2867f3f83a6a91ad4a19a6cd45536152.css again!"
  6. Hooray!
So check out clack-static-asset-middleware, which will hopefully be in quicklisp soon!

This was kind of fun to write!

posted on 2016-01-16 20:43:00

(defun zip (&rest sequences)
(apply #'map 'list (lambda (&rest args) `(,@args)) sequences))

(zip '(1 2 3) '(2 3 4))
;; => ((1 2) (2 3) (3 4))

Making a knitting tool

posted on 2015-11-28 19:01:47

Hey there, friends!  Over the years, I've been aspiring to making interseting web apps in Common Lisp.  At the same time, Sarah's been kniting more and more and I've gotten to see some of that world.  Lots of people use Ravelry for posting their work and it's got a strong community.  I thought it might be interesting to try to make something that formatted knitting patterns in an easy to follow way.  Maybe eventually work up to some sort of markdown extension.  We'll see where it goes.

A few weeks ago, I stuck an early version up at knttl.co!  I'm sure there are all sorts of little bugs in it.  But it's fairly pretty and quick and runs on top of PostgreSQL and Common Lisp!  It's running Eitaro Fukamachi's woo web server, which is nice and snappy.  It's ona very low tier DigitalOcean VM and I haven't configured everything that well for the environment, so it crashes if it gets too much traffic.  While it's working, though, it's pretty darn quick.  I wish I had more time/effort available to work on it, but it's hard to find the motivation after full time programming work during the week.  If you find knttl interesting or useful, though, please let me know!

I have become Programmer

posted on 2015-07-11 23:54:00

As I mentioned last time, I've taken a new job in a new field. I've left academia, which I'd like to write more about, because it was important and difficult. Now, I'm working as a Software Engineer for Octopart, where I've been working in Python and Javascript, mostly doing fronted work.

It's been exciting doing something I've done for so long professionally. Octopart is a small team and it means I have a lot of freedom to work on things I think need work, but some of that is a bit disorienting, especially as someone working in the field for the first time. It's hard to know what expectations are in any situation, but people are trusting me to do what needs to be done and I'm working the confidence to believe my decisions about that are correct.

I also haven't really worked in Python before. It's always nice to learn a new language – especially with such a defined project – so that's been a plus. I am surprised again and again by how different languages handle importing files/packages/modules and how often it feels magic, weirdly difficult or some other sort of uncomfortable. Lisp has the strange duality of packages and ASDF systems, which aren't the same thing but often effectively are. Python has it's own difficulties, but in many ways is simpler. You pretty much just use paths with dots replacing slashes for folders. I'm not sure I'm a huge fan, but that's probably more stylistic than anything else. Even though python has a command line REPL and more sophisticated tools like IPython, I miss SLIME when I'm working in it.

Really, I miss lisp, generally. One downside of working in programming is that I have little enthusiasm left over after work for looking after my own projects. Hopefully, I'll get better at that in time. In any event, go check out Octopart if you're into electronics and let us know how we can make it more useful. It's made by a lot of nice people who care about what they're doing.

Cooking with websockets

posted on 2014-12-05 17:57:00

I finally got Chirp working with websockets! Well, it had worked before, but just sent down all the chirps that existed. Now it updates in real-time when someone else chirps. Though, it sends it to everyone, so don't say anything you'd want kept secret!


Eventually, I'd want to send messages only to followers and have them show up in the nav bar as notifications, but this was mostly meant as a proof of concept, anyway. I'd much rather go write a new sql library for lisp, now, that try to finish everything. At some point, paging chirps so you can scroll to fetch more would be good.

This was sort of my minimum product for this project, but I might polish it more as I have time.

Playing with the MOP

posted on 2014-12-01 20:30:00

I was poking around at my project, Chirp, today and finally decided to deal with a strange bug I was having. When I sent JSON down to the AngularJS controller that renders the page, dates were always coming back as the time the request was made at. This was strange, since the database records of the chirps store when they were created, and the time should have been that. The one I was testing was over a week old.

Eventually, I realized that in the slot definition

(created-at :type clsql:wall-time
:initform (clsql:get-time))

the time stamp would always be the current if the time wasn't set on the instance when the database was read.

I'd recently started using clsql-helper to get instances of classes back from complicated SQL queries. By 'complicated' I mean 'contains a basic join.' Apparently, CLSQL doesn't feel it needs to do that. I wish I knew why.

In any event, when I used clsql-helper, the dates were wrong, when I used clsql's built-in #'select, the dates were correct. So it slowly dawned on me that clsql-helper was doing something wrong. Sure enough, it wasn't converting the string that the time stamp returned into the wall-time instance that clsql expected.

So I took the chance to play around with the MOP a bit and wrote code to convert the values to the right type, looking at their slot definitions in the class.

(defun get-slot-by-name (name class)
(find name (closer-mop:class-direct-slots (find-class class))
:key (lambda (slot)
(intern (symbol-name (closer-mop:slot-definition-name slot))

(defun coerce-value-to-column-type (class column value)
(let ((type (clsql-sys::specified-type (get-slot-by-name column class))))
(coerce-value-to-db-type value type)))

With that code, you just do something like

(coerce-value-to-column-type class coulmn data)

and things come out the right way.

Cooking with shaders

posted on 2013-07-08 10:33:06

Anyone who has bothered reading here will have some idea that I've been learning about making games lately.  I haven't written in a bit, because it's a lot of learning and I have very little that's finished enough to show off.  Yesterday, I got bored of working on my networking code, and decided to learn what these things called shaders were.  Luckily, I already knew about the CLinch framework, which had some shader support built in, so I could dive in without having to worry about linking and compiling them myself.

I played around a bit and made a spinning cube with some shading working, but man, having to work with C syntax again was frustrating.  Even more so was having to define the variables for the shader programs both in their explicit code and twice in the lisp that managed them as well.  That sort of work duplication is the sort of thing I think Lisp is well-suited to avoid, so I decided there needed to be yet another library for parsing lisp syntax into a lower-level language, so I started cl-glsl.

It's 5:30 am now, and it can already turn this:
(defun main ()
(let ((intensity :float)
(color :vec4)
(tex-color :vec4 (texture2D texture01 v-texture-coord)))
(setf intensity (dot light-dir (normalize normal)))

((> intensity 0.95)
(setf color (* tex-color (vec4 1.0 1.0 1.0 1.0))))
((> intensity 0.5)
(setf color (* tex-color (vec4 0.6 0.6 0.6 0.6))))
((> intensity 0.25)
(setf color (* tex-color (vec4 0.4 0.4 0.4 0.4))))
(setf color (* tex-color (vec4 0.2 0.2 0.2 0.2)))))

(setf gl_frag-color color))))

into this:
void main () {
float intensity;
vec4 color;
vec4 texColor = texture2d(texture01, vTextureCoord);

intensity = dot(lightDir, normalize(normal));
if (intensity > 0.95) {
color = (texColor * vec4(1.0, 1.0, 1.0, 1.0));
} else if (intensity > 0.5) {
color = (texColor * vec4(0.6, 0.6, 0.6, 0.6));
} else if (intensity > 0.25) {
color = (texColor * vec4(0.4, 0.4, 0.4, 0.4));
} else {
color = (texColor * vec4(0.2, 0.2, 0.2, 0.2));
gl_FragColor = color;


Which I'm going to call a resounding success and maybe go lie down now. Links to cl-glsl will appear here when and if it proves useful at all.

Good night!


posted on 2013-04-15 20:27:06

Following up on my post with the oddly rendered marbles, I've reached a working version of cl-tga. See the README for instructions on using it with cl-opengl.  It's still lacking the ability to load certain types of TGA files, but is very usable as is!

Working on OpenGL in Lisp

posted on 2013-04-15 04:18:48

Recently, I've started working on a 3D game.  Something I've dreamed about for a while, and then saw a glimmer of in Artemis. I'll write more about that if I manage to get anywhere with it.

In support of making the game, I've been using the waveform file format (.obj) for things I'm working on in Blender. I'm not entirely happy with the obj loader, which is currently living on github, but still subject to tremendous revision.  To work with that, I've just begun working on cl-tga, which loads tga images.  It seems that obj files use mtl files to describe surfaces, and they in turn use tga files to hold the textures.  I haven't written anything to read a binary file format before, so it's been interesting.

It only took me a day to get things almost right, but they way in which it's currently wrong is amusing.

cl-tga screenshotYou'll notice that it's the same image repeated three times, but in each one the marbles are different shades.  For some reason, it's drawing the red, green, and blue as separate images.  Here's the file it's loading from:



As much as it's frustrating to write all of this infrastructure, I'm hoping it'll make the barrier lower for other people who want to use lisp for games like this.


Map Take 2: The Slicening

posted on 2012-11-30 22:05:50

It's been a while since I wrote about my scientific computing library for lisp: map.  Map is a great exercise for me -- in learning Lisp and learning algorithms and optimization.  Since I'm between jobs and not yet in grad school, the time has been ripe to work on the library with some toy cases and try to make it as usable as possible by the time I get back in a position where I could use it for real work.

Because of my copious free time, a lot has been changing lately, including the addition of matrix slices, which were important, and such important features as matrix inversion.  I'm also looking at adding parallelization at some point, as well as a matrix-defining macro in the style of the #v() one for vectors.

In any event, here's what's new:


I promised to write again when I had finished with matrix slices in map.  I'm sure they're not in their final state, but I have implemented them in a way that works. Now you can do things like:
(setf (mref m 1 #r(1 2) t) #2a((2.0d0 1.0d0 3.4d0) (3.1d0 3.2d2 4.2d1)))

Which already has a number of things going on in it.  This is setting a sub-matrix of m, a rank-3 matrix to a rank-2 matrix specified in the end of  the setf call.  Since we selected a single value for the first index of m, mref will not count that dimension as extant for the purposes of comparing with the shape of the matrix we're assigning.  This is really useful, because now things like multiplying matrices can be done like this:
(defmethod .* ((A simple-array) (B simple-array))
(with-result (result (list (array-dimension A 0) (array-dimension B 1)))
(do-matrix (result (i j))
(setf (aref result i j) (dot (mref a i t) (mref b t j))))))

There are three different ways you can specify subscripts to mref:

  1. Integers
    This will work like aref, and the dimension won't be included in the final output

  2. Ranges
    This will select all entries in the matrix that have indices inside the specified range in this dimension, for example, (mref #(1 2 3) #r(0 1)) ==> #(1 2)

  3. t
    This selects all elements in this dimension.  It's great for slicing rows/columns out of matrices a la (mref m t 0), which will take the first column out of m.

Matrix iteration

Since most of the functions in the library were operating on matrices and returning new ones, I thought some macros were in order.  I think it's the reflex of any lisper to write a macro when they see the same code in more than one place.

Of that urge were born (with-result) and (do-matrix)(with-result) is simply a convenient way of defining a new matrix at the start of a function and automatically returning it.  (do-matrix), though, I rather like because it allows looping over matrices in a rather intuitive way.

(defun mandelbrot (matrix)
(with-result (result (list 1001 1001) 'integer)
(do-matrix (matrix (r i))
(setf (aref result r i)
with c = (aref matrix r i)
with z = c
for i from 0 upto 80
if (> (abs z) 2)
return (1- i)
do (setf z (+ (expt z 2) c))
finally (return 80))))))

Here you can see (do-matrix) at work, testing whether each element of an array of complex numbers is in the Mandelbrot set.  It's definition is
(defmacro do-matrix (matrix &optional subscripts) ...)

which means you can then just write element-wise operations inside the loop with all of the indices figured out for you.

I'll end with the figure I made of the Mandelbrot set today on the [-2, 2] range in the complex plane. Along with the (mandelbrot) function above, you need the matrix to call it on,

(setq cm
(let ((center 500))
(with-result (result (list 1001 1001) '(complex double-float))
(do-matrix (matrix (r i))
(setf (aref matrix r i)
(complex (* (- r center) (/ 2.0d0 500))
(* (- i center) (/ 2.0d0 500)))))))))

and then you can run
(mute (setq r (mandelbrot cm)))
(image r)


The Mandelbrot set from -2 to 2 in the complex plane.
The Mandelbrot set from -2 to 2 in the complex plane.

Common Lisp Puzzle

posted on 2012-09-19 15:35:27

In the last week or so, a puzzle has shown up in one or two different places about a certain strange piece of Lisp code.  The code is as follows:
(let (#'1) (+ function 2))
==> 3


It took me a bit to figure out, but the trick is what reader macros expand to.  Also, where certain macros are in the code.

Don't have it yet?  I'll give you a hint:
(let ('1) (+ quote 2))
==> 3


posted on 2012-08-22 21:26:44

For a while now, I've wanted to make some of the great functionality of things like Octave (and it's parent Matlab) to Common Lisp.  Mostly, I want things like the slice assignments (arrays with ranges in the indices) and matrix math.  Also, it never hurts to make higher mathematical functions available in simple ways.  So, I started writing Map, which I hope will do just that.

Lately, I've been working in singular optics, largely with Spatial Light Modulators, which are like very tiny LCD screens in front of a mirror. They're cool because they can transform the wavefront of a laser beam into any sort of beam you'd like!  We use a fair bit of math to pump out the right images to put onto the SLM, and I'd been wanting to make a more general interface for that for a while.  I'm not there yet, but I'd like to share the few things I've done so far.


In Octave, you can type

and it will automatically be expanded to
[1 2 3 4 5 6 7 8 9 10]
.  This is great notation and can be used both for generating a vector or setting a range.  In map, I've implemented those with
#r(1 10)
, which creates a 'range' class that just keeps track of the numbers, and #v(1 10), which will expand to
#(1 2 3 4 5 6 7 8 9 10)
, like the Octave code.  This means you can do things like
(reduce #'+ #v(1 100))

To add up the numbers 1 to 100.  That, of course, isn't hard, especially when Gauss came up with a much easier way, but you get the picture.

One further thing that's special, is that you can apply a function to the vector you're generating by passing another argument (you can also pass a step size by entering three numbers).

(reduce #' #v(1 .1 10 #'sin))

will give you the sum of the sine of the numbers 1, 1.1, ..., 9.9, 10!  Moving on...

Differential Equations

This is a much more meaty section, and was really fun for me, since I hadn't really implemented a ODE solver before (that I remember).  Map now contains a Runge-Kutta Feldberg 45 solver, which is supposed to have an adaptive step size, and currently doesn't seem to, so I didn't do everything right, but it works to a certain accuracy.

Here's a demonstration of finding the angular position and velocity ($\theta, \omega$) of a damped, driven pendulum, taken from some of my old computational physics notes.

(defun damped-pendulum (ti yi)
        (declare (type double-float ti)
(type (array double-float 2))
(ignore ti))
        (vector (+ (* -1d0 (sin (elt yi 1))) (* -1d0 .5d0 (elt yi 0)))
            (elt yi 0)))
(map::mute (multiple-value-setq (ts ys)
(map:rkf45 #'damped-driven-pendulum
#r(0d0 1d-7 4d1)
#(0d0 1d0) 1d-10)))

Now, ts holds the time values, and ys is a 2x(length ts) array of position and velocity values!  Once I implement array slices, it'll be easy to graph them, but for now, you need to manually extract one of the rows of ys to plot against ts. There's still a lot of work to be done. Anyway, the resulting graphs are:

Angular velocity, $\omega$.
Angular Position, $\theta$.

Thanks for reading, I'll probably post more once I've got slices sorted out.

I should have known

posted on 2012-02-07 15:46:44

I was just checking out Shuffletron, which I found by way of this awesome article about writing an assembler for NES programs in lisp.  Apparently, Suffletron is a console-based music player written in Lisp!  Well, I love things written in Lisp, so I obviously needed to try this out.  I cloned the github repo, and was happy to see a familiar Makefile sitting in the new directory, chanted the required "make" incantation, and...
sbcl --noinform --no-userinit --disable-debugger \
--eval "(require :asdf)" \
--eval "(load \"build-sbcl.lisp\")"
"initial thread" RUNNING
component :MIXALOT-FLAC not found, required by #<SYSTEM "shuffletron">

What happened first

So, apparently I'm missing this mixalotlibrary.  Oh no! I never figured out how asdf worked well enough to put all of mixalot's asd files in the right places... Something about

and er...

What should have happened first

Oh, I need this mixalot library.
(ql:quickload "mixalot")

What followed and, ultimately, failed to happen

Unfortunately, after the mixalot stage, I had a lot of trouble getting mixalot-flac, which quicklisp happily loaded in another sbcl which was in the mixalot source directory, and then refused to load through a number of different means until I
(push #p"/path/to/mixalot" asdf:*central-registry*)

'ed and loaded

though asdf.

Then, I needed to install libmpg123, which I didn't seem to have.

Then, I got errors about a library called "gen0.so" that the build script couldn't open. Being named gen0, I had no idea what it was. I tried the 'continue' option from sbcl, and it built, but I was afraid. Suffletron opened, asked me for my library's path, and then dropped into the debugger because of an unknown library call, I assume, to gen0.

Perhaps some day, I'll get this working.

De Novibus Locibus

posted on 2009-04-04 01:48:00

I'm proud to say that cl-facebook has a new life at common-list.net and I've set up a svn repository that's listed on that page. I hope someone finds it and starts using it. Maybe I'll have to get the word out.

De Libri Oris

posted on 2009-04-02 02:00:00

I have finally found a project for learning Common Lisp better. I've been implementing a lisp interface to facebook's REST API. I found some old code at common-lisp.net, which had lots of useful hashing/request forming code, but was using an out of date API and had no documentation.

I'm happy to say it is now much more useable. I applied for hosting on common-lisp.net. So, maybe I'll have a link to that for the interested soon