# Migrating Pigeons

posted on 2018-04-02 11:00

About a month ago, I converted Podcastinet from using mito to postmodern as the database interface. One of the things I wanted was to have finer-grained control over the database. I was relying on mito's ability to automtaically migrate based on database introspection, which was useful, but also didn't allow migrating data between columns or other, more manual sorts of migrations. To enable that sort of thing, I wrote Postmodern Passenger Pigeon, which I still need to document/write a README for, but is basically functional.

It creates a table in your database named pigeon_migrations that stores a bunch of revisions and it works based on a linear idea of your database's state. That is, there are no branches in the revisions, it's a list, not a tree.

It can be configured with a pigeon.toml file that mostly just takes a migration-directory (but can also take a database-url, though it will look at the environment variable DATABASE_URL, too). And then you have a bunch of files in that directory named like [universal-time]-[hypenized-name].lisp that house the actual revisions.

For example:

3727984190-create-people-table.lisp:

"Creates the people table."

(defun up ()
(query (:create-table people
((id :type integer :primary-key t)
(name :type (string 20))))))

(defun down ()
(query (:drop-table 'people)))

The file can have a docstring at the top to describe the revision in more detail, and then expects an up and a down function to be defined. (The revisions are each evaluated in their own sandbox package, so you can refer to other packages by their full name, though I don't recommend it.) Postmodern is used by default.

## Creating migrations

If you want to create a new revision, you can call (ppp.migration:new "Short description of my migration") and it will create the new file for you. If you decide that you don't need a migration you're working on, just revert the migration and delete the file and you can go ahead and make new ones. They're applied in numerical order of the timestamp.

## Running migrations

To apply or revert migrations, use the ppp:migrate function. The first argument is the direction to migrate in: :up will apply new migrations, :down will revert existing migrations. You can give a number as the second argument to say how many you'd like to apply or revert, i.e. (ppp:migrate :up 1) will apply the next migration that hasn't been applied, and no more, (ppp:migrate :down 3) will revert 3 migrations, starting will the most recently applied.

There are also the special values :head and :base which mean "all the migrations applied" and "no migrations applied" (which should mean an empty db schema). In my site, I run (ppp:migrate :up :head) when it starts up to apply any migrations that came with the latest code push.

# Horrible Bug

posted on 2017-06-10 16:01:00

I just figured out the most horrible bug! For nearly two months, I've been watching my deploys of a lisp webapp say

Woo server is going to start.
Listening on localhost:5000.
{1005C85A43}>:
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<WOO.EV.CONDITION:OS-ERROR {1007BD7533}> #<unused argument>)

over and over and over.

Since I only find a little time to work on this stuff, around my day job, it was really pissing me off.

When I finally figured out what the problem was, I was even more pissed off.

It took me forever to try to figure out what that -8 error code was. At least I'd seen C code, before, to know that that was what it was. I must have miscounted, looking at strange webpages that listed the possible errors for getaddrinfo(), and that threw me off for a month alone. Today, I finally decided there wasn't some quick way to solve this, and I had to dig into the problem with due dilligence. I wrote a quick C program to get the description of the error from the horse's mouth:

#include <netdb.h>
#include <stdio.h>

int main(int argc, char** argv[]) {
const char *explanation = gai_strerror(-8);
printf("Error -8 is %s", explanation);
return 0;
}

And it informed me: Error -8 is Servname not supported for ai_socktype. A quick search led me to this stackoverflow, which had a comment I include here for your convenience:

I had this problem with a Tornado/Python app. Apparently, this can be caused by the port being interpreted as a string instead of an integer. So in my case, I needed to change my startup script to force it to be interpreted as an integer.

This turned out to be precisely my problem.

I had written my development configuration as a p-list, which, being readable could contain numbers. In production, I was reading configuration out of environment variables, which are all strings. So, pop in an if statement with a (parse-integer port) and blam.

Two months of hobby time, down the toilet.

I'm gonna have to see if I can make a patch for woo or something to make sure this doesn't happen, again.

# Rally Season

posted on 2017-02-03 18:00

We just got back from a rally in the neighborhood. Unfortunately, we our country is becoming a place where Muslims are made to feel unsafe and the president turns back people who have come to us as guests. Thankfully, our local officials in Astoria, Queens, were clear-headed enough to organize a rally in support of the Muslim community in our neighborhood, so we went out to that tonight.

It felt good to be part of a small, community organized event instead of a larger march like the one supporting women that happened a week or two ago. Those tend to feel less directed and impactful to me. Too many causes spoils the soup, as it were. I'm not sure what I can do to help, but maybe I'll put some time into looking into it, this weekend.

# The Esperantist

posted on 2016-12-05 19:31:00

Mi ĵus trovis la gazeton "The Esperantist" sur Project Gutenberg, kiu estis en Londonon eldonita Gazeto el la lasta jarcento. Kelkaj monatoj auntaŭe, mi trovis la Esperanto kurson sur Duolingo, kaj mi havis interson ekde mi eklernis pri E-on kiam mi en gimnazio estis. Tiam, trovis mi profesoron el Brazilo en Babilejo kaj mi ĝojis kunparoli iun malkun anglan lingvon. Nun, mi retrovas la ĝojon por legi E-on kaj partopreni internacian komunumon.

La paĝo de la redaktulo de la unua aldono ekiĝas kun promeso ke la gazeto ne intencas esti politika. Mi interesiĝis kiam mi tion legis, ĉar E-o ankoraŭ ŝajnas por multaj homoj minaci siajn hejmlingvojn kaj kulturojn. Multoj vidas E-on kiel komunista konspiro (aŭ, alternative juda konspiro).

Ĉiukaze, mi ne kredeble legos tre multe, sed la gazeto estas aĵo el pli frua tempo, kaj interesigas min pri kiel E-on vidis gehomoj en la estinteco.

# An afternoon hack

posted on 2016-12-05 14:42:00

A few months ago, I decided that lucerne had enough things missing that I was rewriting for different projects, that I would bite the bullet and try writing my own framework. I'm generally a fan of Fernando Borretti's work, and I definitely suggest looking at the different libraries he's created for common lisp webapps, but these days he's been more of an inspiration for writing my own versions than anything else.

So, I've been writing nest for a while, now, which doesn't have much documentation and very spotty test coverage, at the moment, because it is still very much in the process of spilling out of my forehead, like some halting Athena, and I wouldn't want to break anyone else's webapps while I work on it. (I am trying to version and tag it, though, so at least you'll get the same sort of broken if you stick to a tag.)

I was looking through fossdroid.com, yesterday, and noticed the shopping list app, which boasted an open source php and mysql backend. It comes with some script to set it up under apache, but the list of technologies I wasn't very familiar with was too long, so I decided to test out my framework to get it running under lisp (and whatever database you want that's supported by cl-dbi).

Happily, I didn't bite off more than I could chew on this one, and I had it all working by midnight so I could sleep, too!

I found the api very strange, because it's all implemented at one endpoint. The android app also adds a trailing slash to whatever you configure as the host, which was odd, because I would get requests to /api.php/ or // depending on how I configured it. The app sends what function it wants to run as a post parameter, and then, hopefully, all the parameters appropriate for that function are along for the ride, too. It was a good excuse to pull out my validation library for a ride, which I generally use with a macro like

(defmacro validated (&body body)
"Handle validation errors (v:<validation-error>) and respond with an error for the client."

(handler-case
(progn ,@body)
(v:<validation-error> (e)
(declare (ignore e))
(api-response :type +api-error-unknown+
:content "Invalid parameter"))))


I've also started using validate in Podcastinet, with a similar macro that sends back a 400 error in the event validation fails. Right now, there isn't a good way to collect failed validations for a meaningful error for the API consumer, but I suppose that will come in time.

In any event, it was a fun project and made me feel pretty good about nest`, since I was able to use it to build an app in an afternoon. Now I need to make a docker image so I can throw it up on a server and keep track of my groceries! Check out the project here and let me know what you think. I'd also really appreciate feedback on validate. Maybe I'll post it to the lisp subreddit, soon, for some of that.

# Fixing our Monoprice Mini Select

posted on 2016-11-26 01:36

A couple months ago, I ordered a Monoprice Mini Select 3D printer. I'd been hearing the buzz for years and at around \$200, it seemed like it was finally an offering with good reception that I could feel fairly safe dropping the money on. We printed Millenilm Falcons and a test elephant model that came with it, but then it got relegated to a shelf next to our bed for the last few months while things were stressful and creativity was wanting.

This weekend, we broke it out so we could try to do something fun with it again. We'd never been able to get it to read an SD card, so we controlled it with a laptop that has since retired and it took me two hours of poking at linux serial point things to realize that I'd just updated my kernel and it seemed like something was having trouble mounting serial devices because of it. A reboot later, we were in business.

Unfortunately, that wasn't the end of that, because our printer started behaving strangely; taking lots of pauses for seconds at a time. It was causing beading in the plastic where it stopped, and was seriously lowering the quality of the builds. I'd long known there were firmware updates to be had at the Google Drive for the printer, but without a working SD card, I couldn't do the updates and we were seriously considering having to call customer support.

Thankfully, digging through random boxes bore fruit and I was able to located two more micro SD cards, one of which pleased the printer. Finally, I was able to update it.

Before we updated the firmware, the printer got into a strange state and ran itself to the extremes of the X and Y axes and then tried to go further. When that happened, something happened and the extruder temperature started showing as 0.

0 Celsius is pretty unusual for indoors, even in late Fall in New York, so I figured something was wrong with that sensor. Sure enough, the little wire connected to the thermometer had snapped. Probably, while it wires were stretched to the edge of the print volume.

Luckily, with a quick run for overpriced solder at Radio Shack, I was able to patch it back up and get back to printing.

# Getting my Nexus 10 to charge again

posted on 2016-10-26 17:33:48

Sarah started a masters in TESOL, recently, and we wanted to give her something to use in class that she could carry around easily that would take some basic notes and be useful for readings.  We'd had a Nexus 10 sitting around that I'd meant to use for my aborted grad school experience and it seemed like the best solution for this situation, too.

Recently, though, it stopped turning on and would only ever show us the charging symbol for a short time.  Today, I sat down and finally did some research into it and the simplest/likeliest cause seemed to be that the battery had just discharged beyond the point that the battery controller would even allow it to be charged.  I found some help in this blog post, which suggested disconnecting the battery from its controller for a short time to reset it.

I had a little trouble pulling the frame off the screen/device assembly, so the tip I would add is that the seam you're looking for is the one between the screen and its plastic bevel, not anything along the side of the device.  I spent a while looking for that.  It was pretty tough to remove it, and I think I shot myself in the foot by trying on two different sides and getting it part-way off, which probably meant I was pulling against myself on each side.  Maybe going from one side and working my way around the device would be better.

I just checked back in and I turned it on for the first time in a month or so!  Hooray!

# Our Bodies, Our Hearts, Our Software

posted on 2016-08-23 22:10:04

I don't really keep up with the various Linux conferences that happen now and then, but I certainly follow various free software planets, which generally filter the good stuff out for me anyway.  Something that slowly made it through my filters recently was Karen Sandler's keynote address at the Australian Linux Conference, where she, unsurprisingly, talked about the importance of FOSS software.  The part that was surprising, though, and really hit home for me, was when she started talking about her internal defibrillator.

Apparently, Karen has an abnormally large heart, in the physical sense.  She is at risk of suffering what her doctors called "sudden death," which means she could simply die with no warning.  Luckily for her, there is a great treatment in the form of the internal defibrillator - a small device that can be wired directly into her heart that will issue a shock to restart her it if she suddenly, well, dies.

I'm going to take this moment to urge you to go watch the video, it's a bit long, but listening to her talk about her own heart is going to do more for you than I can manage.

For those of you not watching the video, there are two scary parts.  First - when she talks about being at risk of sudden death.  The second is the sequence building from her cardiologist not knowing much about the device he was offering to implant to the point where he hangs up on her in frustration.  His frustration came from her desire to see the code running on the device he wanted to put in her, the company's unwillingness to show her the code, and his lack of understanding about why that's important.  If you're wondering why this sort of thing is important at all, it's because this machine is wired to her heart.

## What we put into our bodies...

When we go out to dinner somewhere, we expect to be told what's in our food if we ask, we expect to know what's in the drugs we take, so why shouldn't we expect access to what's in the devices implanted in our bodies?

Let me phrase it another way.  A company makes an electric heart that you need because yours was stabbed like Jean-Luc Picard was in Star Trek.  You get it and want to know what makes you tick, and they tell you that your heart is their intellectual property and you're not allowed to know how it works. Furthermore, you're not allowed to modify it, open it up, or see its source code.  There's a well-known bug in it?  There's a firmware update?  You might not get it if it doesn't make money for the company.  You want to install it yourself?  That might be illegal depending on the time of year and what phase the moon is in.

When Karen told her story, her concerns were that she couldn't check the logic of her device, herself, nor could she have a friend or consumer-advocacy group do it.  Four years ago, when I started writing this, that was where the story ended.

## Things that come out of our bodies

A few weeks ago, I came across this article, which I thought might be a bout Karen, again.  I opened it, hoping to read more about her and, hopefully, what progress she had made.  Instead, I learned about a brave, new person taking up this baton, because the same problems were still around.  It turned out that, on top of Karen's original concerns, people were now taking data from Marie Moe's device and refusing to give her access to it.

At first, as I read that they were collecting the data, I hoped it was available to her.  Wouldn't that be cool?  I could make graphs of my heart rate, if I had that data.  But they seemed to think that the data was theirs.  Her doctor seemed to be too busy to help her get at it either, which isn't surprising.  To me, this feels like going into the doctor to get the results of your blood work and being told, "I can't show you the numbers, but your cholesterol is fine.  The company owns your iron levels, but you aren't anemic."

There's an old maxim that is often forgotten or ignored that you don't own anything you can't take apart.  Warranties are nice, but when farmers can't repair their own tractors you start to see why user serviceable parts, even skilled- or trained-user serviceable parts are important.  In a world where any data that's collected is often sold and firmware updates come twice a year and stop as soon as the new model comes out.  People with implants are at an ever greater risk of losing control of their body parts RoboCop-style.  Malfunctions happen, and a thriving, open community is a great way of keeping the process more transparent and hopeful for everyone.  I would hate to have a malfunctioning defibrilator in my chest and have nowhere to turn, once the company tells me it's outside warranty or the like.

# Link Dump Week of August 22

posted on 2016-08-22 22:05:00

Page Dewarping

Overview of algorithms for converting a warped image of text (e.g. from a photo of a printed page of a book) to a clean, black and white version. Wish I'd known more about this in school, though I probably wouldn't have had the time to make anything with it. Looks like a fun weekend project.

Writing Less Damn Code

Just a bit of a (justified) rant on the complexity of some solutions to web problems. Mostly here to remind me that I really want to take some time to learn about flex box. Well… really want to know about flex box with as little effort as possible.

Writing a JPEG decoder in Rust

Part 1 and Part 2. Good idea for an appropriate project to make something with Rust. Maybe I can do a tga one to see how I can improve cl-tga. Maybe I'll finally do RLE encoding.

# Cool futuristic Art

posted on 2016-08-01 13:15:00

>Sarah showed me the Work of Simon Stalenhag, who seems to do work of a futuristic/crumbling relics of consumerism/world-controlling technology nature. I'd suggest taking a look! It reminds me of the War of the Worlds movie around the middle of the page.