Ryan Finnie

dsari (Do Something and Record It) CI 2.0 released

In 2015, I released dsari (Do Something and Record It), a small but very powerful continuous integration (CI) system. Over the past few years, I’ve been adding bits and pieces to it, and realized I haven’t made a proper release since shortly after the initial release.

Version 2.0 includes the following:

  • 90% refactor, and rewritten in Python 3. Note when upgrading you will need to install Python 3 versions of dependent libraries (e.g. apt-get install python3-croniter).
  • Configurable database support: SQLite (default), PostgreSQL, MySQL, or MongoDB.
  • Added dsari-prometheus-exporter for exporting job/run metrics to Prometheus.
  • Added dsari-info command, allowing for CLI retrieval of various dsari job/run information.
  • Added support for iCalendar RRULE schedule definitions (via python-dateutil), instead of or in addition to hashed cron format (via croniter).
  • New job option: concurrent_runs - Allows multiple runs of the same job to run concurrently.
  • HUP reloading is now immediate, rather than next time dsari-daemon is idle.
  • Triggers may now request a time to run the trigger, instead of as soon as possible.
  • Many minor fixes.

Most of these changes were done gradually over the last two years, but what got me on the latest development kick was actually a recent facepalm moment. Recently I had the idea to take Nagios NRPE definitions, run them, and export the data (exit code, running length, etc) to Prometheus, rather than using Nagios directly. But then I started thinking “I’ll need to deal with tracking forks, concurrency, etc, etc. That’ll be a pain.”

“Oh wait, I’ve already solved this. This sounds like a situation where I need to Do Something and Record It(s metrics).”

With that, I wrote dsari-prometheus-exporter and integrated it with dsari. The metrics it exports can be useful; see below for the metrics from a sample job.

The sample1 job is part of a severely constrained concurrency group where only one job run in the group is configured to be running at once. The actual run time is remarkably consistent at about 90 seconds, but since there are a number of jobs in this concurrency group and they all want to run every minute, there is always a waiting list. As a result, this job runs about 70 seconds late on average. But the spread is quite large, from (effectively) immediately, to nearly 6 minutes late.

(The fact that over 1300 runs, the job has spent almost exactly the same amount of time running as being blocked is also interesting, but a coincidence.)

# HELP dsari_run_latency_seconds Length of time spent between scheduled start and actual start
# TYPE dsari_run_latency_seconds summary
dsari_run_latency_seconds{job_name="sample1",quantile="0.01"} 0.0027895
dsari_run_latency_seconds{job_name="sample1",quantile="0.1"} 34.53752
dsari_run_latency_seconds{job_name="sample1",quantile="0.5"} 69.8487545
dsari_run_latency_seconds{job_name="sample1",quantile="0.9"} 170.2028667
dsari_run_latency_seconds{job_name="sample1",quantile="0.99"} 358.96789803
dsari_run_latency_seconds_count{job_name="sample1"} 1300
dsari_run_latency_seconds_sum{job_name="sample1"} 113835.42681500006

# HELP dsari_run_duration_seconds Length of time spent in a run
# TYPE dsari_run_duration_seconds summary
dsari_run_duration_seconds{job_name="sample1",quantile="0.01"} 5.06363971
dsari_run_duration_seconds{job_name="sample1",quantile="0.1"} 90.03290849999999
dsari_run_duration_seconds{job_name="sample1",quantile="0.5"} 90.060451
dsari_run_duration_seconds{job_name="sample1",quantile="0.9"} 90.0872659
dsari_run_duration_seconds{job_name="sample1",quantile="0.99"} 90.10349761
dsari_run_duration_seconds_count{job_name="sample1"} 1300
dsari_run_duration_seconds_sum{job_name="sample1"} 112009.71985500008

# HELP dsari_last_run_exit_code Numeric exit code of the last run for a job
# TYPE dsari_last_run_exit_code gauge
dsari_last_run_exit_code{job_name="sample1"} 143

# HELP dsari_last_run_schedule_time Schedule time of the last run for a job, seconds since epoch
# TYPE dsari_last_run_schedule_time gauge
dsari_last_run_schedule_time{job_name="sample1"} 1524347585.946814

# HELP dsari_last_run_start_time Start time of the last run for a job, seconds since epoch
# TYPE dsari_last_run_start_time gauge
dsari_last_run_start_time{job_name="sample1"} 1524347824.892004

# HELP dsari_last_run_stop_time Stop time of the last run for a job, seconds since epoch
# TYPE dsari_last_run_stop_time gauge
dsari_last_run_stop_time{job_name="sample1"} 1524347913.404784

# HELP dsari_run_count Number of runs performed for a job
# TYPE dsari_run_count counter
dsari_run_count{job_name="sample1"} 1300

# HELP dsari_run_failure_count Number of failed runs performed for a job
# TYPE dsari_run_failure_count counter
dsari_run_failure_count{job_name="sample1"} 56

# HELP dsari_run_success_count Number of successful runs performed for a job
# TYPE dsari_run_success_count counter
dsari_run_success_count{job_name="sample1"} 1244

Let's Encrypt wildcard certificates

The vsix.us IP tests site is now fully HTTPS. About a year ago I converted all the sites I could over to HTTPS with Let’s Encrypt, and while vsix.us itself was converted to HTTPS, the “nocache” wildcard site remained at HTTP because Let’s Encrypt did not have wildcard support at the time.

About a week ago they announced wildcard support, and today I registered *.nocache.vsix.us. The process wasn’t seamless, and this blog post is meant to be some notes on getting it working, not necessarily a guide.

Let’s Encrypt only supports wildcard registration via the ACME v2 protocol and dns-01 validation. I’m not exactly sure why it’s dns-01 only, as they are not checking multiple subdomains, just using the TXT record of _acme-challenge.example.com to validate *.example.com. Unless I’m missing something, http-01 validation on example.com would be just as secure.

In my case, this means configuring BIND for secure dynamic updates. The documentation for certbot-dns-rfc2136 is straightforward, but complicated by the fact that my zones are DNSSEC signed, so BIND itself needs to be able to re-sign the zone on the fly. (In my case, I re-sign the zones myself after making zone updates.) Ultimately I split out nocache.vsix.us to its own zone and configured BIND so it could re-sign upon update.

certbot needs to be at least version 0.22 to support ACME v2, needed for wildcard. The Ubuntu PPA includes 0.22.2 as of this writing, but it does not include certbot-dns-rfc2136 needed for dns-01 validation. But it’s relatively easy to install manually:

git clone https://github.com/certbot/certbot
cd certbot/certbot-dns-rfc2136
sudo pip3 install .

Also of note is 0.22 still defaults to ACME v1. You will need to pass the v2 endpoint via --server, but keep in mind that the v2 endpoint is essentially a completely different service, and it will once again ask registration questions (email address, etc). With all that in place, here was the final invocation for me:

certbot certonly \
  --server 'https://acme-v02.api.letsencrypt.org/directory' \
  --dns-rfc2136 \
  --dns-rfc2136-credentials /etc/letsencrypt/dns-01.ini \
  -d nocache.vsix.us -d '*.nocache.vsix.us'

Red Beans and Rice

Just a short post here (and no pictures, because let’s be honest, red beans and rice doesn’t actually look that appetizing once it’s been reduced down to a brown mush). Every few weeks I’ll stop by Popeyes – I love their spicy chicken strips with buffalo sauce – and will get red beans and rice as a side. A few weeks ago I decided to try to recreate the recipe, mostly by staring into the spice rack, picking a random spice and thinking “yeah, that’ll work”.

After a few iterations, I’ve got a recipe which is not 100% exact to Popeyes, but is still wonderfully tasty. And as usual, is easy to make with pantry items. (You could of course do things like use real celery instead of flakes, add salt pork or shrimp, etc, but this is a good simple base.)

  • 4 cans (approx 15 oz each) red kidney beans, drained
  • 1 can beef broth
  • ½ can water
  • 1 chicken bouillon cube
  • 2 tsp celery flakes
  • 2 tsp parsley flakes
  • 2 bay leaves
  • 1 tsp garlic powder
  • ½ tsp onion powder
  • ½ tsp crushed red pepper
  • ¼ tsp cumin

Combine all ingredients, cover and bring to a boil. Reduce to a simmer for 30 minutes, stirring halfway. Uncover, remove bay leaves and mash moderately (most but not all beans should be broken). Leave uncovered, re-add bay leaves and simmer for another 30 minutes, stirring every 10 minutes. Consistency should be a thick sauce. Remove bay leaves. Serve over rice.

Home Sweet Home: The "Things"

It all began a year ago with a dot.

Late last year, when replacing all my home’s lights with LEDs (among other projects), I had been spending a lot of time at Home Depot and Lowes. While at Lowes in November 2016, I noticed they sold Amazon Echo products, and the Echo Dot was only $50. I picked one up and started playing with it.

Echo mostly works as you’d expect from a Siri-like device: you say “Alexa, …” and it responds. The full-sized Echo supposedly has decent sound, but the Dot is compact and has, at best, satisfactory sound for music. However, if you want you may hook it up to a speaker via Bluetooth, and that works well. I’ll usually wake up with “Alexa, play some music”, and it often does a good job figuring out music I would like to hear. “Alexa, what’s the weather like?” “Alexa, what movies are playing?” Etc. A little limited in day to day functionality, but worth the $50 as a novelty.

But I was drawn to the developer functionality. I wrote a few apps, one of which I tried to get submitted for publication, but they kept coming back with technical rejections to fix, and I never completed the process. But it’s still an interesting platform.

I was also curious about the “smart home” integration possibilities. I had mostly dismissed the the whole “Internet of Things” idea as gimmicky, but there was one specific use case I could think of for me personally. The backyard patio lights are a string of overhead lights which run to the side of the house and attach to a standard outdoor outlet which is in a slightly inconvenient location. I had been thinking of ways to fix this, and a weatherproof remote controlled relay switch would be useful.

SmartThings screenshot The Samsung SmartThings hub was on sale on Black Friday 2016 for $50 (which as of 2017 is now its regular price), got good reviews, and seemed to have the best variety of device support. Its two main supported protocols are ZigBee and Z-Wave. Both are point-to-point mesh networks: you can pair both Device A and Device B to your hub, and if, say, Device B is not within range of the hub but can talk to Device A, Device A will relay the commands for it. It also supports several other Internet-connected services, such as the thermostat I already happened to have.

I bought the SmartThings hub, as well as a weatherproof control module. The pairing process was simple: press a button on the module to initiate pairing, accept it on the smartphone app, and now I could turn the patio lights on and off from my phone. It also has Alexa integration, so once those were paired, I could say “Alexa, turn on patio lights”.

A week later, I saw that Lowes had their Iris ZigBee (indoor) outlet control modules on sale, and bought a few to play with. Iris is a competing home automation platform to Samsung SmartThings, but as I understood it, SmartThings supported Z-Wave and ZigBee devices from any manufacturer (which gave it a leg up on competitors which tend to be walled gardens). I began the pairing process and… problem. It simply showed up as “Thing” and I couldn’t do anything with it.

I was all set to return them, but it was then I learned about the full extent of SmartThings compatibility. While the app has a very simple interface and limited information and actions available, it turns out there is actually a web interface for developers. Not only does this interface give you a lot more information (radio levels, debugging events, etc), but it actually allows you to write your own device handlers, in a scripting language called Groovy, which is basically the Java equivalent of Lua, and is often used in Jenkins plugins.

With enough work, I could have reverse engineered the Iris outlet and written my own device handler, but SmartThings has a large developer community, and it’s likely someone has already written a device handler for nearly every Z-Wave and ZigBee device out there, even if it’s not part of the core SmartThings platform. I quickly found an Iris outlet handler, imported it, and then was able to pair the outlets.

An interesting feature of these Iris outlets is, while they are controlled via ZigBee, they also have a Z-Wave radio in them and can act as Z-Wave repeaters, strengthening the Z-Wave mesh as a whole. However, SmartThings does not know about Z-Wave devices which literally do nothing but act as a repeater, so when I added the Z-Wave side, it would fall back to a switched outlet. Not really a problem (in the phone UI, it would show a switch to toggle which did nothing), but it did inspire me to learn Groovy and how to write device handlers, and I wrote a proper device handler for Z-Wave repeaters. It was actually a surprising amount of work to write a handler for a device which does literally nothing.

Temperature graph

Since then, I went a little overboard in the novelty of it, and have bought a number of additional devices. As of now, the current list is:

  • Honeywell Wi-Fi thermostat - Previously mentioned; I happened to have this for a few years before I bought the SmartThings hub. It integrates with Alexa, but also with SmartThings, so I have it paired with SmartThings, which exports everything to Alexa and gives me more options than Alexa directly. But honestly, it’s rare I even touch the thermostat beyond its normal schedule.
  • GE Z-Wave outdoor module - Previously mentioned, currently serving the backyard patio lights. They turn on each day at sundown, and off at 11:30PM. (SmartThings supports dynamic “sunrise” and “sundown” in addition to static times.)
  • Leviton Z-Wave appliance module - Currently serving some LED strip lights I have mounted in the alcove above the TV in the living room; I have them listed as “movie lights”.
  • Iris ZigBee (Z-Wave) smart plug - Previously mentioned; I have two of these but they’re not controlling anything at the moment.
  • Iris ZigBee smart fob - A little fob with four buttons. I can’t remember what I’ve set them to do as I never use it.
  • GE Z-wave in-wall light dimmer and GE Z-Wave in-wall light switch - Z-Wave switches which are meant to replace traditional in-wall light switches. They work like normal light switches (and in the dimmer’s case you can hold on/off to increase/decrease the light level), but are also Z-Wave controllable. I have the dimmer on the front porch light and the normal switch on the outdoor rear wall light.
  • Cree ZigBee 60w equivalent LED bulbs - Yes, the light bulb itself is ZigBee compatible. I have four of them, two for the living room lights and two for the bedroom night stand lights. I’ve got each of the pairs in device groups, so each night, it’s “Alexa, turn off living room lights”. As for the bedroom, I have it set to turn them on at 11PM, as I’ll usually go to bed shortly after and read for awhile. Then “Alexa, turn off bedroom lights”. They’re even dimmable (“Alexa, set living room lights to 50%.”) These are the devices I use most often day to day.
  • SmartThings ZigBee multipurpose sensor - This battery powered sensor has several measurements: ambient temperature, contact (like those small battery powered alarms you can add to windows), axis rotational position, and motion. I currently have two, one mounted to the inside of the front door to monitor when it’s opened, and one mounted to the garage door. The device handler’s “garage door” mode is interesting; rather than use the door contact sensor, it uses the axis rotational sensor. When the sensor is vertical, the garage door is closed. When it’s horizontal, the garage door has been rolled up.
  • Zooz Z-Wave 4-in-1 sensor - Another sensor, though this provides some different measurements than the previous: temperature, humidity, light level and motion. It even, oddly, has a tamper protection switch within the case which trips if opened, and is exposed as an acceleration event. Mounted in the hallway, it’s actually sensitive enough that you could figure out when I take a shower by graphing the humidity levels a few rooms over.
  • TP-Link Wi-Fi smart plug - This is actually neither Z-Wave nor ZigBee, but straight Wi-Fi. Normally it calls home and uses a separate app for control (“Kasa”), but it has a local port open, and accepts obfuscated commands (people, XOR is not encryption). I wrote a device handler and proxy to allow SmartThings direct control of it, and it currently controls a floor fan in the living room during the summer.

Be Happy!

I was recently on a date and was asked the most embarrassing thing about myself I would admit to the world. I told her my favorite song, and let me tell you. But first…

My favorite musical artist is They Might Be Giants, and has been since the early 2000s. The circumstances around being exposed to TMBG involved a person and family I want nothing to do with these days (and don’t want to go into here), but suffice it to say the enjoyment in the music itself endured.

I have nearly every album, including oddities such as Flood: Live in Australia. I could probably recite all the lyrics to Flood itself. I play When Will You Die from the album Join Us whenever a prominent Terrible Person dies. (♫ School children stay at home (yeah!) and all the banks will close (yeah!), each year will mark the date on which we celebrate ♪).

I’ve been to several concerts, including arguably the best experience: They opened for themselves in San Francisco as a TMBG cover band, Sapphire Bullets (“the only They Might Be Giants tribute band that matters”). They performed Flood start to finish, left the stage, came back as TMBG and played their main set.

I have a payphone in my kitchen which (besides being fully functional) is programmed to play a random TMBG song when you dial (718) 387-6962, the old Dial-a-Song line.

And yet my favorite song is not a They Might Be Giants song.

There are a number of other artists and albums I enjoy. The Flaming Lips. Joan Jett. Blue Man Group’s studio albums. I have all the albums by Boris the Sprinkler, a relatively obscure punk rock band from Green Bay. (There’s actually a few entertaining stories involving coincidences and them, but that’s for another time.)

I have a thing for female singer-songwriters, usually one-hit wonders who I later discover have other really good content. Sara Bareilles. Dido. Colbie Caillat. My current fling is KT Tunstall.

Often I’ll find an artist and discover an album, get really into it, start looking at their other albums and discover I don’t like their other stuff as much. Architecture in Helsinki and In Case We Die. Björk and Post. Spoon and Ga Ga Ga Ga Ga.

But my favorite song is not by any of the artists I’ve mentioned so far.

Many years ago, I read a series of short stories by a woman named Ali Davis, True Porn Clerk Stories. These were originally a series of blog posts which are no longer available; she later compiled them into a book. Ali worked at one of those “Family Video”-style stores which ostensibly looked like a normal video rental store, but had a large back section for porn rentals which comprised most of their business. The stories are both hilarious and sad, in a similar style to Acts of Gord, which chronicles the adventures in managing a video game store. I definitely recommend reading both.

In one of Ali’s stories, she describes her 7AM opening routine, and her love for Aqua’s 1997 album Aquarium.

What keeps me awake is Aquarium, by Aqua. You may remember Aqua - they were a Danish-Norwegian technopop group that won both worldwide fame and my heart by pissing off Mattel with the song “Barbie Girl”. […] And that’s when I discovered that I love Aquarium. It’s the very finest in Scandinavian synth-pop dance music. It’s incredibly chipper, in a modern Abbaesque sort of way.

She would play Aquarium every morning, from start to finish. The first song is Happy Boys and Girls, and, fittingly for her employment, is about how sex is fun. Needless to say, this album annoyed the customers and pissed off the employees. But it made her happy.

After reading that, I bought the album myself, and grew to love it. And Happy Boys and Girls epitomizes the album as a whole. It’s driving and unabashedly upbeat and happy. I have a few other favorites on the album (I’ll often loop Doctor Jones when running, as the driving tempo lines up well), but Happy Boys and Girls has grown to be my favorite song.

(Before you ask, in my opinion Barbie Girl is not a bad song per se, but is not in my top half of songs on the album. Oh, and go listen to the full album. If you’ve heard Barbie Girl, you probably assumed the girl and guy were putting on over-the-top voices for that song. No, that’s actually their real singing voices.)

« All posts