Tagplanet:canonical

Posts to be picked up by http://voices.canonical.com/

M29, a secure URL shortener

A year ago, I launched M29, a URL shortener with a twist. Apparently I forgot to announce it here. Whoops.

Normal URL shorteners are fairly simple. You submit a long URL. The service generates a short URL. The long and short URL are placed in a backend database. If you go to the short URL, it redirects to the long URL.

This means that the URL shortener service has a large database of URLs available to it. While 99% of the contents of this database may be mundane, it's still a large, centralized source of information. Very relevant to the recent NSA news, for example.

M29's twist is, except when serving the redirect, it does not know anything about the contents of the long URLs. This is accomplished by generating an AES-128 key, using it to encrypt the long URL, and then splitting the key in two. One half of the key is stored in the backend service, and the other half is encoded as part of the short URL itself. This means the only time the two parts of the key come together is when the short URL is requested for the redirect.

Getting from a long URL to a short URL can be done one of several ways. If you go to m29.us and have Javascript enabled, the client side actually loads an AES libary, builds the key, encrypts the URL, and sends the encrypted URL and half of the key to the server, all processed on the client side. If you don't have Javascript enabled, this task is farmed out to the server side, which generates a random key, encrypts, makes the database insert, returns the short URL, then throws away half of the key. M29 also has a featureful API which lets you do these tasks yourself. (It is also compatible with the goo.gl API, which is easy to work with and has several tools available.)

The net effect is, while I currently have a database of about 10,000 entries, I cannot read them. Source IP and URI logging are not done on the server, so the only way I can find a long URL is if I load a full short URL, which is not possible given just the backend database.

Anyway, this weekend I did some work on M29, including adding a bit.ly-style preview option (append a "+" to the short URL to get its info), among other small feature additions and fixes. It was then I realized, by going to that above short URL (the first URL generated and used in documentation) that the one-year anniversary of the service is today.

SteamLink (Half-Life: Uplink for Steam) updated for Linux / OS X

The original Half-Life and Counter-Strike games were quietly released for Linux and Mac OS X last week, and as the maintainer of SteamLink, a repackaging of Half-Life: Uplink for Steam, I went out to see if the mod's files could be installed on these platforms.

Turns out there is a bug in Steam for these platforms, where it tries to launch the Windows version of Half-Life for GoldSrc mods from within Steam. However, Half-Life can be manually launched and pointed at the mod.

I have released a new version of SteamLink as a zip file. If you would like to run Half-Life: Uplink on Linux or OS X, simply download and extract the zip, and run the installer shell script. It will determine the Half-Life installation directory, install the mod, and give you a symlink to a script to launch it.

Linux md RAID 10 disk layout

I'm working on re-doing my home router / VM server to provide better IO. The goal is to have an SSD as the boot drive, and 4 2TB disks in RAID 10 (4TB usable total) for the VMs. I'll be using md RAID for building it, however I want to be particular about where the drives are physically, for a few reasons:

  • The RAID drives are all SATA 6Gbps drives, but the server's motherboard only has 2 SATA 6Gbps ports. I've got a 2-port PCIe SATA 6Gbps controller on the way[0].
  • I want to place the two drives in each base RAID 1 set on different controllers: one on the motherboard controller, one on the PCIe card controller. This may provide extra performance, but more importantly, having each disk on a different controller protects the array as a whole in case of a controller failure.

Linux md RAID 10 is a relatively new mode. In the past, you would manually create multiple RAID 1 arrays, then combine them in RAID 0, thereby knowing where the disks were placed, since you were doing it yourself. The mdadm RAID 10 method is easier, but I found there is literally no documentation on what drives it uses for the underlying RAID 1 arrays. Using loopback devices and some trial and error, I figured out how the arrays are assembled.

In a nutshell, the underlying RAID 1 arrays are paired two at a time, in order given during creation. If you were to do this:

# mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sd{a,b,c,d}1

sda1 and sdb1 form one RAID 1 array, and sdc1 and sdd1 form another:

|---------------------------|
|           RAID0           |
|---------------------------|
|    RAID1    |    RAID1    |
|-------------|-------------|
| sda1 | sdb1 | sdc1 | sdd1 |
|---------------------------|

One other thing to mention is what happens when multiple drives are lost. Say in this case, both sda1 and sdd1 are lost, resulting in a degraded but functional array:

# mdadm --fail /dev/md0 /dev/sda1
# mdadm --remove /dev/md0 /dev/sda1
# mdadm --fail /dev/md0 /dev/sdd1
# mdadm --remove /dev/md0 /dev/sdd1
|---------------------------|
|           RAID0           |
|---------------------------|
|  RAID1 (D)  |  RAID1 (D)  |
|-------------|-------------|
|      | sdb1 | sdc1 |      |
|---------------------------|

If you were to replace sdd and add it back first, you might think it would go in the second RAID 1 array. But no, it takes the first available degraded slot:

# mdadm --add /dev/md0 /dev/sdd1
|---------------------------|
|           RAID0           |
|---------------------------|
|    RAID1    |  RAID1 (D)  |
|-------------|-------------|
| sdd1 | sdb1 | sdc1 |      |
|---------------------------|

So be careful in this situation, if you care about where the devices are physically laid out.

[0] Half of the order actually arrived Friday, including the SSD and a 4-port PCIe 4x SATA 6Gbps controller. The idea was to place two of the RAID drives on the motherboard SATA 6Gbps controller, and two on the new controller, plus the boot SSD (which is also SATA 6Gbps). My system has 3 PCIe 1x ports and a PCIe 16x port. The 4x card was supposed to go in the 16x port, but I learned after failure that many motherboards do not like non-video cards in the primary 16x port. Oh well. The boot SSD will now go on one of the motherboard SATA 3Gbps ports, and I've got an order in for a 2-port PCIe 1x SATA 6Gbps controller.

Introducing apache-vsl

I'm not a programmer, honest, but lately I've had this annoying habit of releasing code. But instead of my normal rambling posts about new announcements, I'll try to keep it brief.

apache-vsl has been one of my "90/90" projects (90% done, just need to finish the remaining 90%) for many years; I believe I started it around 2006, and I finally got the urge to finish and release it to the public. It's an Apache logging daemon (managed by Apache itself), capable of intelligently handling multiple VirtualHosts while being as efficient and extensible as possible. It's specifically built for two use cases -- servers which have a large number of VirtualHosts (hundreds or more), and VirtualHosts which receive a lot of traffic -- but it's easy enough to manage that it's useful with any Apache installation.

You can define time-based log files (monthly, daily, hourly... it's all just strftime), and apache-vsl handles when to log to a new file according to your definition. It manages symlinks to both the current and previous logfiles (for example, access_log.2012-08 -> access_log and access_log.2012-07 -> access_log.old), and has support for program triggers when a rotation happens (say, for running Webalizer against the old logfile, or compressing it).

apache-vsl manages its open files efficiently, so for a server with many VirtualHosts (not all of which may be accessed very often), all of the VirtualHosts' log files won't be open at once, but for VirtualHosts which are accessed frequently, the log filehandle will not constantly be opening and closing.

apache-vsl shares many features with cronolog, but while cronolog is designed more for splitting logfiles after the fact (if you were to use it directly from Apache, you'd need to have Apache open a dedicated pipe for every VirtualHost), apache-vsl is designed to communicate with Apache with a single pipe for multiple VirtualHosts.

A lot of good documentation is available in the manpage, so if you're interested, I suggest you start there.

Raspberry Pi

Raspberry Pi

My Raspberry Pi, the $35 barebones computer, arrived last week. I bought it for three primary reasons:

  1. It was cheap.
  2. It was interesting.
  3. It was cheap.

I ordered early on a Thursday morning, expecting 3-4 weeks before it shipped per Element14's estimates. So I was quite surprised when I got a shipment notification later in the day. I'm guessing Element14 had a batch in, and was processing the backorders from the last few weeks, and somehow my order got mixed into that. So hey, yay instant gratification!

Shortly after that, Adafruit's clear acrylic case became available again, so I ordered that as well. I was expecting a tighter fit, but instead it bounces around a bit within the case. It's still sturdy enough to do its job, and looks nice, though the top panel no longer has the etched Raspberry Pi logo like I've seen on previous photos.

Raspberry Pi

I also ordered some cables and a powered USB hub from Monoprice (gotta love $5 next day shipping to California/Nevada), and a high power iPad USB charger from OWC, but that turned out to be unnecessary. The 7-port Monoprice USB hub seems to provide enough power to power the Raspberry Pi itself, along with some accessories.

In the above setup, I've got it powering the Raspberry Pi (with HDMI output, plus 16GB SDHC card), a keyboard and mouse, a 4GB USB thumb drive, and my Defcon 20 badge. Not only does the Raspberry Pi boot, but I gave it a stress test for a few hours: Running a Quake 3 Arena 1920x1080 demo on loop (GPU), while simultaneously running Bonnie++ on the USB thumb drive (CPU and IO). It successfully did that for the entire test, so I'm confident in the power performance of the USB hub.

Now, I have no real plans for it; it'll probably sit, powered on in the corner of my office, for if I need to test something on an ARM system. But I do like the concept of its primary purpose. My first computer was a Vic 20 (predecessor to the Commodore 64), and then a Commodore 128 (backwards-compatible successor to the Commodore 64). I have fond memories of those computers, and they helped shape the course of the rest of my life. I like how the Raspberry Pi has both HDMI and Composite output, meaning it supports any TV or monitor made in the last 5 years or so, plus at least the previous 50 years of TVs. (Remember RF adapters?) This really lowers the barrier of computer use for your average child.

However, there is one problem with the Raspberry Pi I really disagree with. Namely, it requires an operating system. A 440MB operating system, as well as the means to get it onto an SD card. An operating system that could easily be destroyed by curious fingers. This could be a huge barrier.

One of the things about my Vic 20 and Commodore 128 was I didn't have any cartridges or pre-bought software. Just the computer, a tape drive, and some CR-90 tapes. All value I got out of them was either by typing in BASIC programs from BYTE or Commodore magazines, or creating programs myself.

The Raspberry Pi really needs an onboard ROM with some sort of simple language on it, such as BBC BASIC, and the ability to save directly to SD cards. Upon boot, if the SD card contains a bootloader pointing to a full operating system, boot that, but if not, load the BASIC ROM.

© 2014 Ryan Finnie

Theme by Anders NorenUp ↑