SteamLink (Half-Life: Uplink for Steam) updated for Linux / OS X

The original Half-Life and Counter-Strike games were quietly released for Linux and Mac OS X last week, and as the maintainer of SteamLink, a repackaging of Half-Life: Uplink for Steam, I went out to see if the mod's files could be installed on these platforms.

Turns out there is a bug in Steam for these platforms, where it tries to launch the Windows version of Half-Life for GoldSrc mods from within Steam. However, Half-Life can be manually launched and pointed at the mod.

I have released a new version of SteamLink as a zip file. If you would like to run Half-Life: Uplink on Linux or OS X, simply download and extract the zip, and run the installer shell script. It will determine the Half-Life installation directory, install the mod, and give you a symlink to a script to launch it.

Linux md RAID 10 disk layout

I'm working on re-doing my home router / VM server to provide better IO. The goal is to have an SSD as the boot drive, and 4 2TB disks in RAID 10 (4TB usable total) for the VMs. I'll be using md RAID for building it, however I want to be particular about where the drives are physically, for a few reasons:

  • The RAID drives are all SATA 6Gbps drives, but the server's motherboard only has 2 SATA 6Gbps ports. I've got a 2-port PCIe SATA 6Gbps controller on the way[0].
  • I want to place the two drives in each base RAID 1 set on different controllers: one on the motherboard controller, one on the PCIe card controller. This may provide extra performance, but more importantly, having each disk on a different controller protects the array as a whole in case of a controller failure.

Linux md RAID 10 is a relatively new mode. In the past, you would manually create multiple RAID 1 arrays, then combine them in RAID 0, thereby knowing where the disks were placed, since you were doing it yourself. The mdadm RAID 10 method is easier, but I found there is literally no documentation on what drives it uses for the underlying RAID 1 arrays. Using loopback devices and some trial and error, I figured out how the arrays are assembled.

In a nutshell, the underlying RAID 1 arrays are paired two at a time, in order given during creation. If you were to do this:

# mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sd{a,b,c,d}1

sda1 and sdb1 form one RAID 1 array, and sdc1 and sdd1 form another:

|           RAID0           |
|    RAID1    |    RAID1    |
| sda1 | sdb1 | sdc1 | sdd1 |

One other thing to mention is what happens when multiple drives are lost. Say in this case, both sda1 and sdd1 are lost, resulting in a degraded but functional array:

# mdadm --fail /dev/md0 /dev/sda1
# mdadm --remove /dev/md0 /dev/sda1
# mdadm --fail /dev/md0 /dev/sdd1
# mdadm --remove /dev/md0 /dev/sdd1
|           RAID0           |
|  RAID1 (D)  |  RAID1 (D)  |
|      | sdb1 | sdc1 |      |

If you were to replace sdd and add it back first, you might think it would go in the second RAID 1 array. But no, it takes the first available degraded slot:

# mdadm --add /dev/md0 /dev/sdd1
|           RAID0           |
|    RAID1    |  RAID1 (D)  |
| sdd1 | sdb1 | sdc1 |      |

So be careful in this situation, if you care about where the devices are physically laid out.

[0] Half of the order actually arrived Friday, including the SSD and a 4-port PCIe 4x SATA 6Gbps controller. The idea was to place two of the RAID drives on the motherboard SATA 6Gbps controller, and two on the new controller, plus the boot SSD (which is also SATA 6Gbps). My system has 3 PCIe 1x ports and a PCIe 16x port. The 4x card was supposed to go in the 16x port, but I learned after failure that many motherboards do not like non-video cards in the primary 16x port. Oh well. The boot SSD will now go on one of the motherboard SATA 3Gbps ports, and I've got an order in for a 2-port PCIe 1x SATA 6Gbps controller.

Introducing apache-vsl

I'm not a programmer, honest, but lately I've had this annoying habit of releasing code. But instead of my normal rambling posts about new announcements, I'll try to keep it brief.

apache-vsl has been one of my "90/90" projects (90% done, just need to finish the remaining 90%) for many years; I believe I started it around 2006, and I finally got the urge to finish and release it to the public. It's an Apache logging daemon (managed by Apache itself), capable of intelligently handling multiple VirtualHosts while being as efficient and extensible as possible. It's specifically built for two use cases -- servers which have a large number of VirtualHosts (hundreds or more), and VirtualHosts which receive a lot of traffic -- but it's easy enough to manage that it's useful with any Apache installation.

You can define time-based log files (monthly, daily, hourly... it's all just strftime), and apache-vsl handles when to log to a new file according to your definition. It manages symlinks to both the current and previous logfiles (for example, access_log.2012-08 -> access_log and access_log.2012-07 -> access_log.old), and has support for program triggers when a rotation happens (say, for running Webalizer against the old logfile, or compressing it).

apache-vsl manages its open files efficiently, so for a server with many VirtualHosts (not all of which may be accessed very often), all of the VirtualHosts' log files won't be open at once, but for VirtualHosts which are accessed frequently, the log filehandle will not constantly be opening and closing.

apache-vsl shares many features with cronolog, but while cronolog is designed more for splitting logfiles after the fact (if you were to use it directly from Apache, you'd need to have Apache open a dedicated pipe for every VirtualHost), apache-vsl is designed to communicate with Apache with a single pipe for multiple VirtualHosts.

A lot of good documentation is available in the manpage, so if you're interested, I suggest you start there.

Raspberry Pi

Raspberry Pi

My Raspberry Pi, the $35 barebones computer, arrived last week. I bought it for three primary reasons:

  1. It was cheap.
  2. It was interesting.
  3. It was cheap.

I ordered early on a Thursday morning, expecting 3-4 weeks before it shipped per Element14's estimates. So I was quite surprised when I got a shipment notification later in the day. I'm guessing Element14 had a batch in, and was processing the backorders from the last few weeks, and somehow my order got mixed into that. So hey, yay instant gratification!

Shortly after that, Adafruit's clear acrylic case became available again, so I ordered that as well. I was expecting a tighter fit, but instead it bounces around a bit within the case. It's still sturdy enough to do its job, and looks nice, though the top panel no longer has the etched Raspberry Pi logo like I've seen on previous photos.

Raspberry Pi

I also ordered some cables and a powered USB hub from Monoprice (gotta love $5 next day shipping to California/Nevada), and a high power iPad USB charger from OWC, but that turned out to be unnecessary. The 7-port Monoprice USB hub seems to provide enough power to power the Raspberry Pi itself, along with some accessories.

In the above setup, I've got it powering the Raspberry Pi (with HDMI output, plus 16GB SDHC card), a keyboard and mouse, a 4GB USB thumb drive, and my Defcon 20 badge. Not only does the Raspberry Pi boot, but I gave it a stress test for a few hours: Running a Quake 3 Arena 1920x1080 demo on loop (GPU), while simultaneously running Bonnie++ on the USB thumb drive (CPU and IO). It successfully did that for the entire test, so I'm confident in the power performance of the USB hub.

Now, I have no real plans for it; it'll probably sit, powered on in the corner of my office, for if I need to test something on an ARM system. But I do like the concept of its primary purpose. My first computer was a Vic 20 (predecessor to the Commodore 64), and then a Commodore 128 (backwards-compatible successor to the Commodore 64). I have fond memories of those computers, and they helped shape the course of the rest of my life. I like how the Raspberry Pi has both HDMI and Composite output, meaning it supports any TV or monitor made in the last 5 years or so, plus at least the previous 50 years of TVs. (Remember RF adapters?) This really lowers the barrier of computer use for your average child.

However, there is one problem with the Raspberry Pi I really disagree with. Namely, it requires an operating system. A 440MB operating system, as well as the means to get it onto an SD card. An operating system that could easily be destroyed by curious fingers. This could be a huge barrier.

One of the things about my Vic 20 and Commodore 128 was I didn't have any cartridges or pre-bought software. Just the computer, a tape drive, and some CR-90 tapes. All value I got out of them was either by typing in BASIC programs from BYTE or Commodore magazines, or creating programs myself.

The Raspberry Pi really needs an onboard ROM with some sort of simple language on it, such as BBC BASIC, and the ability to save directly to SD cards. Upon boot, if the SD card contains a bootloader pointing to a full operating system, boot that, but if not, load the BASIC ROM.

twuewand 2.0 released

You may remember about a year ago when I released twuewand, a TrueRand implementation. TrueRand is a hardware entropy generation technique, implemented in software. In a nutshell, it works by setting an alarm for a few milliseconds in the future, and flipping a bit until the alarm is reached. It works due to the fact that time (your computer's RTC) and work (your computer's CPU) are not linked, so the result when the alarm comes due is unpredictable.

TrueRand was invented in 1995, and had mostly been forgotten for the last decade, until I started doing research on it last year. So it was quite a surprise when I was at Dan Kaminsky's talk at DEFCON a few weeks ago, and one of the topics he brought up was TrueRand. (Go check out his presentation slides; I just want to point out that while I'll be focusing on entropy and debiasing here, he goes into a lot of other interesting topics.)

Dan came to roughly the same conclusion as I did, that entropy sources have gotten worse over time, not better, and systems like VMs are almost completely devoid of entropy. Even more worrying, a paper published this year came to the conclusion that approximately 1 out of every 200 public keys on the Internet are easily breakable, not due to weaknesses in the encryption, but by bad entropy being used when generating the keypair. TrueRand may have been forgotten, but it's needed today more than ever. Dan and I talked for awhile after his talk, and went over a few things by email in the week following. twuewand 2.0's new features are influenced by those discussions.

Dan proposed a number of enhancements for TrueRand, mostly centered around other ideas for measuring variances given only a CPU and RTC, but what caught my eye was his idea of enhancing debiasing.

Many forms of random data are random in the technical sense, but are prone to bias. As an theoretical example, take a gun placed in a sturdy mount and pointed at a target not too far away. Most of the time, shots from it will hit the same spot every time (0), but occasionally they won't (1). So you're left something like 00000001000001001100000010000010; mostly hits, but with random misses. So it's random in a technical sense, but the distribution is heavily weighted toward one side.

The simplest method of debiasing is known as Von Neumann debiasing. Bits are processed as pairs, and any pair that is both 0 or both 1 is simply thrown out. Out of the pairs that are left, {0,1} becomes 0 and {1,0} becomes 1. So in the example above, the Von Neumann debiased output would be 00101. The data is now distributed better, but as you can tell, a lot was lost in the process. This is an extreme example since the data was heavily biased to begin with, but in data without a lot of bias, you still lose at least 50% of the bits (I've found 70-75% in real-world twuewand usage).

Dan thought, "Hmm, that's an awful lot of data simply being thrown out. We can't use the discarded data in the direct output, but perhaps we can use it to better (de-)influence the final output." He came up with a method he called modified Von Neumann, which I refer to in twuewand as Kaminsky debiasing.

The incoming bit stream is still run through Von Neumann, and put into an output buffer. However, all bits (whether they pass Von Neumann or not) are fed to a SHA256 stream. Occasionally (after the input stream is finished, or a sufficient number of bytes are put into the output buffer), the SHA256 hash is computed[1], and used as a 256-bit key for AES-256-CBC encrypting the output buffer. This way, only the bits which pass Von Neumann influence the output directly, but all bits help indirectly influence the output as well.

So twuewand now supports Kaminsky debiasing, and will use it by default if Digest::SHA and Crypt::Rijndael are installed.

Now, I want to clear up a mistake I made in my last post. I said that feeding twuewand output to /dev/urandom on Linux systems influences the primary pool, increasing entropy. First, you can actually write to either /dev/random or /dev/urandom, the effect is the same. But more importantly, entropy is NOT increased by writing to /dev/[u]random. It's merely "stirring the pot". If your system is out of entropy and you are blocking on /dev/random, no amount of writing to /dev/[u]random will unblock. (Directly, that is. If you're banging on your local keyboard to do this, you're slowly increasing entropy, but you could be doing the same thing in a text editor, or to /dev/null.)

Unfortunately, there is no way to increase entropy in the primary pool via standard system command line tools or character devices. However, there is a Linux ioctl, RNDADDENTROPY, which does this. So I wrote a small C wrapper, which takes STDIN and feeds it to the ioctl. This requires root of course. The utility is called, boringly enough, rndaddentropy, and is distributed with the twuewand tarball. It will be built by `make` on Linux systems.

I must point out that this utility gives you a very excellent method to shoot yourself in the foot. The lack of command line tools to directly access the primary buffer is most likely by design, since this bypasses most of the in-kernel filters for random input. Injecting, say, the contents of /bin/ls to the primary pool would be a great way to become one of those 1 in 200 statistics. Only use this utility to feed high quality entropy (such as by twuewand, or something like an entropy key).

Dan Kaminsky will be publishing software in the future called DakaRand, which does much of what twuewand currently does, but incorporates some of his other ideas. He provided me a work-in-progress copy, which looks very interesting, but it is not available to the public yet for a number of reasons. Be on the lookout for that when it is released.

Update (2012-08-15): Dan released DakaRand 1.0 a few hours after I made this post. Go check it out.

[1] In Dan's proposal, after the SHA256 hash is computed, it would then be run through Scrypt's hashing algorithm. This is not done in twuewand for two reasons. First, Crypt::Scrypt does not currently provide a low-level method to just do hashing; instead it wants to create a full digest which is unsuitable for this purpose. Second, Dan has been debating whether this step is necessary or desirable at all, and Scrypt has "undefined effects on embedded hardware".