Introducing Unladen, easily scalable object storage

In 2002, I managed a datacenter's IS infrastructure. We had a few dozen servers, and a terrible backup server which, in theory, backed up servers to tape. In theory, anyway. I've never encountered a commercial backup server which has worked well. Anyway, I thought, "We have these servers, and they all have free space to some extent. I'd like a system which did backups of servers, split them into chunks, encrypted them, and sent multiple copies of them to the other servers." Nothing ever became of that idea at the time.

Fast forward 12 years. Everything is now Yay Cloud, and many are familiar with object storage, likely through Amazon S3. OpenStack is quickly becoming a large part of many organizations, and OpenStack's object storage component is called Swift. From a client perspective, Swift is a very nice system. The standalone command-line client is decent, and the API is HTTP and very RESTy: "PUT /v1/account/container/object" to upload an object, DELETE to delete it, etc.

The problem is I've never been very happy administering it. The server software is finicky, and for the most part, it requires a very homogeneous storage infrastructure. You'll want to have storage nodes with the same amount of storage per node, and if you have multiple disks per node, you're required to work out the optimal weight map of the disk compared to the entire cluster. The location of data on the nodes is done via hash rings. In theory this is nice since you can work out the location of an object within the cluster based solely on the ring definition file, but in practice it means a ring configuration which you must (manually) keep up to date on all the storage/proxy nodes, and any sort of change to the cluster (losing or adding a disk or node, etc) means multiple total rebalances of the cluster.

This made me think of my idea from 2002, and the result is (or will be) Unladen. Unladen is a Swift client-compatible system which takes a much different approach to the backend.

First, let me say that while I've written a lot of code in the last few weeks and things are looking very usable, this is nowhere near production quality. Everything is still very, very pre-alpha, and it seems that every commit I make does things like breaking compatibility with the old schema, etc. While I encourage you to download and play with it, don't trust it with anything more than "Hello World". Oh, and there's no sort of authorization yet, so everyone (including unauthenticated users) have full access.

Unladen is Swift API-compatible, so if you have an application which supports Swift, it'll likely work out of the box with Unladen. But on the backend is where things get more interesting. All object data is encrypted, and is sent to storage nodes that way. Only the trusted catalog nodes have the keys to each object's data, so storage nodes do not have to be trusted.

In addition to data trust, Unladen also has a concept of availability trust, also known as confidence. Say you have a core set of storage nodes, and give a confidence of 100% to each. You can also have nodes which you do not trust to be as available. You can then define replica targets for certain sets of data. The default is a replica target of 3.0, which means an object's (again, encrypted) data would be stored on three 100% confidence nodes, or six 50% confidence nodes, or a combination of them.

Replica targets and confidence determine how much to store, but weight defines where to store it. And weighting is easy with Unladen. Each storage node can have multiple internal "stores", which are just directories. You just tell Unladen how much each store can hold. (In most cases this will simply be mount directories of disks, and "how much" would be ~90% of the disk's filesystem capacity.) When data is placed on a mount, the weight of the stores' sizes determines the balance. Likewise, each node advertises its total storage capability (or a portion thereof), and balancing across the cluster is done according to the automatically-determined weight map of the cluster.

Unladen is designed to be a cheap, easily scalable object storage system. "Cheap" in the sense that you can easily use your infrastructure's existing servers' spare space to create (or add to) an ad-hoc cluster. Or you could build massive dedicated 500TB storage nodes. Or a combination of those two extremes.

Again, this is a very early work in progress. I was a bit hesitant publicly announcing it before it was "ready", but things have been looking good, and I didn't want this to turn into a "90/90" project (90% done, just need to finish the other 90% before I release it, which of course never happens). And the last major project I announced before it was ready was 2ping, which turned out to be a great success.

Note that this project is purely a personal endeavor, and is not supported or endorsed by the OpenStack project or my employer.

A green bike

New bike
(Is it green? Yes, very.)

Six years ago, I bought a $360 entry-level "hybrid" bike from a local bike shop. This weekend, I decided to step up to the next level... by buying a $200 big-box bike from Target. Wait, what?

When I bought my last bike, I knew almost nothing about bikes, went into a shop armed with only a little research, and walked out with something. It was a decent quality bike, but there were lots of things I ultimately didn't like about it. It was billed as a "hybrid" bike, but it would be hard to explain how it was different from a mountain bike. The handlebars were way too high and I didn't like the curve of them. (The stem was adjustable, and I eventually did lower it down, but that brought it very far forward.) It was quite heavy. The tires were very wide and very knobby. I didn't like the front fork suspension. It was very high up and the seatpost had its own suspension which took up a lot of space, and as a result the lowest possible position of the saddle was the bare minimum acceptable for me. And while it had a partial chain guard, I never used the front high or low gear.

None of these downsides by themselves were terrible, and I told myself I wouldn't use them as an excuse to not ride, and I wouldn't address them until I had been riding more. Well, I didn't ride much until about the last year, and in particular I've been riding at least three times per week throughout the spring, so recently I decided to look around.

I ruled out new bikes as I didn't want to spend more than about $400. Last weekend was Earth Day, and I biked to the celebration downtown, where the local Kiwanis bike group had brought their inventory. Some of the bikes looked good, but didn't satisfy everything I wanted. I was planning on continuing looking for used bikes (I had not yet visited the Reno Bike Project), but this weekend somehow ended up on Target's site. There they had a Schwinn Median bike which looked good on paper:

  • Lighter and lower
  • 700c wheels
  • Tires are wider than a road bike, but narrower than a mountain bike (700x38), and are semi-slick (or at least not as knobby as a mountain bike)
  • 7-speed, no front derailleur, and the front chainwheel is recessed into a nice chain guard area
  • Mostly flat handlebars
  • $200

I found a review of the Schwinn Median which says, "Never has bike had a more appropriate name. It occupies the exact space of crossover between mountain bike, road bike and cruiser." Given the specs, that is quite accurate, and is almost exactly what I was looking for.

The one at the store was decently assembled, so I bought it. I spent some time adjusting the derailleur and brakes, but the rest of the components were decently aligned and tight. I transfered over the lights, rack and saddle from the old bike, and have been enjoying it.

I still have the old bike, but it's been downgraded to "Burning Man bike" status.

(N.B.: San Francisco's Bike to Work Day is this Thursday.)

Now, I'm sure I just spat in Bike Church. I bought a big-box bike as an upgrade, and I'm happy about it? Heresy! To which I reply, meh. There was one bike which happened to satisfy my personal checklist, and it happened to have the name "Schwinn" on it, and was likely assembled by a teenager. The frame is fine. The components are good. The gear system is acceptable. It feels comfortable. And it was priced decently.

I should point out that's not guaranteed to always be the case. I'd consider myself firmly a bicycle semi-amateur these days, so I at least knew what to look for. Interestingly, while the Schwinn Median has probably sold about a million bikes, there's not much mention of it online. One of the top results is a blog post by a family showing it off, where the pictures clearly show the front fork was attached backwards.

(N.B.: There is actually a forum dedicated to discussing bikes from big-box stores.)

Compiling BusyBox with uClibc

Finnix uses BusyBox for its initrd, and that BusyBox installation requires a custom patch. In the past I've compiled BusyBox with uClibc to keep the size down: a full static BusyBox binary is about 700 KB when compiled against uClibc, and about 1.8 MB when compiled against glibc. For a LiveCD distribution which prides itself on its balance between size and features, 1.1 MB of unneeded bloat is a lot.

In the past, I've used a development Buildroot chroot to compile BusyBox, as the documentation recommends. However, the pre-built chroots on the Buildroot site are quite ancient, and I never had success in having buildroot compile an equivalently-featured chroot. On top of that, recently they deprecated building a target toolchain in the chroot itself.

See, it's not just a matter of "build against uClibc". The libc is tightly coupled to the toolchain, a set of utilities such as gcc, binutils, etc. In my case, I don't care about cross-compilation and just wanted to build a native uClibc development toolchain chroot on each supported Finnix architecture (x86, amd64 and powerpc) for the purpose of building BusyBox.

As it turns out, I don't need the full chroot, just the toolchain, and it's rather easy to do, though a little time consuming and not immediately obvious.

Download and extract Buildroot and BusyBox. Here we're assuming:

BUILDROOTDIR=/home/ryan/buildroot/buildroot-2013.11
BUSYBOXDIR=/home/ryan/buildroot/busybox-1.20.2

Configure Buildroot:

cd $BUILDROOTDIR
make menuconfig

Be sure to select the proper architecture information under "Target options". Since Buildroot is primarily a cross-compilation tool, it defaults to i386, not the host target. Indeed, you can even use this method to cross-compile BusyBox.

Under "Toolchain", enable the required uClibc options. In my case, my rather fully-featured BusyBox config required the following toolchain options:

  [*] Enable large file (files > 2 GB) support
  [*] Enable IPv6 support
  [*] Enable RPC support

That should be it! You don't need to bother with anything under "Target packages", since we're not actually going to be compiling the full target chroot, just the toolchain. Start the build:

make clean && time make toolchain

This will download a bunch of sources and will require some time to compile; on my relatively fast AMD64 build environment, the build portion takes about 10 minutes.

Once done, the uClibc toolchain is in output/host/. You may use this to compile BusyBox:

cd $BUSYBOXDIR
make menuconfig
export PATH=$BUILDROOTDIR/output/host/usr/bin:$PATH
make clean && time make CROSS_COMPILE=$(cd $BUILDROOTDIR/output/host/usr && ls -d -1 *-buildroot-linux-uclibc)- busybox

Note that the toolchain is not easily relocatable.

On phones of a cellular nature

In 2000, I walked into a Pacific Bell Wireless store on Market St. in San Francisco and signed up for service. For the next twelve years, I stayed with them through six different names:

  • Pacific Bell Wireless
  • Nevada Bell Wireless (technically different service when I moved from California to Nevada, but they were both owned by the same company)
  • SBC Wireless
  • AT&T Wireless
  • Cingular Wireless
  • AT&T Mobility

In 2012, with the release of the iPhone 5, I upgraded from my iPhone 3G. At the same time, I switched from AT&T to Verizon, to save a little money (from $100/mo down to $90/mo), and because at the time, Verizon had LTE service in Reno, but AT&T did not yet. (AT&T did launch LTE in the area a few months later.)

On Wednesday, I heard the announcement that T-Mobile would be paying off the Early Termination Fees of up to $350 if you switched to them, traded in your smartphone for up to $300, and bought a new smartphone with them. Their plans are decent, offering unlimited talk/text/data (though the data is throttled after a certain point depending on what you pay, but they tell you this up-front) for as low as $50/mo. They've also got the system where you essentially finance the unsubsidized cost of the phone over the course of 24 months, rather than the telco subsidizing the cost of the phone into higher monthly plan costs. For most people, assuming a new phone every 2 years, the savings would be modest, but there would be savings.

After I looked at this offer, I didn't think it was worth it in my specific case. ETF on Verizon would be $200 ($350 minus 15 months at $10/mo), and they offered $215 for my iPhone 5 32GB. I'd have to get a new phone, so say the equivalent iPhone 5S 32GB for $100 down and $25/mo for 24 months. Net savings after 9 months (when my Verizon contract would have come up): $250, except I'd still have 15 months of a "contract" left in the form of $25/mo payments on the new phone.

Instead, I decided to switch to T-Mobile anyway, eating the $200 Verizon ETF and keeping my existing phone. I'm now paying $50/mo with T-Mobile, instead of $90/mo with Verizon for effectively the same service. Net savings in 9 months will be $200, $50 less than the scenario above, but I will have no obligation at any point from now going forward, unless I decide to buy a new phone sometime soon.

And therein lies the mental benefit. If I would have stayed with Verizon, in 9 months when my 24 month contract would have been up, I would have still been paying $90/mo. It mentally encourages you to get the latest and greatest when your contract is up, to justify the price, but that locks you back into a contract. With this new T-Mobile service, I can continue to use my existing phone as long as I want. Or hey, maybe the iPhone 6-X With Blast Processing is released in 9 months and I decide to get that. I've got choices.

"But Ryan," you ask, "your current phone was bought through Verizon and was still under contract. And isn't it CDMA anyway? How does it work with T-Mobile?" In short, it works, and well, but not to its full potential. The iPhone 5 A1429 (CDMA) model includes CDMA, a handful of GSM bands, and a handful of LTE bands. The LTE band T-Mobile uses just isn't available on my phone, so no LTE. A year ago, I would have been limited to EDGE/GSM (2G), as 1900MHz was the only common frequency, and T-Mobile had UTMS/HSPA+ (3G/"4G") on 1700MHz, which isn't available through my phone. However, in the last 6 months, T-Mobile has been moving UTMS/HSPA+ over to 1900MHz on a city-by-city basis, and Reno has already been done. HSPA+ is not LTE, but it's plenty fast. The only real downside is if I'm in a non-major metro area, it's likely I'll be limited to EDGE, but I can live with that.

As for the contract issue, here's a fun fact: All Verizon LTE-capable phones, including the iPhone 5/5S, are completely unlocked. They're actually required to as an FCC condition of their allocation of LTE band 13. So my "CDMA" phone is actually a fully capable GSM phone out of the box. This was useful for me as I've been to London twice in the 15 months since I got the iPhone. I can literally walk up to a vending machine in Heathrow and buy a short-term SIM card. (Though crazily, T-Mobile offers unlimited free roaming data in many countries, including the UK, and the majority of why I'd get a SIM card in the UK was for mobile data. Woot.)

As a side note, when I switched cellular service in 2012, I got a new phone at the same time. My mind had linked the process of switching cellular providers with the reward of new tech. As a result, after switching providers again this week, for about 24 hours I would occasionally think, "I switched providers! I should go play with my new... oh, it just means I get my bill from another company."

(I think I have lost the ability to write small posts anymore. I may have a problem.)

PlayStation 4: The Review

OK, a month ago I said I'd make a review of the PS4, and tonight I finally sat down and started writing. And my god was it boring. I was a thousand words in and had only mentioned the physical console and the controller so far. So I deleted it all and started over.

Here's the tl;dr review: yawn.

It's not bad. It's just not very compelling right now. I've had an Xbox 360 for over 7 years now, and honestly, that's what I've been playing all this month (mostly GTA V). If you have a 360, play that. If you have a PS3, play that. Maybe the PS4 (and presumably the Xbox One) will be a better sell in a year, but for now, meh.

The Console: It's a boring black slab. That being said, the angled front is slightly less boring than the Xbox One.

The Storage: 500GB is not enough. It's nice that the HDD is user-upgradeable, but even 1TB aftermarket is not enough in the long run. The PS4 has USB 3, but the only ports are on the front, and USB mass storage is not supported. When each game wants to copy about 25GB to the HDD and the games' point updates are over 1GB, you're going to quickly run out of space. Sony, re-think this.

The Controller: Is it bad? No. Is it better than the Xbox 360 controller? No. In my opinion, the Xbox 360 controller is perfect in every single way, except for the D-pad, which is horrible. Conversely, the DualShock 4 D-pad is the best I've ever used. Somebody give me an Xbox 360 controller with a DualShock 4 D-pad, now!

The UI: It's not offensive, though the background music is. Or so I seem to remember; I don't remember what it actually sounds like, since I disabled it two minutes after first turning it on.

The Store: Hard to use. And it wants to sell you PlayStation Plus everywhere, even if you're already subscribed.

Knack: Technically impressive with the number of moving parts, and overall inoffensive, but not very exciting. I see this as very similar to Xbox 360's Kameo: a cutesy technology demo launch title.

Need for Speed Rivals: Every time EA releases a new NFS title, I long a bit more for 1998's Need for Speed 3: Hot Pursuit. Rivals is almost enjoyable, except now it's open world, and when you're done with a race, you can lose all the points you earned if you're busted in the open world. So after you do a race, you run back to your hideout to bank them. And there are loading screens everywhere. Graphics are pretty, though.

Killzone Shadow Fall: I only bought this because it was on sale for $30 somewhere online, about a week after the console launch. I hate console FPS games. Everything is "realistic" (brown) and indistinguishable, so it's hard to pick out enemies. So to get around this, there's a sonar feature you can activate to make enemies and objectives stand out momentarily. You need to do this every few seconds. I played for about an hour and gave up.

Resogun: This is a download title, free with PS+. It's a side scrolling ship shoot-em-up, and was a blast to play. Unfortunately, unless you're a high score chaser, there's not much replay value.

Contrast: Another free PS+ download, a third person puzzle game with a twist. Decent story, excellent gameplay, but a little short.

So there you go. $150 in AAA disc titles purchased, and my favorite so far were the two free download titles.

© 2014 Ryan Finnie

Theme by Anders NorenUp ↑