Esoteric Curio - OpenSolaris

A talk with Ben Rockwood

nospam@example.com (Theo Schlossnagle) — Sat, 03 Jan 2009 11:07:31 -0500

It has been a long time since I've done an interview with anyone. Recently, I had the pleasure of being interviewed by the esteemed Ben Rockwood for his SA Pro program. Ben has been in the industry a long time and his standards are what standards should be. It is nice to speak with someone in the systems administration field that works in an Internet-centric environment and still respects his work as a trade and presents himself as a craftsman.

While it is a long listen, clocking in at over an hour, I'm quite pleased with the interview. It covers a variety of topics like how OmniTI started, a bit about the culture that is instrumental to staff retention, service quality and customer satisfaction. Feel free to spend an hour of your time listening to a bit about operations, technology and culture at OmniTI.

A month later... like clockwork.

nospam@example.com (Theo Schlossnagle) — Mon, 08 Dec 2008 22:40:52 -0500

A great success occurred for the Apache httpd project today. A month ago at ApacheCon I presented my "Apache Projects on DTrace" talk where I presented some magic mojo around a DTrace enabled Apache httpd. A few days ago Paul Querna invested the time to putback some of the patch set required to make this go. The code is all in, just a bit of build magic remains to utilize the parts. Hopefully we can work to put that part back soon. Paul, I know your time is scarce and valuable, many thanks for the effort!

DTrace and Apache

nospam@example.com (Theo Schlossnagle) — Wed, 05 Nov 2008 19:05:20 -0500

DTrace kicks ass. Everyone knows I love ZFS, but in my opinion DTrace is the most significant and disruptive systems technology in at least 10 years.

Today I gave my talk at ApacheCon; it was well received. I hope that I inspired some people to go use DTrace and look in their systems' underpants.

Here are my slides and the scripts I used for my live demo.

I'll update this entry with the video as soon as I have the link.

I mentioned a Linux port underway. I got the name wrong when I talked about it (sorry Paul!). The person working on the port is Paul Fox, the author of the CRiSP editor. For more information, check out his blog.

Eloquence in technical prose.

nospam@example.com (Theo Schlossnagle) — Wed, 05 Nov 2008 01:00:15 -0500

I met Bryan Cantrill a few years back at OSCON where he gave a session on DTrace. As any reader of my blog would know, I'm a big DTrace fan. Bryan's presentation style was passionate and animated and I was quite convinced there was an illegal substance involved. I've come to learn that style is Bryan's style.

Since then, I've had a few opportunities to dialogue with Bryan and on every occasion he reinforced my opinion that he is one of the more insightful minds in today's computer engineering field. His critique and vision has an element of clarity and completeness to it that is far too rare. Combine that with truly elegant written prose and I am unable to resist consuming what he writes.

Bryan's latest communication on transactional memory is an exquisite example of poise in both thought and communication. At one point he states that "experience has taught me to be wary of crooked problem statements." I must admit that in my efforts to play devil's advocate I often construct straw-man arguments and unjustified counterpoints. I do this in an effort to force lead my counterpart to conciseness and deeper well-founded justification in their argument. Backing out of a solid justification is, at the very least, revealing.

I took two things away from his prose. First, his output is very worth digesting. Second, when I present solutions to problems I should vehemently avoid perfunctory problem definition and invest equal efforts in the completeness of both the problem statement and the solution proposed.

Zetaback. Respect.

nospam@example.com (Theo Schlossnagle) — Fri, 19 Sep 2008 10:02:15 -0400

With the release of ZFS on Solaris 10, I sat down and marveled at the opportunities for off-site backups. I have already written a bit about ZFS detailing why I think it kicks so much ass. With zfs send and zfs receive, one can manage block-level incremental backups and restores. What's missing? An elegant hack leveraging that to provide a simple and reliable backup infrastructure for a network of ZFS capable machines (including Mac OS X and FreeBSD now, BTW).

So, I sat down and wrote Zetaback -- which is currently 1032 lines of perl code (including complete documentations) plus a thin agent on remote machines that is 290 lines of perl code (including complete documentation). I'd like to note that the only reason there is documentation, let alone complete documentation, is because of Eric Sproul. This really demonstrates to me that "Keep It Simple Stupid" still works for important tasks.

Zetaback is a rather full features backup and restore system. It can manage multiple hosts, multiple ZFS per host, both frequency and retention policies on full and incremental backups. It can report policy violators (things that haven't been backed up within the policy). It can manage the archiving of backups. It provides both non-interactive and interactive restores. It has an excellent command line syntax. And most importantly, it has saved my ass more times than I can count.

I'm not usually big on awards... I find the single unexpected email from someone saying: "damn that was useful, thanks!" to be more gratifying most of the time. However, Zetaback was one of the first projects we put up on labs, so being a 3rd place winner in the OpenSolaris Community Innovation Awards is pretty exciting.

Varnish, get your patch on.

nospam@example.com (Theo Schlossnagle) — Sun, 10 Aug 2008 21:50:00 -0400

Varnish is a "bad ass" new HTTP caching accelerator. It's developed by some crufty old BSD hacker and has a lot of Linux users. By and large, it has ignored Solaris. This sort of neglect isn't malicious, it is just neglect... you know: "out of sight, out of mind."

Well, check out Varnish trunk and give this patch a spin. Let me know what you think.

Perhaps one day, the Solaris networking team (or someone else) will satisfy this pretty abysmal shortcoming: BugID 4641715.

OSCON2008 Presentation

nospam@example.com (Theo Schlossnagle) — Thu, 24 Jul 2008 03:58:09 -0400

Hello from OSCON. I gave my full-stack introspection crash course talk today. It has been quite a while since I've presented anything in a 40 minute format, but I think the talk went quite well. I got a lot of positive feedback.

I decided to take a risky approach inspired by dtrace.conf(08) by demonstrating dtrace on a live, mission-critical system we run at OmniTI. The risks of this are: network connections flake out, dtrace doesn't work correctly or I do something stupid and cause some service unavailability. Well, as I use dtrace just about every day, I wasn't worried about the breaking things. And while my network connection winked out for about one minute and dtrace has some annoying sub-second aborts, I think the demonstration was quite effective.

Many people gave positive commentary at the end and afterward. People asked for the slide to be put online... and while they have no real content of value (as the demo was everything), I put them here anyway:

OSCON2008 Full-stack Introspection Crash Course

view presentation (tags: oscon dtrace)

In addtion to the slide stack, I've included the simple scripts that I used during the demonstration. I ran qps.d, query_speed.d, and query_speed2.d on the database server and r3.sh, perl.d, pcpu.d, papcpu.d, and papcpu2.d on the web server. These require The Devel::DTrace perl module, PostgreSQL patches, and Apache 2.2.8 patches. Some of them are approximations of correctness, so weigh the output appropriately (the perl ones). Enjoy!

Reconnoiter and another platform

nospam@example.com (Theo Schlossnagle) — Fri, 27 Jun 2008 17:27:27 -0400

Reconnoiter is coming along. Unlike most open source project, I tend not to talk about mine until their are really useful to people. Over the last year, I've adopted the unhealthy attitude that useful means "shiny front-end." So, I'm blogging to break that attitude and talk a bit about project that doesn't have a shiny front-end... yet.

Reconnoiter is built out of years of frustration using tools like RRDTOOL, Munin, Cacti, ZenOSS, Nagios, etc. etc. I have a lot of problems with these tools. First, they are not efficient. I need a powerful machine to monitor a mere 10k services. And it actually gets to be an engineering challenge to monitor 100k services with these tools. Also, the graphs are about 10 years old with respect to design and usability. I want something new, something fresh, and something that doesn't need a damn web UI to configure. Several people have asked, why are you reinventing the wheel? Why don't you just improve an existing product? My answer is that I want a well-thought-out product foundation so that I can trust all the bits. I want reponsibilities decoupled at the right spots. I want data in a form that the world can query and run reports the likes of which I have not concieved. I don't want the load on my monitoring machines to be 8. I want my monitoring system to check services and metrics when it planned to, not several minutes (or event 2 seconds) after it told me it would. Simply put, I expect it to work well, all the time. And, of course, I want it to work how I would expect it to work.

Reconnoiter was born out of the need to monitor the internals of many disconnected data centers with between 10 and 1000 machines in each facility. Monitoring can mean a lot of things, here I consider it to be the collection of metrics and awareness of their availability. In and of itself, monitoring is pretty useless, but it is the foundation for two critical pursuits in Internet infrastructure and business management: fault detection and trending.

Fault detection is as simple as understanding when something has faulted. However, knowing something is broken is easier than knowing something is about to break. Is it better to know that your machine just crashed because the chip slagged to the motherboard, or that the temperatures in rack 043 are rising unexpectedly? Answer: both, but I hope I only learn the latter and not the former. Truly, there are too many things to monitor... hundred or thousands of metrics on each piece of equipment. I can't reasonable go in and configure good/bad thresholds on each one. I want anomaly detection. I want a system that I can say: "this looks right, tell me when it stops looking right." That, to me, is a much need companion to tradition fault detection.

To me, trending is much more than drawing graphs... it is about intelligent data correlation, regression analysis/curve fitting and looking into the past to see how much you fucked up getting where you are now -- in the vain hope that you learn from your mistakes and plan better next time.

Reconnoiter is an attempt to build these things. Building a system requires starting with pain (need), solid structure and plumbing (good engineering). So, reconnoiter is underway. And this post is in mid-step:

It started on OpenBSD, and added support for FreeBSD, Mac OS X, Linux.

As of changeset [292], we have Solaris/OpenSolaris support.

We have a pretty nice front-end for trending under construction, but it isn't there yet. We'll have numeric data combined with textual "event" data on the same graphs. All that convenient stuff. Here's the rather plain-Jane graph you get now (because some people won't even read a post if it doesn't have a pretty graph):

Honestly, I don't know what the value of this post is, but people around here keep telling me that people should be aware of an open-source tool like this, even if it isn't finished (read: usable) yet. I say it isn't usable yet, but on our development instances here, we monitor 2892 production metrics across two data centers and the load never peaks past 0.10. I'm pretty excited about where this is going. Honestly, my favorite part right now is that I can configure and control the noitd checking nodes via a telnet console and it acts as if it is a piece of network equipment rather than an "application" -- as it should be IMHO.

ZFS. Respect.

nospam@example.com (Theo Schlossnagle) — Tue, 22 Apr 2008 23:16:32 -0400

Today someone asked me: "You speak about ZFS a lot. I know other people that talk about the latest filesystems with praise, but generally speaking they just don't have much to offer. Is ZFS that different?"

My answer is "yes." But, of course, I can't leave it at that. I'm not going to make a performance argument -- ZFS is fast in some cases and slow in others -- just like everything else. I think one of the things we've seen in the last 10 years is that everyone felt the need to come out with their own filesystem -- at least on Linux. So, you have to as yourself why. My personal opinion is that filesystems on Linux suck.

Most filesystems on the market support snapshots. No open source filesystems on Linux (that I'm aware of) support snapshots. Of course, you can use LVM to do block-level snapshots. First off, that's a pain in the ass w.r.t. storage provisioning. Other systems make the process of allocating and managing snapshots "not my problem." (simple and easy). Let's be frank, ext2 and ext3 are nothing to write home about. reiserfs, xfs, jfs, the list goes on and on.

There are a few closed-source filesystems that are really nice. Specifically Veritas Filesystem (VxFS) and its excellent layered volume manager VxVM which appears to have heavily inspired geom on FreeBSD. DEC thought it was so cool that they pulled it white-label into Tru64. Respect.

So, what makes ZFS so different? ZFS is a disruptive technology as it abolishes the sacred line in the sand between block devices, volume management and filesystems. This means it just make storage management easy. When I say easy... I mean easy.

So you want more space? Add more disks. Want to move from from failing disks to replacements? Tell zfs to add the new ones and tell it to remove the old ones. Read that report by Google about disk errors? ZFS checksums all data. My personal experience says checksums are good. Snapshots? Sure snapshot to your heart's content. We snapshot some systems hourly and never ever delete the old ones. Snapshots are really cool, but what if you could rollback to a snapshot? zfs rollback. What if you wanted to make a read/write copy of the fileystem or an old snapshot? zfs clone. You want to store a lot of raw data? zfs has built-in compression. Oh, and it is open-source.

Simply put. ZFS. Respect.

PostgreSQL: Looking under the hood with Solaris

nospam@example.com (Theo Schlossnagle) — Sun, 30 Mar 2008 09:21:05 -0400

For those interested, here is my slide stack from PostgreSQL Conference East '08. I think the title of the talk was "PostgreSQL: Looking under the hood with Solaris."

The presentation was 90 minutes long and had lots of shell-based show-and-tell. Obviously that stuff isn't available in the slides. I think it went over quite well. The audience was small, but hopefully people took away the a lasting impression of what DTrace has to offer and at least one person had the response: "By the power of Greyskull." Regardless, enjoy the slides!

Talking w/ Sun

nospam@example.com (Theo Schlossnagle) — Sun, 16 Mar 2008 23:21:55 -0400

New acquaintances walk away from their first conversation with me and either think that I am in love with a particular vendor or technology or they think I truly hate all technology. Both are true in some fashion.

The fact that I have an OpenSolaris feed on my blog might indicate that I'm a fan of Sun. The truth is I am and I am not. As is true of any large organization, it's really tough to be enamored with all of it. I am a huge fan of Solaris 10 and Sun's initiative to support strict ABI compatibility for stable interfaces, and I'm downright giddy about their ZFS and DTrace technologies. I think Zones/Containers are cool and I think their engineering team has some brilliant shining stars and is on the whole smarter than average. Yet, the OpenSolaris community is challenged in a lot of ways due to the corporate involvement by Sun that leaves me with a funny taste in my mouth. I'm luke-warm about Java and feel like their hardware initiative is going down-hill with bad quality problems on some of their new offerings compared to their spectacularly rock-solid history (sans the E4500).

Recently, I did an interview with Mark Thacker of Sun about our use of Solaris for their Solaris Podcast series. We've had some bad experiences here and there, but all-in-all it has been a win. DTrace has been a god-send and ZFS has saved my bacon several times. Anyone who's talked to me knows that I'm brutally honest and appreciate those that return the favor. I'll look at solution and I tell you what you did wrong. I don't tell you what you did correctly... after all it was all supposed to be done correctly. So, upon listening to this Sun podcast, several of my colleagues said: "that had to have been edited." As marketing people usually do, they attempt to limit the negative exposure as much as possible -- most notably, they removed a section about lack of tight integration between ZFS and Zones which has made for some very painful upgrade paths. We have marketing here at OmniTI too, I know the drill. All-in-all, I think the interview went rather well and fairly represents the benefits we've realized by deploying Solaris 10.

dtrace.conf(08)

nospam@example.com (Theo Schlossnagle) — Fri, 14 Mar 2008 21:28:49 -0400

As many people already know, I'm a big fan of DTrace. Well, today I attended dtrace.conf(08) -- the first (un)conference revolving around planet DTrace. It was awesome. Many people who know me well have heard me say, "my good days are when I'm the dumbest person in the room." That's not to be confused with "I like having bad days." Instead, I like to be at my best and still struggle to keep up. Here at dtrace.conf(08), the people here are damn smart -- a bunch of higher level thinking.

I gave a demo of the PostgreSQL stuff we do using dtrace. I put back my SDT postgres probes into our internal packaging systems -- I hope that I can get those back into PostgreSQL soon... I talked very briefly about the ZFS magic we've experienced, but as the conference was focused intensely on DTrace, I saved most of that for PostgreSQL Conference East 08.

As usual, I particularly enjoyed Bryan Cantril's infectious excitement and pungent humor. I'd like to thank Sun and the DTrace Team for running the conference (as well as the sponsors). I really appreciate the effort. DTrace has empowered OmniTI to make our customers more successful. Hands-down awesome technology.

I've said it before and I'll say it again. ZFS and DTrace are the most impressive operating system advances I've seen this decade.

Spine-tingling dtrace euphoria: "What the SCSI?"

nospam@example.com (Theo Schlossnagle) — Thu, 20 Dec 2007 08:52:01 -0500

I am oft in the position where SCSI is not working right for me. Whereas I always seem to be in the position where SATA over promises and under delivers. When SCSI misbehaves, it usually really matters. Where's my disk go? Time outs? And the king of all frustrations: wassup with that tape robot?!

This clearly puts me in the sysadmin geek category, but I had to share. I had a spin-tingling euphoria when I read the details of SCSI packet sniffer written in dtrace. OMG this is cool: http://blogs.sun.com/chrisg/resource/scsi_d/scsi.d-1.12.

Zimbra on ZFS and Zones.

nospam@example.com (Theo Schlossnagle) — Mon, 29 Oct 2007 22:13:59 -0400

So, we're moving from an old Cyrus installation to Zimbra. We considered moving to Dovecot, but the need for tight corporate calendaring that isn't hosted by big brother (Google or Y!) was too strong.

So, I call up Zimbra...

Me: Gimme that for Solaris.
Zimbra: No.
Me: Umm... It's open source and the "closed" bits are non-native Java.
Zimbra: Umm... It doesn't work on Solaris, just install Linux.
Me: Just give me a sound off-site backup strategy that works as well as ZFS.
Zimbra: Umm... Install Linux, it's easy.
Me: You're kinda missing the point.

I then turn to Sergey, master of brute force, on our SA team and said: you've got five days to run this on Solaris. Sergey says: "done."

From here, we test quite a bit, doing migrations from the existing solution and installation tests. Whiz, bang, whirrrrr. We're happy, and I go back to Zimbra with some trepidation.

Me: How much?
Zimbra: Fill out this form and fax it.
Me: Okay... [faxed form]
Me: I sent it.
Zimbra: Got it, you should have your license.
Me: Wow, that was easy.

While I like free software a lot. Boy is it refreshing to find a company that knows how to take your money without pain and suffering, like some companies I know.

I'm sure that Zimbra is just trying to limit the support platforms to help control their engineering costs. The support for Linux is a no-brainer -- it's the most popular new deployment platform for just about everyone. But, support for Mac OS X... as if there is demand for that in the enterprise?! WTF? I know all about developing mail products and I understand that Mac OS X support is often iintroduced due to the engineering staff owning MacBook Pros. But, come on... Tier 1 support for Mac OS X before Solaris, not very good market research here fellas.

Anyway, we've got Zimbra cooking on a sweet Solaris box attached to a Sun StorEdge T3. One zone runs Zimbra Network Edition and another zone runs Zimbra Open Source Edition. Both of these running on some nice ZFS mount points that are conveniently backed up with Zetaback... Bliss.

Oh yeah. One detractor to my bliss: I spent 2 days deleting old mail folders and eliminating cruft (built up over the last 10 years). 11 hours of imapsync to get my now down to 14 gigabyte mail store over is just painful. There should be a universal mail store format so that I can "burn" by old mail into a universal archive format and just plug that into mail any mail server. Right.

PostgreSQL warm standby on ZFS crack

nospam@example.com (Theo Schlossnagle) — Mon, 02 Jul 2007 21:06:32 -0400

So, the state of open source database replication is pretty sad. MySQL replication just doesn't cut it in many serious environments because the slaves can't keep up with the write load on the master. So, PostgreSQL right? Well, not so fast. PostgreSQL replication is handled in one of two ways: Slony or PITR (point-in-time recovery).

Slony provides all the same features as MySQL's replication (except that it is much harder to setup and maintain), but also boasts the same acute performance issues -- a busy master can easily outpace slaves, leaving them in the dust. Query-log-based replication is pretty flawed and while creative people will do whack shit to try to push the envelope this still doesn't make it a good method.

PITR is much more like Oracle's replication mechanism. PITR takes the WAL (write-ahead logs) and ships them over to the slave to be reapplied. This leaves you with an identical database (block for block) and a weak machine can easily keep up with a beefy master. In Oracle terminology "WALs" are called "archive logs."

So, with PITR, all our problems are solved, right? No. When using a PITR-style warm standby configuration the database is in "recovery mode" all the time. This means the database is "sorta up" waiting for the next WAL log to appear so that it can play forward through the transactions and "catch up" to the master: "continuous recovery." This means the database isn't available for queries. Now, Oracle works the same way. While Oracle is recovering, you can't use the database. However, using Oracle you can cancel recover, mount the database read-only, do some queries, unmount and begin recovery again picking right up where you left off. In PostgreSQL you cannot open the database in read-only mode and then later continue recovery as the act of opening the database (even in read-only mode) will deviate from the path of the recovery -- can we say design flaw?

While Oracle's "got game" on PostgreSQL, the concept of stopping recovery so that we can run queries on the slave isn't ideal. If my queries are substantial my "warm standby" will get colder and colder as it sits neglecting to apply archive logs. So, I want my warm standby and I want to be able to run long, heavy queries against it. And someone's going to give it to me!

However, I'm impatient. So, I'm going to make it work myself. Using the power of ZFS, I'm going to snap my PITR slave and clone it into a "disposable" "point-in-time" copy. This is really useful for running heavy reports.

The basic concept is this:

I've got PostgreSQL running on my box as a PITR slave. The master is pushing WAL logs over and this box is running in recovery mode.

Per best practices, my postgres data directory, xlogs and WAL archives are on different filesystems (ZFS of course).

intmirror/postgres/82_xlogs  64.1M  66.8G  64.1M  /data/postgres/82_xlogs
store2/postgres/82    10.5G  1.97T  8.15G  /data/postgres/82
store2/postgres/82_walarchives 14.4G  1.97T  3.89G  /data/postgres/82_walarchives

zfs create store2/clonedb
I create zone called "clonedb" with a zonepath /zones/clonedb
I make 'store2/clonedb' subject to an 'add dataset' in the clonedb zone configuration.
I setup the zone to run postgres just as the globalzone does.

Now, all I have to do is get a copy of the PITR stuff into that zone. There are a few caveats: (1) due to postgres' design the copy must be read-write as it will be destructive even in read-only mode and (2) it will still be in recovery mode, so I'll need the last WAL archive so that it can "finish" recovery before I bring it online.

ZFS gives us cheap, fast read-write clones of filesystems:

In the global zone:
zfs snapshot store2/postgres/82@clonebase
zfs clone store2/postgres/82@clonebase store2/clonedb/82

In the clonedb zone:
zfs mount /store2/postgres/82
zfs set mountpoint=/data/postgres/82 store2/clonedb/82

I need to make sure I copy the latest WAL archive from the PITR slave into the pg_walarchives directory on my clonedb zone. Then I just startup my postgres instance in the zone and touch the /data/postgres/82/failover file (this file tells my recovery script to stop recovering and start up normally). Viola.

It might sound complicated, but we just run ./clonedb_startclone.sh and in about one minute we have a fully operational read-write database and the PITR slave is merrily continuing recovery.

# ./clonedb_startclone.sh
[Mon Jul  2 20:37:14 EDT 2007] Stopping postgres in clonedb
[Mon Jul  2 20:37:20 EDT 2007] Dropping clone and base snapshot
[Mon Jul  2 20:37:38 EDT 2007] Snapshot store2/postgres/82
[Mon Jul  2 20:37:39 EDT 2007] Clone to store2/clonedb/82
[Mon Jul  2 20:37:41 EDT 2007] Mount store2/clonedb/82 at /data/postgres/82 in clonedb
[Mon Jul  2 20:37:43 EDT 2007] Copy last WAL [0000000100000016000000FA]
[Mon Jul  2 20:37:43 EDT 2007] Make it active [induce failover]
[Mon Jul  2 20:37:43 EDT 2007] Start postgres in clonedb
[Mon Jul  2 20:38:07 EDT 2007] System up

Now I can run long data mining queries and other complicated reports against my standby database. No load is induced on the master database at all (so no concern about negative production service effects) and the standby recovery is continuing on unaffected, so from the failover point-of-view nothing has changed either. I am not even limited to one zone! Any time I'd like to, I can just "snap" myself a new query slave. It's a cheap, mutable, entirely disposable copy. Nice.

It's worth noting that this same technique should work like a charm on Oracle as well. Also, it should work well with any other filesystem that supports copy-on-write cloning -- though I don't know of any other than ZFS.

This, in a long line of things, just lets you know that when your database doesn't quite have the spunk to finish the race, today's operating systems are actually powerful enough to drag them across the finish line.