Herein, a collection of random notes from being sat in the audience of day 2 of the UKUUG Spring 2010 conference. Links inwards to the notes from specific talks follow:
Ubuntu Server, in which kitten death is minimised.
libferris, in which my mind is blown.
Hudson hit my Puppet With a Cucumber, in which the title was clever but the content largely obvious.
PostgreSQL High Availability and Scalability, which mostly wasn't about that.
MySQL HA with Pacemaker, in which much software is hated.
Sadly I epically failed to have my laptop out during the lightning talks except for while giving one. Anyway, herein the talk notes:
----
Ubuntu server development - arrived half way through due to detour for coffee. Never mind.
Taking about time based releases versus the usual "when ready" model.
"You can never find all bugs and a known bug is better than an unknown regression"
"If it doesn't kill your kitten, release with it and fix it in an update"
Notes that Debian stable released with a broken Samba due to lack of people testing with windows - not as a snipe, just as a note that different approaches and user communities yield different testing patterns.
Interesting note that LTS is still 5 years support for the server but only 3 for the desktop now. This means you have three upgrade cycles - release to release on the bleeding edge versions, desktop/server LTS to next LTS and server LTS to next-but-one server LTS - and therefore have to QA all three.
Feature freeze at release minus 11 weeks (9 normally), kernel at minus 7 weeks (3 normally).
10.04 brings integration with etckeeper and puppet - making sure that when puppet modifies your /etc etckeeper is triggered to make sure that everything is in sync.
Tomcat is now closer to how upstream wants it to work (he didn't mention how, forgot to ask at the end, sorry).
He mentions a focus on "server papercuts", minor annoyances that aren't high priority according to a normal schedule but drive admins insane, so they try and do the worst of those where there's time in between the showstoppers.
"Sysadmins test interim releases rather than milestones"
----
libferris
First slide: "every time you make a powerpoint edward tufte kills a kitten"
First slide is a ferris wheel with: Apps XML ISAM Relational DB Morphisms Awk around it. No, I don't know what that means yet either.
... ohymgod that's XQuery. I can understand this slide but I wish I didn't.
Except. It's XQuery that appears to be operating on a BDB file instead of an XML document.
ferris-doc( 'customers.db' )
Seriously.
And this can resolve an XPath expression with an attribute match to a BDB lookup rather than doing a scan. Whee.
ferrisls appears to have extended attribute support and can treat name, size and width of a PNG as such. Apparently also the content. And the md5. Pass the crack pipe.
It also has filters and sorting ... the expression syntax sort of reminds me of LDAPquery.
Slide: "Cute ... but - why?".
create virtual table fs using libferris(
The idea seems to be to treat a file as a table. Except it can treat any file as a filesystem too. Except ... I think my head hurts.
Indexing of metadata into postgres. And then you can treat the metadata as an extra set of attributes. And get them back via ferrisls.
My head definitely hurts.
echo whatever | ferris-redirect facebook://status gfcp ~/fuse/my-phone/latest.mp4 youtube://upload/
(or you can use fuse to mount things so that you see them as filesystems properly - but nice that you don't have to)
You can mount flickr, vimeo, google docs, ...
... and using several of the above mount a google spreadsheet as a sqlite table ... whee ...
"Anything that becomes a virtual filesystem can then become a target for sqlite or xquery"
X11 clipboard. Window location - "if you were so inclined you could move windows around just by catting files". Emacs buffers.
Editing into files - 'fedit new.xml/rootelement/child1'
I looked away for a second and he used XSLT two ways so you could treat a virtual filesystem as an open office document. I. Wow.
I have no idea how practical this is but it's definitely worth checking out.
----
Puppet, Hudson and Cucumber.
"Rock star developers, rock star sysadmins ... and the stubborn guys from operations."
Talking about the usual fight between devs and admins - shipping NOW versus "FFS can you test it first?"
Talking about system installation - manual, then kickstart, then custom scripting, then config management. Seeing LCFG and cfengine as "generation 1".
Mention of DSLs and the pie-in-the-sky of business people writing domain scripts.
Quick intro to puppet - great slide of sysadmin as dog, puppet as robot dog.
Talking about versioning scripts - but testing of systems stuff doesn't tend to happen as much.
Cucumber for BDD.
Noting that xUnit style testing provides TDD but doesn't really provide introspection onto what's being tested.
BDD as being about using language that describes the features rather than specifying the steps to test.
Apparently the human-readable behaviour def form for Cucumber is called Gherkin.
Showing a Gherkin file specifying mailer behaviour. Seems to adapt across to systems testing rather than dev testing quite nicely.
Sadly their syntax highlighting is pastel blue. On white. So we can't read shit.
Apparently somebody's written a cucumber-nagios. Interesting.
Talking about continuous integration for systems testing - notably Hudson.
The idea seems to be that a puppet config update triggers the provisioning of a VM which then gets everything run on it, then cucumber tests run against the results to verify it produced a working system image.
Now they're talking about dev and ops working together in harmony. You know it'll never happen ...
----
PostgreSQL High Availability and Scalability
First slide: explaining why Postgres is awesome. Notably the aim for "it just works" and the standard of quality that requires.
(your humble commentator would note at this point that it beats the ass off "it just eats your data", aka MySQL, but anyway ...)
MySQL vs PostgreSQL scalability slide - very nice graphs demonstrating how MySQL performance drops off on larger hardware and at larger scale compared to pg. Fascinating that a fork based model does so much better than a thread based one.
"16-way write scalability since 8.3, 32-way read scalability since 8.3"
Heh. Debunking arguments of MySQL + commercial extensions being more effective for data warehousing by benching it against the commercial Postgres versions - MonetDB and Greenplum basically kick the shit out of everything. Very nice.
Plus: asynchronous commit lets you turn off total safety for performance on a transaction by transaction basis.
Mention of the hstore engine to do EAV in a performant way - basically a column store but with pg as a front end.
Interesting point that the array datatype is fundamentally a performance denormalisation mechanism.
Talking about the addition of Hot Standby - so you can how have a hot slave rather than a warm slave, which is read only. i.e. a query slave that can become a master later without problems. Basically much like what MySQL replication provides except that you probably don't need a script to restart it every so often because it wedged.
And the future looks exciting - multi-node write clustering, synchronous replication and even a RESTful data access API built in. RESTful API to simplify clustering - the less state your clients have the easier it gets (Tim Bunce's work with DBD::Gofer is interesting to illustrate this).
----
MySQL HA with pacemaker
He's ... explaining why not being down is good. To a systems administration conference. "Because your boss cares" ... ok, good answer.
Interesting graph of nines of availability (nines versus downtime per year):
99.9999% 30 s 99.999% 5 min 99.99% 52 min 99.9% 9 hr 99% 3.5 day
Now summarising consistency in real time versus eventual consistency.
Split brain - http://linux-ha.com/BadThingsWillHappen
Shared storage is expensive - dedicated storage hardware, then networking, plus you need to guard against two things mounting the db at once.
DRBD mentioned (this is the simplest way I've seen MySQL HA done that actually worked) - nice to hear it's made it into the linux kernel now.
Smaller InnoDB binlogs means quicker recovery - obvious once you hear it but worth remembering.
MySQL Cluster (NDB) mentioned - data doesn't have to be in RAM anymore but the indexes do - and he's saying it's not wonderful at complex JOINs.
There's a direct API to NDB and -that- tends to be very reliable - don't go via MySQL, he says (anecdotes from friends of mine mirror this, btw)
Currently going through an overview of the master/slave commands - nothing new to me but if you find his slides it's not a bad summary.
Talking about multi-master via two-way master-slave - works fine for auto-inc provided you make sure e.g. one INSERTs odds and the other evens (slashdot does this). Except of course if you copy the config file out and forget to edit it ...
MySQL Proxy allows decisions as to where to connect/send queries to using lua code. Talking about using that to split read/write without application changes (or even to refuse WHERE-less UPDATE/DELETE statements ;) - but of course then the proxy becomes a new SPOF.
Goodpoint that monitoring replicated setups means checking that the slave has the right data, not just that it's online at all. Mention of MaatKit for checking lag and consistency and OpenARK for tracking binary log usage.
http://mysql-mmm.org/ - "Multi-Master Replication Manager for MySQL" - except that the manager code becomes a SPOF.
Flipper - interesting concept but project appears dead.
On previous heartbeat versions: "XML usage was a consulting opportunity"
Pacemaker: the cluster resource manager code extracted out of heartbeat.
ARGH! ILLEGIBLE FLOURESCENT SLIDE DISEASE STRIKES AGAIN!
Heartbeat verus AIS - AIS has heisenbugs, heartbeat was unmaintained for a while (pacemaker can use both) - appears to be a "suck it and see" sort of situation.
Talking about different ways to run stuff - basic LSB script, heartbeat resource, OCF (Open Cluster Framework) scripts, "clones" (which the presenter says tends to explode in heartbeat v2)
"I can't focus enough on the monitoring"
"Complexity is the enemy of reliability"
----
Aaaand we're done. Excellent conference, so a big thank you to everybody involved in organising it, and to the other speakers, and to the people who weren't speaking but were interesting to drink with, and presumably to at least one other group of people I've completely forgotten to mention. Looking forward to next year!
-- mst, out.