Stop making Dave look like an asshole

Another in my seven-hundred-ninety-one part trilogy on why any up-to-date recording of a sequence of ongoing events can exist only a single point in space.

The story of Ann, Carol, Dave, and Tiger

It's summer, prime season for plaid pants, hip flasks, spiked shoes, and polyester shirts. GolfCo stores everywhere are jam-packed with the leisurely, the retired, and bluetooth-earpieced executives. This all may seem like an idyllic american scene, but trouble is afoot!

It's Monday morning. Dave, manager of the flagship Palm Springs location, is reading his email. He's kinda hung over from a rowdy night at Hunters.

  • Memo A1 from Ann at corporate in Chicago - Sent 6:52pm on Saturday
    "Take down the posters of Tiger Woods Immediately"

"Uh, I think we have one in the back. I'll go take it down" Mutters Dave.
He finds the poster in the back of the store from six months ago, and takes it down.

  • Memo C1 from Carol at the regional office in LA - Sent at 3:45am on Sunday
    "Hey, we sent you some new promo posters featuring TW. Can you please put them up?"

"Ah, they must have had some contractual branding requirement or something. Gotta keep it fresh!" Dave puts up the new posters prominently in the front windows.

Later that day, Dave seems to notice some sour looks from the customers, but nobody says anything.

He goes home, cracks open a PBR, and flips on the news. To his surprise, he sees that Tiger was in the news again. Apparently he was on the Ashley Madison list, and crashed his car and his genitals into half of a Swedish women's soccer team, not necessarily in that order.

Dave immediately realizes what had happened:

"Goddammit, the time on Carol's computer is fucked up again! Her email was actually sent first!"

If only Ann had referenced Carol's email, and said something like:
"This supersedes Memo C1" then Dave wouldn't have looked like such an asshole.

This is called causal consistency. It's not just a good idea at GolfCo, it's also a good idea for databases. The reason is that sometimes clocks can be wrong, but sometimes it's impossible for them to be right.

What if...

What if, rather than LA and Chicago, Carol and Ann were on Earth and Mars respectively? Mars experiences time in a completely different frame of reference than earth. The closest Mars ever gets to the earth is 34 million miles. The farthest is 250 million miles. Depending on the alignment of the planets, that's somewhere between 3 minutes and 22 minutes just for light to travel one way! This means that no event on earth can, according to the laws of physics, effect anyone or anything on mars for at least 3 to 22 minutes. The reverse is also true.

Well that's preposterous!! you might say. What kind of bullshit Elon Musk kool-aid are you drinking anyway? Mars is just an abstract plaything for NASA, and the subject of mediocre movies featuring Liev Schreiber. What the hell does that have to do with reality?

Here's the fun part: The same laws of physics apply to LA and Chicago, albeit on a much smaller scale. They may be only two thousand miles apart, but the fact remains, nothing can travel faster than 671 million miles per hour. Nothing; not light, not heat, not gravity, and dammed sure not GolfCo memos. The fastest ANY information can ever make that trip is 10 milliseconds.

Well shit son, 10 milliseconds? Isn't that faster than the blink of an eye?
Yup, the average eye blink is at least 100 milliseconds, so who cares?

Well, scenarios such as the GolfCo memos happen all the time in computing, but on a vastly smaller timescale. Sometimes it's on the order of picoseconds (that is, trillionths of a second.) In a network, two programs on different computers (technically even on the same computer) whether across the country, across the room, or even inches apart can disagree on the order of two or more events.

Using clocks to determine their sequence seems like a fine idea at first, but it breaks down very fast when the Carol program and the Ann program are sending their memos a few billionths of a second apart. It's bad enough when one of the clocks is "wrong", but worse still, it's physically impossible for the clocks to be "right" when they're further apart in distance than the two events are in time. (They can be said to be outside each other's light cones)

Ahhh, but why wouldn't Carol and Ann simply use a common memo numbering system? Then it'd be Memo #1 and Memo #2. Easy to tell which was the most recent!

Not so fast, this is where our nemesis linearizability comes to bite us:

If the company uses a single sequential series of memo numbers, then neither Ann, nor Carol, nor anybody else can send out a memo, without first checking with everybody who is authorized to send memos to make sure that the next memo number hasn't been taken. If the phones are down for an 20 minutes at the Chicago office, then all business has to stop, because the LA office can't be sure they aren't duplicating a memo number. If memo numbers could be duplicated, then the stores couldn't be sure of what order they were sent, or if they had consulted each other in the decision-making process for the instructions handed down.

This is Brewer's (CAP) theorem. It's actually mathematically, provably impossible to determine a total order of events among disconnected/distant systems without waiting. You have to either wait to coordinate, or settle for a partial order of events. No exceptions.

Just about every relational database in common use today uses a linearizable, "CAP vulnerable" approach in the interest of using simplified, sequential record numbers. The problem is: when the phones go down in Chicago, business in both offices must screech to a halt. That's bad for the bottom line. It's also a pain in the ass to have to call Chicago so often. They have better things to do with their time.

On the other hand, causal references, such as: "Memo A1 supersedes C1" CAN allow GolfCo to continue to do business when the phones are down in Chicago. The stores can tell at a glance if Carol knew about Ann's memo, or not. (Also known as a partial order: C1 before A1)

So, the moral of the story is:

  • Don't use clocks, they're wrong.
    "A person with a watch knows what time it is. A person with two watches is never sure"
  • Don't use total orders, they're limiting
    (Unless you're very patient)
  • Use causal references, they're compatible with the laws of physics!

Whatever you do, don't make Dave look like an asshole.

Ripping off the band-aid - My RICON CFP Submission

Well, I did it. I've applied to speak at RICON2015 ( )

I'm pretty new to the whole public speaking thing, so I figure it's a bit of a long shot, but apparently I like the sound of my own voice, and you can't succeed if you don't try!

Here's the synopsis:
(Feedback / overripe fruit welcomed)

Reality-based computing

All computing requires data persistence to achieve useful work. Data persistence systems employing eventual consistency with guarantees is believed to be the key to transcending CAP, but doing so on the server alone is a half-measure. In order to transcend the limitations of linearizability and create the next generation data-persistence paradigm, we must explore and evaluate certain foundational beliefs in order to expose deeper truths.

  • The notion that there exists a single temporal frame reference across the whole of earth is endemic, and false. Though it may be close enough for human purposes, the reality is that Tokyo experiences a subtly different timeline than does New York. Assuming your data traveled through the center of the earth, these cities are 40ms apart, but that's not just travel time, this is relativity in action; This is concurrency in the truest sense. These cities experience reality itself in different temporal frames of reference, 40ms askew.
  • The notion that "Client" and "Server" are fundamentally different types of participants is endemic, and false. In reality, the difference is only one of policy and consistency model; A client is only a less privileged participant in the persistence scheme, which may or may not be permitted to implement business logic.
  • The notion that a "Database" is the bounding box of a consistency model is endemic, and false. In reality, A resultset is not just a resultset; It is a fractional replica with a poor consistency model. There is no conceptual difference between a node on the other side of the planet, versus a lower tier of CPU caching; The data consistency model permeates all participants, all the way to the L1 cache of each and every CPU; From big iron, to web browser, to wrist watch; they are all the same.

Unbalanced - Why your load balancer hates you

Your centralized load balancer secretly hates you. You just haven't realized it yet.

(Another in my 947 part series on why anything centralized inherently sucks. I barely even like central air)

Lets rip off the band aid: It might work well enough for brochureware, but for web-based applications, centralized load balancing is just a dumb idea. It started out as a hack to make a bunch of servers look like one server; a workaround for the fact that HTTP and the web wasn't designed for modern applications. It's an unnecessary middleman and a bottleneck. It forces you toward certain strictures that are not suitable for the modern web. It doesn't like you, and it doesn't like your friends.

HAProxy load balancing sucks

  • Because it ties you to one IP address / data center. (Do you really want to deal with setting up anycast?)
  • Because monitoring bogged/failing hosts has an unacceptable delay. Your users will get a hung/refused connection for at least several seconds before the monitoring catches it. Your client has fresher health data than your load balancer.
  • Because it's a single point of failure. I've had more load balancers fail than I have fingers - That's at least eleven times too many

DNS round robin sucks ( ok, actually it's HTTP 1.1 that sucks )

  • Because some browsers will only try one IP address from the group.
  • Some browsers will try another IP after 30 seconds of your users sitting there wondering why they hired you. Browser authors are squeamish about DOS'ing your servers with concurrent connection attempts.

Active DNS sucks even harder

  • Because their monitoring assumes that your clients have a route to the server just because they do (and the inverse)
  • Because some ISPs ignore DNS TTL values
  • Because 30 seconds is way too long to wait for a good connection
  • Because DYN is a horrible company


If you're schlepping adult toys or hotel reservations, then sure, you could get away with a regular, old fashioned centralized load balancer; BUT, if you're doing something serious, like gaming, automation, or that you otherwise give a carp about your customer, you might start to realize that 30 seconds is wayyy too long to wait if you're having an issue with server X. 

The solution is simple: Push the load balancer to the client

The fix is stunningly simple, in principle at least. Load the basic app via your CDN or whatever, and include a list of servers. Use websockets (or ajax, if you must) and connect to at least two of them simultaneously. You can make the client as gentle, or as aggressive as you like. Want to take a server offline? go for it. Design your client to simply try another one. If raw performance is desired, connect to multiple servers and measure RTT between them. Server A is responding within 30ms, server B takes 300ms, therefore, route requests to server A, use B as a standby, or maybe give server C a poke.

For my use case, we use an exponential backoff. The app could have 10 or more connection attempts going at a time if necessary, and that's ok. If your servers are geographically distributed, and you are contending with backbone cut or other routing problem, it'll automatically find a node it can reach. Try that with a centralized load balancer.

The moral of the story

Cut out the middle man. Load balancers in the middle are not your friend. They're a liability. Push the load balancing and failover to the client. Your users will be much happier, and you'll get a lot fewer calls at 4am.

Deus meus sine genitalibus

What is consciousness?

"Je pense, donc Nietzsche peut le sucer."

"Je pense, donc Nietzsche peut le sucer."

Merriam Webster defines consciousness as:
The quality or state of being aware, especially of something within oneself.

In 1644, French philosopher René Descartes proffered, via his Principia philosophiae:
Cogito ergo sum — I think, therefore I am. 

Sure, later philosophers beefed over what was doing the thinking, who was doing the existing, and what the hell an "I" is anyway... but all that aside, the general idea is that thinking of any sort, kinda requires that you exist first.

"Holy shit" you might say, "Who the hell ever doubted that I existed in the first place? certainly not me! I'm totally secure with my existential preference."

Well Dorothy, have you considered the possibility that perhaps your entire worldview could be a sham? Maybe, just maybe, Kansas is bullshit, and you were just a figment of Mr Baum's imagination. Ok, sure its preposterous at first blush... but humor me for a moment; ask yourself the following question: ( It's ok, I'll wait )

"Do I exist? Or is this all just a dream?"

Now, according to Descartes and his pals: The fact that you're able to conceptualize that question at all means that yes, you probably exist. One could argue that the fact that you bothered to doubt your own existence even for a split-second, would imply fairly strongly that you're also self-aware in at least some limited fashion; That you have an ego, that you probably recognize yourself in the mirror, and generally think of yourself as being discretely separate from the rest of the universe.

"Ok, so we've established that I exist, and I'm self aware... Now what does that have to do with the price of garlic in Spain?"

Well, why do you even have a self to be aware of? What makes your self separate and distinct from the self of your mailman, your accountant, or your taxidermist?

Surely I, as a human, am one discrete being, Right? I have no conjoined twins to speak of, and my fleshy capsule (AKA meat-sack) moves about in the environment as it pleases. Given that this meat sack inexplicably has the wherewithal to recognize, and conceptualize it's own existence, it would seem fair under our working belief system, to say we can assign it one "self".

How about other meat sacks? do they have selves?

Imagine you could find some way to translate this and ask a dog the same question. Does a dog have a concept of a self? Who the hell knows really. It gets much less clear though, that's for certain. A dog is probably conscious; he recognizes when you call his name, and he poops on your favorite rug when he's lonely... but it's really kind of hard to tell from an outside perspective if he's truly self-aware. Maybe.

"Hey there sexy, how'd you get on the other side of this here glass?"

"Hey there sexy, how'd you get on the other side of this here glass?"

Ok wise-guy, how about a goldfish? Well, It's even harder still to discern whether a goldfish is aware of it's own existence... Though this could have something to do with the fact that Mr Goldfish (Carassius auratus auratusis a lot stupider than Canis domesticus.

All of these things are meat sacks, all the way down, including (drumroll please) all of your own cells!

Is a human cell conscious of it's own existence? Can it think?

Hmm... What is thinking anyway? Well, it does respond to stimulus. It regulates it's internal structure and chemistry according to an elaborate molecular program. It eats, it excretes, it generate proteins and enzymes. It gives birth to baby cells; It can at times defend itself and its brethren from attack. So, does it think? Is it self-aware?

One thing is for sure: Your cells are dammed sure not aware that you have a self, and that you have thoughts, because you are on such a different scale. Of course, your cells are the things actually processing your thoughts, yet they cannot conceive of the whole picture, because a single thought is so much bigger than one cell. You are their their world, and unbeknownst to them individually, they are collaborating on a massive scale to make you exist; to make you think. They are on you, and in you ( as well as a host of other cooperative, yet "non-human" microbes. ) There is no part of your body which is not comprised of, or manufactured by your cells. 

You came to be by a slow process of collective learning. A long time ago, by mutation or whatever, one bacterium to got together with another, and learned to cooperate. Over the course of billions of years, a swarm of cells learned collectively to become who you are today. By any reasonable definition, A human is a sack of cells. This human is arguably a meta-entity comprised of a collection of other entities.

Well, there's been a pretty good amount of research that would seem to indicate that humans also learn collectively. This is evidenced quite plainly by the fact that the sum of documented human knowledge is so much more than could ever fit in one person's head. Each generation, ( and with shocking rapidity ) we are able to build on the knowledge of our fore-bearers. How else would it be possible for you to be reading this configuration of glowing dots on your screen? 

It is therefore arguable that humankind is a sort of meta-consciousness, much like your brain is a meta-consciousness built up of billions of thought-fragments within each of your neurons.

So, lets take a bit of a leap into the void, and ask:

Does this pattern recurse on a larger scale? Is the universe simply a meta-consciousness made up of the innumerable machinations of you and me, and our cells, and the orbits of innumerable suns, galaxies, and black holes? If the workings of a single cell are dictated by the sum of it's sub-cellular programs, the workings of a human dictated by the the workings of it's cells, the workings of a society dictated by the workings of it's humans, could it not be plausible that this pattern is bigger still than we can currently comprehend?

When we each "die", it could more accurately be said, that our constituent components are broken down into their constituent components, and so on... and the cycle begins anew. Those raw materials are converted into plants, fish, other people, and ultimately into the light of the stars from whence they came.

GOD - A Funny Meta-consciousness

As a society, we tend to think of an anthropomorphized god, if not consciously, then at least subconsciously. Even though a gender-specific pronoun could be dismissed solely as a linguistic device (and many still do ascribe the label "He") -- The attachment of any pronoun is an indicator of personification. Even the most dogmatic theologians would agree that god is not a person. God is love, God is light, God is everything, right?

I would posit for you, my dear metahuman-co-constituent, that an anthropomorphized (capital G) "God" is nothing more than a made up persona; created by man in the interest of political expedience.  Whereas the non-anthropomorphized "god" is a word for the stunning machinations of a system in which we all participate; a system that is more beautiful than you could possibly ever imagine. "god" does not judge you any more than you would judge one of your own cells. "god" doesn't have an ego, and it doesn't have a penis either (or humans clamoring over it for political power.)


NuoDB: Very good at technologying, Not so good at communitying

Lately, I've been very excited about a new commercial database offering called NuoDB. NuoDB takes a fresh approach to a very old problem, and solves it spectacularly well I think, especially as pertains to web-scale systems.

The NuoDB company clearly has it's head on straight with regard to the technology; it's a brilliant and scalable approach. Alas, the same cannot be said for their support community. 

In the strongest possible sense, I would hope that NuoDB management understands that in order to succeed, NuoDB must create a vibrant community in the spirit of openness and honesty. Absent this, they will certainly not get my business, and I suspect the same may be true for others.

Below is a post which I had attempted to submit to the NuoDB forums, but which has thus far been rejected or ignored by moderators. Suffice to say, this kind of moderation is a sure fire way to kill the community, and my interest in the product as well.

Greetings NuoDBicans,

I'm facing an interesting architectural challenge, and I'm in the process of evaluating NuoDB as possible part of the solution.
Perhaps someone might be able to point me in the right direction, as I'm a bit of a Nuo-Noob.

Our application requires that we propagate a high volume of change data out to various UIs and other third party systems. These items are time sensitive, and as such, we have opted to use a reactive programing approach. Our application requires ACID and a high level of redundancy/reliability, so we very much like NuoDB as a migration path away from the management nightmare that comes with a Mysql system-of-record approach.

At present, we are effectively using a single large Mysql master for our persistence layer, and RabbitMQ Pub/Sub for our messaging layer.
Every time we commit to Mysql, we immediately publish messages to RabbitMQ detailing what was changed. The various consumers then propagate these changes in a reactive fashion to other systems. This generally provides the high degree of performance that we require (sub-500ms in many cases.) We can tolerate higher latencies occasionally, but it must be fast and efficient overall otherwise we will encounter significant message queue backlogs, and response time will suffer.

One challenge with this approach, is that it's possible for the Mysql commit to succeed, and the RabbitMQ publish to fail. This is no good.
One possible alternative approach would be to have an agent consume the Mysql replication stream, grok any useful change data, and feed that into the pub/sub messaging system. Technically there are a few challenges here too, as we're interested in publishing semantic change data (like the application user_id who made the change,) rather than the somewhat-lower-level DB record edits.

So, I've read the white papers, and consumed as many NuoDB-under-the-hood type articles as I can, but I must confess that I'm a bit stumped here. I don't mind maintaining an out-of-band message passing layer if necessary, but I'm having a tough time conceiving of a way to ensure that the message doesn't arrive before the data shows up in the NuoDB transaction engine on the other end. What I guess I'm really asking here, is if there's a way to perform basic message passing [I]through[/I] NuoDB, in such a way that the messages are correlated to each commit. This is important to us because we generally have to read multiple different tables to render the final representation of the data to be transmitted to each 3rd party. It would therefore be a bad thing if, in response to receiving said message when we looked up the supplemental data in NuoDB, it hadn't shown up yet.

Without a doubt, we could simply use a big blobby table as the initial part of the message passing system, and use a single polling process to perform the fanout. We'd insert a large blob of JSON or Protobuf containing the aforementioned semantic edit data just before each commit. We could also employ triggers to do something similar... the issue is that we wish very much to avoid having to poll for new messages on the receiving end. If we went with this approach, we'd have to poll the table many times per second. Certainly, this could work, but aside from the very serious yuck-factor of high-frequency polling, it could also break quite miserably in the the face of high traffic.

Is there at present, or could there ever be a way to efficiently pass messages to a very simple consumer (presumably per each transaction engine) in such a way that it was guaranteed to be consistent with the transaction when you're dealing with more than one transaction engine? Might this sort if thing be possible/buried within the guts of NuoDB?

Cheers, and Happy holidays!

I was blogging things *before* they were relevant.

"First Post!" he says, mouthing the words slowly into the mirror. He takes sidelong glances at himself from different angles, to ensure the optimal duck-face.

I can see it now: It's the year 2093. You are standing in the holoarchive of the San Angeles municipal hypermuseum. It's a tremendous place. An exhibit presents a new look into early 21st century life. An image hangs in midair - "whoa, two-dimensional" you think to yourself. Thankfully, it has a much more readable mental-transceiver-compatible caption:

This grainy instagram, captured in 2013 Southern California shows two hipsters on their way to the apple store. Historians speculate that rather than neurocortically-enhanced-powdered-food-ration-injectors (NEPFRIs) like you and I, they were powered by some mysterious beverage called "Pabst Blue Ribbon" - It appears to give them the energy to live in a constant state of irony, thus making them invulnerable to criticism. Much research remains to be done about this dark time in our history.

This grainy instagram, captured in 2013 Southern California shows two hipsters on their way to the apple store. Historians speculate that rather than neurocortically-enhanced-powdered-food-ration-injectors (NEPFRIs) like you and I, they were powered by some mysterious beverage called "Pabst Blue Ribbon" - It appears to give them the energy to live in a constant state of irony, thus making them invulnerable to criticism. Much research remains to be done about this dark time in our history.

I suppose this site is largely intended to be a journal of sorts; created in the interest of interacting with other like-minded heuu-mons. That said, it would be self-deceptive to think that blogging wasn't at least a little bit narcissistic. Admission is the first step, right?

Now! Streaming live for 26 hours a day! Wolf Blitzer's credulity, and Anderson Cooper's lovely silver hair - Only on CNN

 So, stay tuned for mundane rantings, peculiar observations, and accounts of generally questionable social interaction. As our modern society swirls slowly round the drain of vapidity, corporatism, and self-promotion, I'd like to offer you a nice cool drink of refreshing oddity; lets just hope it's not Pabst.