Lies, Damn Lies, And Now Statistics

We've been at the game at least since Democritus, and god knows I've made it a pet subject on the blog over the years. At what point, we wonder, does a pile of stuff instead become collection of tiny things? When is it appropriate to break something up and consider it in terms of little constituent parts, and when are the small-parts contributions better thought of as a collective whole? Once you get past the philosophical wankery (and let's face it, I'm unlikely to), then I suppose there are really only two practical answers to that: when you can't avoid the quantized nature of things, and when the math is easier. I think it's a little bit funny that a few centuries of science based on analog mathematics (with continuous, nicely differentiable functions) finally coughed up everything digital (breaking it up into approximate chunks), and I have often found it fascinating how the same systems can be usefully described as discrete, continuous, and then discrete again, depending on what scale you dial in at, or what you're trying to prove. Electronics, for example: you start with quanta (electrons), which average out to make analog structures (semiconductor devices, let's say), and then put those together to make a logic network that'll do it all bitwise, allowing only ones or zeroes (I'm using one to write). And sometimes your semiconductor theory gives you localized states to deal with; sometimes the analog nature of a transistor or diode is important. I think one reason that things like macroeconomics and evolution appeal to me is they're large-scale ensemble effects that are logical extensions of (well, evolution is anyway), but seemingly independent phenomena from, the things that make them up, which in those cases are our very lives.

Maybe you'll forgive me for dipping into this well yet again. I had to sit through a weeklong industrial statistics class earlier this month, and this is the sort of thing that I was daydreaming about (well, once I got tired of thinking up wiseass comments and imagining people naked). It was an effort to fuzz over the whole mind-crushing boredom of it all.

Most people loathe stats for the terminally dull math it throws at you, and that's a reputation that's probably deserved, but at least digging through the justifications and proofs has a way of adding a kind of legitimacy of knowledge. Getting through it makes you feel like a smart person. That's not the class I took the other week. There we were training with a computer program to run through all the equations behind the scenes--elegantly enough if you stick to the problems it was designed for (but what kind of engineer would I be if I did that?)--and the practical application got taught without drumming up even the mathematical gravitas you'd need to count back change. It's a well-oiled teaching method that got across how to use a mathematical tool without an underlying idea of how the math might work, and okay, knowing how to use it is the take-home you'd want even if you did take the time to watch the gears turning, and he did a good job of getting across what he tried to. But it's a special kind of tedium to spend a 40 hour week absorbing the knowledgable huckster routine from someone you're pretty sure isn't as smart as you. Christ, it reminded me of those long ago nights of sitting through driver's ed.

(Full disclosure: I had a stats class back in college that taught nothing of perceivable relevance whatsoever. It taught some math, but I didn't learn any of that either, or at least none of it stuck in my head beyond the final. I didn't feel the least bit smart, but still got an A. Not sure how that happened.)

Anyway, the dorky daydreams. It struck me that when you hit that border between chunky and creamy, where you can't really decide whether to count things up or do clean math on some variable, then that is exactly where you have "statistics." Implying a distribution function is exactly the point when you know damn well the data consists of tallied events but you're going to call it a smooth curve anyway, and statistical analysis is supposed to be what tells you if that's worth doing and how legal it really is, when things go one way or the other. In the manufacturing world, one primary concern is sampling and measurement: it's an important question whether you can compare results from measurements that will vary, that is, whether the data are really telling you anything. We're all used to thinking like this, but most scientists I've known aren't terribly rigorous about considering error in the experimentation and data-gathering, although then again, we are usually more about understanding relationships that come from somewhere.  More curve fitting, fewer t-tests.

Statistical understanding gets buried under a lot of science and engineering anyway, without always thinking about it as such.  One advantage of spending a decade and a half as a technical whore is that I got exposed to a variety of interesting fields and thinking (a disadvantage is that I got to be a whole lot better at bullshitting my way around ideas than studying and implementing them).  The very basis of band theory, which is used liberally to design solid state devices, is a smooth approximation of entities that are known to be discrete, assigning effective properties, imagining a continuous density of electrical states, generating a smooth probability function to populate them.  When you can't quite get away things so easily, when you have to admit you're counting electrons or photons, or doing signal processing in general, then you have to fall back to the statistics.  It's interesting to consider how shot noise will plague you in low-signal collection (when just a few electrons are passing through, they are less likely to be representative), or derivations of signal to noise on a larger scale, and any kind of diagnostics will also require a statistically-derived decision based on the quantity of signal.  I spent a few inadequate efforts in past years thinking about the implications of size distributions of small particles, and I'm getting lately into something like that again.  It's a case when it's not just the size of the the little guys govern properties, but the shape of the distribution will affect what you measure too, and if spread out, it'll behave much differently than if they're all the same size.  You might call this a property of your sample, or you might call it the properties of differently-behaving individuals.

But it's important to remember that science and statistics aren't really the same thing. One remarkable part of that course (and, I think this is the hollow heart of certain fields which rely on statistics, some of which happen to control the world), is that you can sometimes get to feel you are evaluating things astutely, while knowing fuck-all about what's going on down there. I think that makes certain kinds of people feel very smart indeed, but it scares the shit out of me. When you can study something in detail while remaining relatively ignorant of it, you have a good opportunity to lie to yourself and others.

I'll leave the economist-bashing aside today and note that as researchers, if we're chasing something like the scientific method, then we have some working assumptions and models going in. We have some prior experience, sometimes whole fields of it, of how things tend to relate, might relate, or fucking well better relate. One of the most annoying things that got pushed in the class, and I know is used in industrial research, is the development of "models" though statistical design of experiments.  The idea of that is to throws a bunch of ingredients together in a way to best infer dependencies, which is a neat scientific tool, and sometimes exactly the right one, but the problem is that it also offers no real understanding.  It is meant to address the what, but utterly leaves off the why.  [I feel better about things like evolutionary algorithms, where a solution is chased down through randomly mutated generations, and maybe you don't know the intimate details inside there either, but it's a really clever approach at that higher level of granularity]  If it's formulation work you're doing, then you end up doing chemistry with a completely optional understanding of, well, chemistry, and this just annoys me on some level. You really should have some fundamental understanding of how materials are known to interact. The instructor called these sorts of insights, a little dismissively, as "local knowledge," but if it's science, the local knowledge is what you are getting at.

Initiation

Not long ago, I found an old letter to the editor that I wrote, and (thank god) never sent, from back in my college days, defending that redoubt of callousness and misogyny, the greek fraternity. The argument wasn't that godawful--amounting to "it's different for a bunch of socially inept nerds in engineering school, we need to drink irresponsibly, waah!"--but I know the style, the sort of precocious, self-affirming, fake-earnest crap that I'd later associate with bullshit-reasoned libertoonian writing, and I'm embarrassed that it ever came out of me. I have strong thoughts about going back and burning it, but on the other hand, it's my moral duty to belittle this kind of thing, maybe even more so now that I've gone and caught myself doing it, twenty years ago. In my limited defense, my conscience was making noises even in that one about people who might live beyond a bubble of other straight middle-class white dudes, and, generally speaking, I tried to talk to people sometimes at that age, cultivate some empathy, read freely, and accept that I might be wrong, and some would say that I generally grew the fuck up over time. Also in my defense, even if the fraternity system in general can be a hiding spot for a whole lot of college-guy nastiness, I think it was much better in my microcosm, and these sorts of things make a more easily assailable target than, say, college sports. The basic thing is that if you get a collection of pigs in one place, then they'll act like a bunch of pigs. A bunch of drunken geeks like we were? Some were better people than others, but I mostly spent those days sitting around with my friends making sarcastic jokes at their expense and mine. Good times.

People have made a lot of assumptions about this over the years, and one that's galled (because it's true), is that an organization like this is pretty inauthentic at its core. The design of a fraternity isn't really to accomplish anything other than pulling in students before it knows them especially well, and jam them into a sort of friend-making boot camp. I can't come down on whether this is fundamentally a bad thing or not. Some of them became real friends and some didn't, and if an opportunity comes your way to get to know people you have a good chance of getting along with, then why not take it? Especially if you're a dork in engineering school. (And there's the 2012 version of the argument.)

The trick to indoctrinate new guys is to build experiences together, make them hang out with each other and also with the people in the group. That was most of the hazing we gave, and the purpose of all of it, and if I had to show up and fetch beers for a month, or if I got encouraged to try and get away with things that were funny and harmless, then hell, it was nice when it was my turn to drink and laugh. It's pretty obviously a knockoff of military initiation, or gangs, or the clergy, or high school, or any group that makes a case for its bureaucratic existence rather than just its members. And the format of the rite is similar. Set a group of people apart from the rest, push them into a shared experience that's hard to understand in any other context, and let them join fully when they start to bond. It's team-building, in the horrible corporate sense. If my fraternity had conducted this with anything but mock-seriousness, then I'd have been deeply offended, but when it comes to human organizations, mockery is the exactly appropriate response to an abundance of seriousness. (And this may be how they got to me, the sneaky bastards.)

It's an adage that as empires crumble, life tends to go on for most of us--meet the new boss and all that. I hate to take that one too far: it's terrifying how much they can take down with them (and have taken down), but there's truth too that the failure of power structures most seriously threatens the empowered. [It's a thought that often pops up with me whenever the "ready flow of credit" is allegedly held at gunpoint by the financial establishment, but of course it applies generally to straight middle-class white dudes too.] I don't want to beat on Debt much more than I already have, but I find the historical contention interesting that when externally imposed institutions or governments lost their local grip, as he says they did for much of the middle ages in much of the world, then civilization tended to replace them with persistent hierarchical arrangements, where people were guided by tradition into castes, instead of by institutions. The roles in that arrangment are passed on by collective habit, and over time become difficult to confront. Inequality is assumed instead of coerced, and you learn it at birth. I'm not convinced this makes a better world, and in fact, it unnerves me that this is the only alternative that's much panned out on a civilizational scale, but the argument was that it was somewhat less violent.

You can't really escape it, and it's a topic that I periodically wander back to, maybe moreso than your typical engineering nerd. We are initiated into the game before we even get a chance to question it. The fact that we share non-genetic information from one generation to another, one group to another, that we pass it on, is the very essence of what makes us human. We're pulled into existing conversations before we even know how to talk. To really re-think things, you'd need to make a clean and thorough break from ten thousand years of evidence of other people existing, and who'd get to be the architect of that experiment anyway? In the real world, if there is a better and more fair way to organize humanity, it will be because a tradition of such practice emerges slowly, and we will still have to initiate people to get up to speed with that improved starter packet of information. Even if the guiding idea is a rejection of system, you still have to initiate people into the tradition of challenging it. You have to be conditioned to reject authority. Is it any wonder this species is so fundamentally confused?

Review: Vortex, by Robert Charles Wilson

[Probably contains some spoilers.]

Vortex is another loosely-tied sequel to the big-sweeping science fiction novels Spin and Axis by Robert Charles Wilson, and much like before, I picked this one up while pretty desperate for a read, and Wilson is a reliable satisfaction. You can follow back to those other reviews, but the short version is that I'd found Spin to strike a near-perfect balance of big, cool sci-fi mystery and sensitive, humanist storytelling, while Axis, although well-conceived, somehow managed to lose a lot of urgency. With Vortex, I'm on the train again, and I made the uncharacteristic move or reading the whole damn thing over the space of a day or two. The novels can't be called a series but they're connected. Although they share a broad plot arc of a deepening puzzle on a celestial scale, each one resolves neatly enough on its own terms. (The titles themselves don't signify a heck of a lot.) It's more as though Wilson writes each one in the same universe, more or less, and enough time passes in between both the events and the publication that the remembered details fuzz up. I think there might be some subtle contradictions in the descriptions of the connected worlds and their makers among the books, even if the general idea remains the same.

And the general idea, I will reiterate, is really well put together for a science fiction conceit. Human society has been visited by some unknown and gigantic technological presence, one that (in Spin) isolates the planet, holding it in a temporal bubble as the universe ages around it. Wilson wisely leaves the how as perplexing as the why. At the end of the first novel, earth finds itself linked to other livable worlds through huge archways, and with Axis, we learn that the makers have some use in sampling and preserving (and swapping at long intervals in time through another series of arches) the information that these oddly fostered societies tend to generate. The "hypotheticals" themselves are an enormous and slow-growing network of tiny self-replicating machines, the sort of devices that short-lived biological civilizations tend to generate at some point in their development, and the von Neumann devices of hundreds or thousands of such worlds have, over impossible eons, grown, merged, and evolved into a system that is symbiotic with the more frenetic societies that rise and create and fall. What's really brilliant about this, and I mentioned it before, is how Wilson imparts on them the function an impersonal and mighty god (and related subplots ensue, not unsympathetically). He gives us a watchmaker that's real, and it's as insensate as an ecology, and we are left to learn, again, what it is to be alone(-ish) and also to be Chosen in a universe that's more complex and wonderful and humane than anyone would have ever guessed. Vortex eventually sees life, the universe, and everything right through to the end, shows us an earth scoured, inadvertantly, by humanity's own efforts, to the stars winking out, to expansion and cooling of space right on through to its impending heat death. And he still gives us to care about the people in it. It's why I love his writing.

Vortex tells two stories simultaneously. One is a mystery set amid interesting times in 21st century Texas, where a state psychiatrist and a cop try to understand why a strange boy is writing unsettling journals well above his intelligence level, narratives that are loaded with information that, despite the gonzo setting, contain elements that relate to contemporary crime activity. It's a frame that is just compelling enough to keep the story moving and the tension up, characters just compelling enough to give a damn about, and which lets things get revealed at just the right pace.

The other half is contained in the journals themselves. It begins on the other side of the temporal gate that erupted at the end of Axis, that also absorbed Turk Findley (among others), who wakes up ten thousand years later to a world that's unrecognizable. Wilson leaves the other holdover character, Isaac Dvali, the child prodigy who's been loaded up with human attempts to interact with the hypotheticals off until near the end. Turk is one of two cobbled-together ambassadors to the reader, and they're both of them compelling ones. (And if Turk is not quite the same Turk in the last novel, and if maybe the problem with the character all along was that he couldn't be any more random, then Wilson gives himself a fine excuse for all of this.) They are relateable person in a new community that's integrated beyond current capabilities, and literally adrift in the next-door world (that's dying), floating on its way to the arch to get to earth (which is dead). Our almost-mundanes are in a good position to observe how all the bizarre new technology doesn't prevent human pettiness, zealotry, visciousness, love or decency. The 31st century technology does eventually offer the characters some tools to confront, and, since there is really nothing there to talk to, finally start to use the hypotheticals as a medium to bring all the threads together.

It takes Isaac to do this, to connect it all, and while he have been strange enough to leave out of most of the story, it's his simple and strong motivations that give us, the readers, the story at all, revised, just a bit, to allow a moment or two of grace.

[Sorry about the continued sparse posting, by the way. It's been a sinister combination of time and motivation. I have some other stuff I want to get to at some point or other.]