The Null Hypodermic

Shot Selection and the Secretary Problem

2012-01-27T17:33:00.000-08:00

One of my favorite problems in all of recreational mathematics is the so-called secretary problem. In this problem, you are interviewing a hundred candidates for a secretarial position. For the purposes of discussion, we'll assume that the various candidates have a precise suitability rating, and of course, you want to maximize this rating for your hire. Ideally, then, you'd interview all hundred candidates first, get their ratings, and then hire the best one.

Unfortunately, that's not the way things work in this problem. You only get the candidates one at a time, and you have to decide then and there whether or not to hire them or not. Once you've rejected a candidate, they're lost to you forever. One could, theoretically, lose the best candidate on the very first interview.

The question then is, what is your best strategy, and what is your probability of making the best possible hire using that strategy?

(By the way, if you think this problem is formulated in a politically incorrect way, I first encountered it in a form called the sultan's dowry, in which a suitor for the sultan's daughters had to select the one with the largest dowry. If he picked the right one, he got to marry her, but if he didn't—well, let's just say an unsuccessful suitor and his head are soon parted. But it wasn't entirely unproductive; I eventually formulated a variation called the iterated sultan's dowry, in which a second suitor, seeing the first suitor's unsuccessful head roll down the hill, gets to use the information in choosing a prospective mate, and then the third, the fourth, etc. This variation has an interesting solution which is unfortunately too large to fit into this parenthetical comment.)

It can be shown, fairly easily, that the best strategy must be of the form "Skip the first n candidates, recording their suitability ratings. Then choose the next candidate whose rating exceeds theirs." The reason is that as you plow through the candidates, the probability that the best one is yet to come never increases, whereas the probability that you've already encountered the best one never decreases. So the question reduces to figuring out what the right choice for n is.

Ultimately, following a strategy like this, you could end up choosing no candidate at all if the best candidate is already in the first n, since you've already skipped all of those. But if you do pick a candidate, it will be number k > n.

For that one to be the best overall, the best must be in the last 100-n. Furthermore, the second best of the first k (that is, the best before the ultimate choice) must belong in the first n. Now, let's work out the probability that both of these happen. In order to do this, we have to break down the possibilities into all the different cases.

The first case is that the best candidate is the very next one—candidate number n+1—which happens with probability 1/100. You'll choose that one provided that there's no other candidate between the first n and number n+1 that is better than the first n. Since there are no candidates in between, that probability is 1. So the incremental probability for this case is 1/100 times 1, or just 1/100.

The second case is that the best candidate is the one after that—candidate number n+2—which again happens with probability 1/100. You'll choose that one provided that there's no candidate between the first n and number n+2 that is better than the first one. That will be true provided the best of the first n+1 happens within the first n, so the incremental probability for this case is 1/100 times n/(n+1).

Following the same line of reasoning, the third case—that the best candidate is number n+3—provides an incremental probability of 1/100 times n/(n+2), the fourth case provides an incremental probability of 1/100 times n/(n+3), etc., until the last case—that the best candidate is number 100—provides an incremental probability of 1/100 times n/99.

Putting all these cases together, this strategy "wins" with probability

n/100 × [1/n + 1/(n+1) + 1/(n+2) + · · · + 1/99]

It can be shown, using relatively straightforward calculus, that this expression reaches a maximum when n = 37, and yields a probability of success of about 0.37.

That's not a coincidence, incidentally. For large candidate pools (and a hundred candidates qualify as a large pool), of size N, the best strategy is to skip the first n = N/e, where e = 2.71828+ is the base of the natural logarithm, and to take the earliest best candidate thereafter. The approximate probability of success (that is, choosing the very best candidate of them all) is very close to 1/e = 0.36787+.

For a lot of people (including myself), that's rather stunning. It implies that even if you have a million candidates, you have a strategy that picks the very best one of them with better than a one-in-three chance.

The reason I'm putting basketball in the mix is that there's a fairly straightforward application to a vital aspect of scoring: shot selection.

Consider: A possession in basketball lasts for anywhere from 0 to 24 seconds (neglecting offensive rebounds). You can't always guarantee that you'll make the shot; the next best thing (at least before the endgame) is to select the very best shot—that is, the shot that has the best probability of going in (neglecting fouls and three-point shots).

In other words, ahem, optimal shot selection.

But you don't always know when that best shot is going to come, especially when you're working out of a halfcourt set. Is it the very first one? Is it the next best one? Maybe the best one will come at least twenty seconds into the possession. You just don't know. But maybe, now, you have a rule of thumb for selecting that best shot. You skip the ones that come in the first 24/e = 9 seconds (approximately), and take the next best one that comes thereafter.

Obviously, this rule makes lots of assumptions, such as (a) the defense is equally tenacious across the entire possession, (b) the offense is equally productive of shot opportunities across the entire possession, (c) the best shot opportunity is equally likely to come at any time during the entire possession, etc. But to the limited extent that these assumptions are approximately valid, it's not a bad rule of thumb. It suggests that the Phoenix Suns of the early-to-mid-2000s were a bit hasty.

But not by much. Just a second or two.

Constructions with Compass, Straightedge, and Flatiron

2012-01-05T15:59:00.000-08:00

There may not be many of you out there who remember high-school geometry fondly, but those of you who do probably share my appreciation for compass-and-straightedge constructions. I (re-)started thinking about this today, when someone mentioned, jocularly, preparing for travel at Warp Cube Root of Two, and it occurred to me that this was one of the three classical things that one cannot do with compass and straightedge.

Those strangely intermediate folks who are at once unfamiliar with compass-and-straightedge constructions and yet not intimidated by their spectre may find the Wikipedia page a reasonable starting place. But what I'm proposing today is something different, which I'm a bit startled hasn't been discussed more prominently: three-dimensional constructions. I've had this in my virtual back pocket for a while; this seems as good an opportunity as any to pull it out for a looksee, and figure out if anyone else has encountered anything like this.

The idea here is to extend to three dimensions what ordinary compass-and-straightedge constructions do in two dimensions. The first thing is to define the tools and rules for their use. For instance, in two dimensions, the tools are a compass and straightedge (like a ruler, but with only one edge and no markings), and with them, one may:

Draw a line between any two distinct points.
Draw a circle with one point as the center, and any other point on its circumference.
Draw an arbitrary point on a line or a circle, or off it.
Draw the point at the intersection of two lines (if they intersect).
Draw the point (or two) at the intersection of two circles (if they intersect).
Draw the point (or two) at the intersection of a line and a circle (if they intersect).

That's it; that's all you're allowed to do. With these restrictions, the Greeks discovered that it is possible to construct an astonishing variety of geometrical objects, but they were unable to construct three famous examples. They were unable to construct the square root of pi (also known as squaring the circle); they were unable to trisect arbitrary angles (that is, construct an angle with one-third the extent of a given angle); and they were unable to construct the cube root of two (also known as doubling the cube).

It turns out that these are impossible, and can be proved to be so, using some notions from field theory. That has not, of course, stopped people from submitting reams upon reams of alleged constructions of one of these three objects, all of which (you may be assured) are somewhere bogus.

But enough of that for now. In three dimensions, the canvas is not a flat plane, as it is in two dimensions, but all of space. And we introduce a new tool, which I will call a flatiron, which permits you to draw planes. The flatiron rules are as follows; in addition to the above, one may:

Draw the unique plane containing any three non-collinear points.
Draw a sphere with one point as the center, and any other point on its surface.
Draw an arbitrary point on a plane or a sphere, or off it.
Draw the line at the intersection of two planes (if they intersect).
Draw the circle (or point) at the intersection of two spheres (if they intersect).
Draw the circle (or point) at the intersection of a plane and a sphere (if they intersect).
Draw the point (or two) at the intersection of a line or circle with a plane or sphere (if they intersect).

As an example of what one might do in a three-dimensional construction, consider the following fairly simple task: Given points P and Q, construct a regular tetrahedron with PQ as edge. We proceed as follows:

Draw spheres of radius PQ around both P and Q.
Draw the circle C at the intersection of spheres P and Q.
Draw R, an arbitrary point on circle C.
Draw a sphere of radius PR around R.
Draw S, one of the two points of intersection between circle C and sphere R. PQRS is then a regular tetrahedron.

It should be straightforward to see that one may construct a regular cube as well. Does this mean that one can construct the cube root of two in three dimensions? I suspect not, although I cannot yet prove this.

So, a couple of questions, one easy, and one not so easy:

Suppose that indeed, the cube root of two is not constructible in three dimensions. What about the fourth root of two? In which dimensions might that be constructible?
Is it possible to construct all five of the regular polyhedra? In addition to the tetrahedron and cube, these include the regular octahedron (eight faces), dodecahedron (twelve faces), and icosahedron (twenty faces).

It's True

2011-12-09T10:58:00.000-08:00

This man has no gut.

Could You Be a Crackpot?

2011-07-27T11:13:00.000-07:00

Of course not! That's why I invented the following barometer for identifying crackpots, so you'll see how much you're not a crackpot. This won't take long. Just read the following essay, then answer the question afterward.

You have always, from an early age, been fascinated by the workings of the universe. For longer than you can remember, the suspicion that something is amiss in the prevailing scientific theory has gnawed at you, but only recently, in your maturity, have you recognized how to rectify its flaws.

This is no patchwork repair—it is a fundamental revision in how to perceive the universe. You have discovered this shift by virtue of your stark insight, uncluttered by years of outdated academic instruction. You have been labelled a crackpot, but this has always been the mark of the true genius. Witness Galileo: He had to withstand the prejudices and superstitions of his time, even amongst his colleagues, in his pursuit of the truth! And you, like Galileo, are no crackpot.

Your breakthrough must be distributed personally, as part of a grass-roots campaign, in order to avoid the censorship that is part and parcel of the mainstream scientific press. But you have no doubt that you will succeed; you are brave enough to face the naysayers who have shouted down other talented scientists who lacked your resolve. Witness Einstein: His original mindset differed from the battered viewpoint that jealous criticism reduced him to! You can quote effectively from his writings to support this.

The prevailing scientific theory requires—in fact, relies crucially on—a body of mathematics that was clearly invented ad hoc: for the purpose of showing exactly that for which it was invented. This circular line of reasoning is a basic but crushing flaw. Your alternative, for those who have the vision to properly appreciate it, is conversely a work of great elegance and beauty, which resonates with rather than challenges your acute intuition.

When at last it is properly recognized, your breakthrough will enable great boons for humanity. You will be offered vast rewards for your work, but you will turn most of them down, asking only to continue your scientific investigations without the further distraction of interacting with the scientific community. Those who doubted you will permit this reluctantly, out of respect for your talent, but all the while will admire your work from afar.

OK, now that you've read that (haven't you?!), here is your one question:

On a scale of 0 to 10—0 indicating complete disagreement and 10 indicating complete agreement—how well does the foregoing essay represent your viewpoint?

Scoring:

10: You are a crackpot.
0-9: You are not a crackpot. True crackpots are in for a pound if they're in for a penny.

The Myth of Common Sense

2011-06-22T18:20:00.000-07:00

How many times have you heard someone say, "It's just common sense"? I just heard it myself (well, read it in an e-mail) two days ago, and it was in relation to something that I might argue wasn't really common sense at all—at least, not beforehand. With the benefit of hindsight, it became common sense. (If you're inordinately curious, it had to do with how hot a playground surface could get on a cool day. Pretty hot, as it turns out.)

Right now, as I write this in late June 2011, if you google "it's just common sense," you get the following things that are supposed to be common sense:

Domestic drilling for oil
Creation science (I think—the post wasn't entirely coherent)
The necessity of broad-ranging budget cuts
Wearing a bicycle helmet reduces the risk of injury
The use of backscatter scanners (so-called "naked X-rays")
Avoiding texting while driving
Essentially any Republican viewpoint on fiscal policy
Showing discretion on social networks
Allowing schoolteachers to bring guns to class (!)
Using alternative medicine

Those are just the first ten hits that made an assertion that something or some position was "common sense." I ignored any example that used the term in an ironic sense (most notably, "Repeal child labor laws? It's just common sense!"), or were talking directly about common sense.

Now, one thing I expected was that there would be a split between things that were asserted to be common sense in a descriptive way (that is, people do commonly agree on them, or would if they were asked), and those that were asserted to be common sense in a prescriptive way (that is, people should agree on them). And indeed there is, but the split was fairly unbalanced: I'd say that out of the ten hits I listed above, just three—the bicycle helmet one, texting while driving, and discretion on social networks—were even close to the descriptive sense, and the bicycle helmet one only alluded to common sense to set up the contrasting finding that apparently, it doesn't reduce the risk of injury. (Very interesting, by the way. But a post for another time.)

The remainder were all prescriptive; in general, they even conceded that a large segment of the population—be it liberals, non-religious people, gun-control advocates—were opposed to their viewpoint, but they then went on to say that these people were mistaken, and they were mistaken because they went against common sense. In most cases, they don't really explain why their viewpoints were common sense; it was enough to say simply that they were.

And that demonstrates the appeal of saying that something is common sense: It removes the burden of proof from the person making the assertion, and places it on anyone who disagrees with it. Essentially, it abdicates any responsibility for backing up your position. More than that, it demeans anyone who disagrees, as they obviously lack common sense (whatever that might be).

Granted, it's always been a bit hazy exactly who has the burden in any particular case. The negation of an assertion is, of course, another assertion, so who really has the burden of proof? A convenient rule of thumb is that anyone who goes against the conventional wisdom position (the descriptive common sense, basically) assumes that burden, but there are, I'm sure, plenty of exceptions to that. But I argue that in any borderline case, where there's some dispute as to who has the burden of proof, both sides should assume that burden.

So when someone writes that something is "just common sense," it almost always turns out (and I'm being as generous as I can here) that they don't exactly know why they hold their position. Or won't say. Or it's just too much trouble to actually work out and explain what their position is. To which I'd say, "Well, then, why are you wasting your time explaining your common-sense position?"

To its credit, the bicycle helmet post actually points this out. From Doug's Darkworld:

"It’s just common sense" is probably one of the most seductive and deadly false arguments out there. When someone says "it’s just common sense" what they are really saying is "reality conforms to my idea of what makes sense."

I would say there's other cases, but that is a big reason that people say something is common sense. It puts me in mind of a point made by Michael Shermer. Shermer's a skeptic of possibly the most compelling kind: a recovering occultist (an anti-skeptic, if you will). He wrote a book in 1997 entitled Why People Believe Weird Things; he updated it five years later, most significantly including a new chapter entitled "Why Smart People Believe Weird Things." In it, he argues the following thesis: Smart people believe weird things because they are skilled at defending beliefs they arrived at for non-smart reasons.

I can't emphasize strongly enough what a transforming revelation that was for me. For much of my life, I'd encounter people holding what (to me, at least) appeared to be some kooky position or another, and my reaction was nearly always something along the lines of "How can you possibly believe that?" And that was a rhetorical question; I wasn't really interested in how they came to believe that, I just wanted to point out that it was a nonsensical position. As you might expect, I eventually came to realize that most people didn't particularly take kindly to that sort of question, so I stopped saying it. But I still thought it.

Shermer's thesis, however, made me start asking that question again, but internally, and this time at face value: Why do they hold that position? It's very often not for the reason they espouse. (For instance, most of the common sense cases, I pointed out, are not in fact commonly held.) Maybe it's because of their own personal experience; people tend to overvalue personal experience. Maybe it's because of their religious or cultural upbringing. Or maybe it's a position that has to be taken in order to avoid cognitive dissonance with something that they've done. It's an interesting intellectual exercise, and sometimes I can work it out without coming straight out and asking them, "Well, why do you think it's common sense?"

But the other lesson is important, too: When someone says something is common sense, and that you should act in such-and-such a way because of it, it's vital not to adopt that common-sense attitude, if you don't already agree with it. It can be surprisingly compelling, if you're not careful (after all, who wants to be demeaned?), and you may sooner rather than later find yourself espousing the same position, seeing as it's "just common sense."

When Does It Start?

2011-04-18T16:53:00.000-07:00

So I'm driving to work the other day, and I'm stuck behind this car whose driver has decided that today, freeways shall be traversed at the speed of 42 mph. (In reality, I suspect this decision applies to most days, but I'm trying to be conservative here.) And it's almost impossible to pass him, because the stream of cars passing both of us is too dense and too much faster than we are to enter safely.

Eventually, I manage, and at a relatively safe moment, I cast a quick sidelong glance at him and affirm that he's northward of 80 years old. Now, there's a lot of talk that drivers that old should be looked at fairly hard and often to establish that they're able to drive safely, but I'm actually not thinking about that. What I'm thinking about is, at what point did he become a 42 mph kind of driver on the freeway? Was he always like that, or did he start out as what most of us would consider an ordinary kind of driver, and over time got slower and slower? I mean, maybe there are places where 42 mph is considered sort of daring, and that's where he grew up.

At this point in the discussion, someone invariably pipes up and mentions that such drivers are in fact safer than those driving at some higher speed, say, 80 mph, on the assumption that 80 mph is just inherently less safe. There's something to be said for that point of view, in that there's less time to avoid impacts if you're driving at a higher speed, and any impacts you do end up in are more dangerous. But that's only part of the picture.

The reality of the situation is that although (in Los Angeles) the freeway traffic occupies a continuum of speeds, most of the traffic—perhaps 90 to 95 percent—falls between 60 and 80 mph. And your risk of impact depends primarily on how often you encounter cars travelling at that range of speed. A long time ago, in grad school, I spent a little time figuring out how often you encounter cars on the road: either passing slower cars, or being passed by faster ones. And what I found was that the details of the speed distribution of cars matters very little. There are only four parameters of interest: the density of cars on the road, the percentage of cars you're faster than, the average speed of those cars that you're faster than, and the average speed of those cars that are faster than you.

That means that you could pretty much figure out the rate at which both my 42 mph driving friend and the hypothetical 80 mph driver would encounter cars by simply assuming everybody else was driving 70 mph. Our superannuated man behind the wheel would encounter cars nearly three times more often than the 80 mph driver, and encounter them at nearly three times the relative rate of speed. To be sure, the combined energy of an actual collision would be greater for 70 mph and 80 mph than it would be for 42 mph and 70 mph, but the increased frequency of encounters and the much shorter period of time drivers would have to avoid them would, I think, more than compensate for that.

All in all, I think if you want to drive slower to be safer, you're better off driving 60 mph, or whatever the lower end of speeds is for your road of choice.

A Different Kind of Search

2011-03-01T13:02:00.000-08:00

Because I have a bad habit of imagining all sorts of bizarre, improbable scenarios, I dreamt up this one: Suppose I woke up in a strange town, with no idea of who I was or where I came from, or indeed of any of my past. How do I find out?

Of course, I might go to the police or something like that, but perhaps they'd be no more helpful than myself. So I imagined I might start a blog, mostly unlike this one, where hopefully someone would recognize me. I'd start out by saying I'm most likely a missing person. Over time, as I remembered more about my past, hopefully, I'd start putting that down in the blog. Ordinary blog stuff I'd post much as anyone would, but all the identifying information I'd collect in a single post, accumulating edits, so it'd be more easily indexed by Google.

But what kind of stuff would be useful identifying information?

It occurred to me that the usual "obvious" stuff isn't necessarily all that useful in a case like this. My height and weight are not really specific enough to be a useful filter if someone else were looking for me. There are about 100,000 active missing persons in the U.S. (as of the end of 2009, according to the FBI); there must be hundreds that are my height and weight, or thereabouts. A picture would be more distinguishing, but it's hard to search for a picture of someone if all that's known is that they're missing.

So I think I'd try to post stuff that's more distinctively me. Maybe I'd somehow realize that I like to play jazz piano, or enjoy recreational mathematics, or have a rather deep interest in sports and statistics. Maybe I'd recollect some piece of poetry I'd memorized (or even composed). Along with any other more personal tidbits, I'd put them all in my Google flypaper for my old identity.

OK, obviously, I'm surpassingly unlikely to ever need to do anything like this (and it's unclear whether or not this planning would be something I'd remember if I ever did), but then it also occurred to me: Why don't we post this kind of information when we're looking for someone?

When a missing person poster is put up in my neighborhood, it always gives out some basic statistics for them, but the information is mostly generic: a photo, name, date of birth, height, weight, date missing, etc. If a person is amnesiac or doesn't wish to be found, that information is nearly useless—even the photo, since the person may have changed appearance quite dramatically.

What about favorite activities? What about peculiar habits or reflexes? What about a recording of the person's voice? Even people who have forgotten who they are, or wish others would, would not easily change such essential characteristics of themselves. And with the Web as adjunct for missing persons posters, putting up a voice recording is simplicity itself. I'm not suggesting that anything be put up that the family not be comfortable with, but lots of this stuff would violate privacy but little, and would (it seems to me) provide substantial aid in finding the missing person. I'm sure it's done to some limited extent—some posters do list some minor personal details—but a casual survey of local posters shows precious little of it. Can anyone explain why this isn't done more, if only on a voluntary basis? I know if a family member went missing, I'd want the poster to have as much trivial (i.e., non-security) detail as possible on it.

A Little Learning (Game Theory, Part Deux)

2011-02-17T14:35:00.000-08:00

Here, as promised, is the dangerous thing.

Suppose you're getting a sequence of playing cards, and you're trying to figure out some statistics for the playing cards. At first, the cards seem utterly random, but after a while, a pattern emerges: There are slightly more black face cards than red ones, and there are slightly more red low-rank cards than black ones. You're a statistician, so you can quantify the bias—measure the correlation coefficient between color and rank, estimate the standard error in the observed proportions, and so forth. There are rigorous rules for computing all these things, and they're quite straightforward to follow.

Except, you're playing gin rummy, and the reason you're receiving a biased sequence of cards is that you're trying to collect particular cards. If you change your collection strategy, you'll affect the bias. You may have followed all the statistical rules, but you've forgotten about the context.

It might seem entirely obvious to you, now that I've told you the whole story, what the mistake is, and how to avoid it, but I contend that a wholly parallel thing is happening in sports statistics. I'm going to talk about basketball again, because I'm most familiar with it, but the issue transcends that individual sport, yes?

I've previously touched upon this, but this time, with the first post on game theory as background, I'm actually going to go through some of the analysis. Again, we won't be able to entirely avoid the math, but I'll try to describe in words what's going on at the same time. If calculus makes you squeamish, feel free to skip the following and move down to the part in bold.

In our simple model, the offense has two basic options: have the perimeter player shoot the ball, or pass it into the post player, and have him shoot the ball. The defense, in turn, can vary its defensive pressure on the two players, and it can do that continuously: It can double team the perimeter player aggressively, double the post player off the ball, or anything in between. We'll use the principles of game theory to figure out where the Nash equilibrium for this situation is.

We'll denote the defensive strategy by b, for on-ball pressure: If b = 1, then all of the pressure is on the perimeter ball-handler; if b = 0, all of it's on the post player. An intermediate value, like b = 1/2, might mean that the defense is equally split between the two of them (man-to-man defense on each), but the exact numbers are not important; the important thing is that the defensive strategy varies smoothly, and its effects on the offensive efficiency also vary smoothly.

Each of the two offensive options has an associated efficiency, which represents how many points on average are scored when that player attempts a shot. We'll call the perimeter player's efficiency r, and the post player's efficiency s. As you might expect, both efficiencies depend on the defensive strategy, so we'll actually be referring to the efficiency functions r(b) and s(b). The perimeter player is less efficient when greater defensive pressure is placed on him, naturally, so r(b) is a decreasing function of b. On the other hand, the post player is more efficient when greater defensive pressure is placed on the perimeter player, so s(b) is an increasing function of b.

Now let's look at this situation from a game theory perspective. Will the Nash equilibrium of this system involve pure strategies, or mixed strategies? (A pure defensive strategy in this instance consisting of either b = 0 or b = 1.) Right away, we can eliminate the pure strategies as follows: If the offense funnelled all of its offense through one of those players, and the defense knew it, they would muster all their defensive pressure on that player. On the other hand, if the defense always pressured one of the players, and the offense knew it, they would always have the other player shoot it. Since those two scenarios are incompatible with one another, the Nash equilibrium must involve mixed strategies. Our objective, then, is to figure out what those mixed strategies are.

The offensive mix, or strategy, we'll represent by p, the fraction of time that the perimeter player shoots the ball. The rest of the time, 1-p, the post player shoots the ball. The overall efficiency function of the offense, as a function of defensive strategy b, is then

Q(b) = p r(b) + (1-p) s(b)

The objective of the defense, in setting its defensive strategy, will be to ensure that the offense cannot improve its outcome by varying its strategy p. That is, it will set the value b such that the partial derivative of Q with respect to p (not b) is equal to 0:

∂Q/∂p = r(b) - s(b) = 0

which happens when r(b) = s(b)—in other words, when the efficiencies of the two options are equal. The offense, in setting its strategy p, will aim to zero out the partial derivative of Q with respect to b:
∂Q/∂b = p r'(b) + (1-p) s'(b) = 0

which happens when

p = s'(b) / [s'(b) - r'(b)]

where b is taken to be the point where the two efficiency curves meet, since the offense knows the defense will play there.

But let's not worry about the offensive strategy; the important thing to take away is that at the Nash equilibrium, the defense will adjust its pressure until the efficiencies of the two offensive options are equal. Let's show what that looks like graphically.

We'll see here how game theory tells us what should be common sense: If the current defensive strategy were somewhere else than at the Nash equilibrium—say, if it were further to the left—the offense could improve its outcome by shifting more of its offensive load to the perimeter player, since he's the more efficient option on the left side of the graph. The reverse holds on the right side of the graph. Only at the point where they cross is the offense powerless to improve its situation by changing its offensive mix, which is exactly the outcome the defense wants.

As a corollary, the exact location of the Nash equilibrium depends vitally on the efficiency functions of the offensive components. If, for instance, one of the efficiency functions drops, the observed efficiency of the offense (that is, the efficiency measured by statistics) will also drop. Let's take a look at that graphically:

In this figure, the efficiency function of the post player, represented by s(b), has dropped. This has the effect of sliding the Nash equilibrium point down and to the right, which indicates increased ball pressure and a decrease in the observed efficiency of both the post player and the perimeter player. It's important to recognize that the efficiency function of a player refers to the entire curve, from b = 0 to b = 1, but when we gather basketball statistics, we merely get the observed efficiency, the value of that curve at a single point—the point where the team strategies actually reside (in this case, the Nash equilibrium).

Consider: Why might the efficiency function of the post player drop, as depicted above? It might be because the backup post player came in. It might be because a defensive specialist post player came in. In short, it might be because of a variety of things, none of which have to do with the perimeter player and his efficiency function—and yet the perimeter player's observed efficiency (whether we're talking about PER, or WP48, or whatever) drops as a result.

There's nothing special about the perimeter player in this regard; we would see the same effect on the post player if the perimeter player (or his defender) were swapped out. In general, the observed efficiency of a player goes up or down owing, in part, to the efficiency function of his teammates.

We see here an analogy to the distinction, drawn in economics, between demand and quantity demanded. Suppose we see that sales of a particular brand of cheese spread have dropped over the last quarter. That is to say, the quantity demanded has decreased. Does that necessarily mean that demand itself has dropped? Not necessarily. It could be that a new competing brand of cheese spread has arrived on the market. Or, it could be that production costs of the cheese spread have increased, leading to a corresponding increase in price. Both of these decrease the quantity demanded, but only the former represents a decrease in actual demand. Demand is a function of price; quantity demanded is just a number. If all we measure is quantity demanded, and we ignore the price, we haven't learned all we need to carry on our business. As economists, we would be roundly criticized (and rightly so) for neglecting this critical factor.

We are, in the basketball statistics world (and that of sports statistics in general), at a point where all we measure is the number. We don't, as a rule, measure the function. We apply our statistical rules with rigor and expect our results to acquire the patina of that rigor. But we mustn't be hypnotized by that patina and forget what we are measuring. If our aim is to describe the observed situation, then the number may be all we need. But if our aim is to describe some persistent quality of the situation—as must be the case if we are attempting to (say) compare players, or if we are hoping to optimize strategies—then we are obligated to measure the function. Doing so is very complex indeed for basketball; there are an array of variables to account for, and we have at present only the most rudimentary tools for capturing them. It is OK to punt that problem for now. But in the meantime, we must not delude ourselves into thinking that by measuring that one number, we have all we need to carry on our business.

Matching Up in Hyperspace (or, Thirty Dancing)

2011-02-14T16:16:00.000-08:00

Maybe it's because I've been writing about basketball a lot, but I thought today I'd do something a little different before continuing on, as promised, with a second game theory post.

A while ago, I remember reading an analogy about why it is that oil and water don't mix. (I don't remember where I read it, though, so if you recognize it, please tell me.) Is it that water molecules only "like" water molecules, and oil molecules only "like" oil molecules? Not at all—they all like water molecules!

A water molecule is often drawn as H-O-H, but that drawing is a bit misleading. The hydrogen atoms are actually attached at an angle, as below.

This one looks a bit like a Japanese cartoon character, if you ask me. At any rate, this asymmetry, top to bottom (as drawn here), means that we can speak of an oxygen end (the bottom) and a hydrogen end (the top). What's more, because of the way that electrons are arranged in each atom, the oxygen atom tends to draw electrons away from the hydrogen atoms. The oxygen end, so to speak, has more electrons hanging around it than the hydrogen end. Since electrons are negatively charged, the water molecule has a positive pole (the hydrogen end) and a negative pole (the oxygen end), and we say that the water is a polar molecule.

Water molecules attract each other because they are polar. The positively charged hydrogen end of one attracts the negatively charged oxygen end of another. In steam, the gaseous form, this is almost impossible to make out, because the molecules are too far apart and energetic, bouncing around far too wildly to show any mutual attraction. However, in ice, the solid form, the attraction is much more obvious.

It's a bit hard to tell which hydrogen atoms are associated with each oxygen atoms, but that's because in ice, the bonds are a bit confused. Even so, however, it's clear that we don't have water molecules bonding together oxygen-to-oxygen, or hydrogen-to-hydrogen. They only attach oxygen-to-hydrogen (in the hexagonal arrangement that yields those lovely snowflakes), because the molecules are polar that way. That's the way water molecules "like" each other.

Liquid water is intermediate between ice and steam. The molecules aren't fixed in place to each other as they are in ice, but neither are they bouncing wildly as they are in steam. Instead, they wander amongst each other, like people milling about in a crowd. And as they wander around, they stick to each other a bit, on account of their polarity. They attach and cohere, which makes water bead up, among other things.

What about oil molecules? Oil molecules tend to be symmetric in such a way that there is no clear polar end as there is in water. As a result, they are much less polar than water molecules are. Nonetheless, being weakly polar (under appropriate circumstances), they "like" other polar molecules, too. So why don't they attach to the water molecules, too?

The reason is that there is only so much room for molecules to attract each other. And here's where the analogy I mentioned earlier comes into play. You often find, at a school, that the most popular kids date other most popular kids (when they date), and the least popular kids date other least popular kids (again, when they date). Why is that? Is it that the least popular kids aren't attracted to the most popular kids? Well, it might sometimes be because of that, but often, they are attracted to the most popular kids; that is, after all, part of what makes someone most popular.

What gets in the way, however, is that the most popular kids, like most others perhaps, are also attracted to the most popular kids, and since such pairings satisfy both attractions, they get paired first. Then the next most popular kids pair up with other next most popular kids, they get paired next. And so on down the line. Or so the story goes.

Of course, it isn't quite that neat and clean with kids, but it is a reasonable approximation with what happens when you combine oil and water. They don't mix because the most popular water molecules hook up with other most popular water molecules, while the least attractive oil molecules are left hooking up with each other.

So much for oil and water. But now let's go back to that analogy, which as it so happens is what I really wanted to talk about. (The rest of that science was just for show?!) It doesn't ring true because we all know couples where we think, "Wow, she paired up with him?" How does that happen? It happens because people aren't one-dimensional.

Suppose all people were one-dimensional. Then you could rate each person with a number x—say, from 0 to 100. (I hate it when things are rated from 1 to 100. What's middle-of-the-road on such ratings? 50.5?) In such a case, if you have two 100's, wouldn't they choose each other above all others? You couldn't easily see a 100 pairing with a 25, if there's another 100 to choose from. Under such circumstances, the nth highest-rated male would always match up with the nth highest-rated female. Just like the kids at our hypothetical school.

Note: For reasons I despise (expositional convenience, basically), I'm writing this out heterosexually. Let it be clear that this isn't mandated in any way, and I'm aware of that. This treatment unfortunately makes it easiest for me to separate out two groups and draw what amounts to a bipartite graph between them. Sorry!

We might say, callously, that only one pair of people would say they feel completely satisfied with the pairing; everyone else is "envious" in the sense that there's someone else with whom they would rather have paired up. That's inevitable with one-dimensional people.

So let's give people another dimension: Let them now be rated with two numbers (x, y). Now, there is no universal and complete ordering on people. We might agree that if someone has both numbers higher than someone else, they are more appealing, but there is no universally accepted way to compare two people with one number higher and one number lower. This is akin to the problem with PER. It's entirely possible that everyone could be envy-free.

Here's what I mean. Suppose you have three males and three females. The three males are (60, 30), (50, 50), and (30, 60). So are the three females. Now there's no way you can say that the (60, 30) male is inherently superior to the (50, 50) male, or vice versa. The same is true of any other two males, or any two females. To decide amongst the alternatives, one needs a discriminating function of some sort. Let's say your function is 2x+2y. Then you would rank your three choices 180, 200, and 180, and you would choose the (50, 50) over either the (60, 30) or the (30, 60). If, on the other hand, your function was 3x+y, you would rank your choices 210, 200, and 150, and you'd choose the (60, 30) over the other two. Finally, if your function was x+3y, you'd pick the (30, 60) first. So it's possible for each of the alternatives to be first in someone's eyes.

Of course, to be a completely satisfactory pairing, both sides of the pairing must feel they got the best catch. But consider the (60, 30) male. Being a high-x kind of guy, he naturally values x more than y, perhaps, and his discriminating function will reflect that. (Some people, all they care about is x.) He might be exactly the sort of guy with a function like 3x+y, and would therefore pick the (60, 30) female. She, thinking likewise, would pick the (60, 30) male back. Likewise, the (50, 50) people might pair up with each other as mutually optimal choices, and the (30, 60) people too. It doesn't have to match that way, of course; it just has to match one-to-one. Maybe the (60, 30)'s love the (30, 60)'s, for instance, and vice versa.

On the other hand, this matching leaves someone who's (40, 40) out in the cold, because no discriminating function will rate them ahead of everybody else. Whoever matched up with them would always be upset that they didn't at least match up with ol' (50, 50).

It boils down to who's on the hull. The hull is made up of everyone who isn't universally worse than any other option. An illustration of this in two dimensions should hopefully make it clear why it's called the hull:

It's called a hull because everyone is contained in it. [Note: I had previously written convex hull here, but it later occurred to me that what I mean is just a hull.] Everyone on the hull could be someone's optimal choice; everyone else would be a consolation prize. It's possible that everyone would be on the hull, but it's unlikely, given a random selection of people.

Let's not be too hasty, though. There's an interesting dependency between dimensionality and being on the hull. In one dimension, exactly one person is on the hull (barring ties); everyone else is beneath him or her. In two dimensions, it's a bit more complex, but suppose you had a hundred people, evenly spread out between (0, 0) and (100, 100). On average, maybe five people would be on the hull. (The actual average is the sum 1 + 1/2 + 1/3 + ... + 1/100.)

Now let's increase it to three dimensions. If you have a hundred people spread out between (0, 0, 0) and (100, 100, 100), on average about 14 people would be on the hull. All 14 could be the optimal choice for some prospective mate. As the number of dimensions goes up (and the number of possible discriminating functions, too!), the percentage of people on the hull also goes up. With four dimensions, the average number of people on the hull is 28; with five, it's 44; with six, 59—more than half! Ten dimensions are sufficient to push it up to 94, and by the time you have, oh, let's say thirty dimensions, the odds are about ten million to one in favor of every last person being on the hull. Remember, it isn't necessary to have a highest value in any of the dimensions to be on the hull; all you need is to not be lower than anyone else in all of the dimensions. As the number of dimensions goes up, it becomes awfully unlikely that you'll be lower than anyone else in every single dimension. We can have an entirely envy-free matching, all with the help of increased dimensionality.

OK, this may seem completely crazy, and I wouldn't blame you for calling shenanigans. Who would actually go and rank people using a set of thirty numbers? But this is exactly what one of those on-line dating sites advertises it does. Well, not exactly; it actually claims to use 29 dimensions. Why 29? I would imagine because it sounds somewhat more scientific than thirty. But beyond that, I think that they use as many as 29 because it makes it almost inevitable that you'll be on the hull, that there'll be someone who you find optimal (or very nearly so), for whom you will likewise be optimal (or very nearly so). And although I think that's partly a marketing gimmick, I think there's some truth to it, too; if there weren't, the human race would have died out long ago.

I mean, how else does Ric Ocasek land Paulina Porizkova? For real, I mean!

A (Kind of) Gentle Introduction to Game Theory

2011-02-08T16:14:00.001-08:00

Prefatory to more basketball talk, I want to take a bit of time out to describe what I think is a rather elegant area of mathematics: game theory. Even the name is elegant—simple and to-the-point. As its name implies, game theory is the study of game strategies and tactics from a mathematical point of view. Rather than describe its foundations and move on from there, as a textbook would, I'm going to leap right in and use game theory in a couple of simple situations, which I hope will be a less obscure way of conveying what it's all about.

Suppose that David and Joshua are playing a friendly game of Global Thermonuclear War. At some point, both players have to decide whether to launch an attack or not. If David attacks and Joshua does not, then David wins and earns 5 points (this is a game, after all) and Joshua earns 1 point for being peaceable. Conversely, if Joshua attacks and David does not, Joshua earns 5 points and David earns just 1 point. If both attack, the Earth is rendered a wasteland and neither side earns any points; if both sides do not attack, everybody wins and both sides earn 6 points. The foregoing can be summarized in tabular form, as below.

David's payoffs are shown in blue, Joshua's in red. Let's run through a simple game-theoretical analysis of GTW. If we focus on just the blue numbers (David's payoffs), we see that if Joshua attacks, David's best strategy is to stand down (1 > 0). If Joshua stands down, David's best strategy is, again, to stand down (6 > 5). No matter what Joshua does, in short, David should stand down.

Moving over to the red numbers, we come to a similar conclusion for Joshua's strategy: No matter what David does, Joshua is better off standing down (either 1 > 0, or 6 > 5). As a result, both sides stand down; the only way to win is, indeed, not to play. Whew!

But maybe we shouldn't all relax quite yet. Suppose that we were to adjust the payoff matrix (which is what we call that table up there). Heads of state often get a little nationalistic, and they may well decide that a world without the enemy is better after all, even if we do have to suffer from a little radioactive fallout. At the same time, perhaps it is better to go out fighting and take out the enemy, even as we ourselves are getting wiped out. Then, possibly, the payoff matrix would look like this:

The numbers have changed only a little, but the conclusion is quite different: This time, no matter what Joshua does, David is better off attacking, and no matter what David does, Joshua is better off attacking. The upshot is that both sides end up attacking and wiping each other off the map. The rest of you, I hope you look forward to serving your cockroach overlords...

In the two examples above, the eventual solution has the property that the strategy for both sides was the best they could do, no matter what the opponent did. Such a solution is called a Nash equilibrium, after John Forbes Nash, who won a Nobel Prize in economics for his work in such games. (Yes, the Beautiful Mind guy.) In fact, in each case, the winning strategy was a single choice: either "always attack" or "always stand down."

That is not always the case. Consider a rather less violent game of football (well, somewhat less violent). On a crucial third down, the Steelers can choose to run the ball, or throw the ball; the Packers, on the other hand, can choose to defend the pass or defend the run.

Here's how we might model things. We'll let the payoff be simply the chance, the probability, that the Steelers make a first down. We'll also say that if the Steelers pass, they make a first down 60 percent of the time when the Packers defend the run, but only 20 percent of the time when they defend the pass. If the Steelers run, they make a first down 50 percent of the time when the Packers defend the pass, but only 30 percent of the time when they defend the run. Here's the payoff table:

(We've included the Packers' payoffs in red, although they can be derived from the Steelers' payoff by subtracting from 100 percent.) This time, matters are not as clear cut: For obvious reasons, the Steelers' best option depends on what the Packers do, and the Packers' best option depends on what the Steelers do. But, for our purposes, they both have to show their hands at the same time. What does game theory have to say about this kind of situation?

To figure that out, we'll have to consider a new kind of strategy, called a mixed strategy. A mixed strategy (as opposed to the pure strategies we considered above) is simply one that chooses each of the options with a certain probability. For instance, one possible mixed strategy the Steelers could employ is to run the ball half the time, and pass it the other half. Similarly, the Packers could defend the pass 60 percent of the time, and defend the run 40 percent of the time. There are an infinite number of different mixed strategies both teams could employ. How do we figure out what mixed strategies are actually the best for each side?

Here's where game theory gets a bit hairy (hence the "kind of" in the title). Essentially, what the Steelers want to do is to make their strategy "resistant" against the Packers, in the sense that no matter what the Packers do, they can't damage the Steelers' chances of making their first down. And the Packers want to set their strategy so that no matter what the Steelers do, they can't improve their chances of making the first down. Such a situation, where neither side can do any better without the other side changing what they do, is also a Nash equilibrium. The brilliant thing that Nash did, which earned him that Nobel Prize, was to show that in such games, there is always a set of mixed or pure strategies that yields a Nash equilibrium.

What follows is pretty heavy mathematical stuff. If you don't want me to go all calculus upside your head, feel free to skip it and go to the conclusion in bold, below. Here's what we do. We characterize the Steelers' strategy by p, the probability that the Steelers pass the ball, and the Packers' strategy by q, the probability that they defend the pass. From the Steelers' perspective, there are four distinct possibilities:

Steelers pass (p), Packers defend the pass (q): Payoff is 20 percent.
Steelers pass (p), Packers defend the run (1-q): Payoff is 60 percent.
Steelers run (1-p), Packers defend the pass (q): Payoff is 50 percent.
Steelers run (1-p), Packers defend the run (1-q): Payoff is 30 percent.

Putting it all together, we get an expression for the Steelers' payoff S:

S = 0.2 pq + 0.6 p(1-q) + 0.5 (1-p)q + 0.3 (1-p)(1-q)

S = 0.3 + 0.3 p + 0.2 q - 0.6 pq

We want to find the value of p that makes the partial derivative of S with respect to q equal to 0. That is, we need the value of p that makes it utterly irrelevant what the Packers do with their q (as it were).

∂S/∂q = 0.2 - 0.6 p = 0

which happens when p = 1/3. We can do a similar expression for the Packers' payoff P from their side of the payoff matrix:

P = 0.8 pq + 0.4 p(1-q) + 0.5 (1-p)q + 0.7 (1-p)(1-q)

P = 0.7 - 0.3 p - 0.2 q + 0.6 pq

and then the optimal strategy for the Packers is dictated by

∂P/∂p = -0.3 + 0.6 q = 0

which happens when q = 1/2. So the Nash equilibrium happens when the Steelers pass the ball 1/3 of the time and run the ball 2/3 of the time, and the Packers defend the run 1/2 of the time, and defend the pass 1/2 of the time. Under these strategies, the Steelers make their first down 40 percent of the time, and nothing the Steelers do on their own can increase it, and nothing the Packers do on their own can decrease it (given the payoff table). That's what makes it a Nash equilibrium.

Notice that none of the pure strategies work as well as the mixed strategies do. If the Steelers always ran the ball on third down, the Packers knew that, they would just defend the run and limit the Steelers to making first downs 30 percent of the time. It's even worse if the Steelers passed all the time; they'd make a first down only 20 percent of the time. Conversely, if the Packers always defended the run, and the Steelers knew that, they'd just pass all the time and make their first down with 60 percent efficiency. And so on.

The salient thing to take from all this, though, which I'll get into in my next post, is that although the Steelers' odds of making the first down don't depend on the Packers' strategy, at the Nash equilibrium, their best strategy does depend on how good the Packers are at defending the various options (which is represented by the payoff matrix). Although that is hardly earth-shattering in this particular case, we'll see that has interesting repercussions when trying to rate individual player achievement.

How to Be Wrong, With Statistics!

2011-01-28T15:33:00.000-08:00

Please, just stop it. You're hurting me.

Anyone who understands statistics at all cannot dispute that Kobe Bryant does not perform well statistically, in the clutch. But anyone who understands statistics well cannot dispute that the current statistics are woefully under-equipped to discern who is the clutchiest player in the league.

Look: Nothing happens in a vacuum. We look at crunch-time statistics because it's the most exciting part of the game, when it happens. But it's only one way to condition a play.

What do I mean by condition? I mean "to restrict the characteristics of." With respect to comparing players on their clutchiosity, the objective should be to condition the crunch-time plays sufficiently that we are comparing apples to apples, and oranges to oranges. And here, as with many other aspects of basketball, we simply don't have the statistics to do it at our disposal.

For instance, suppose that we wish to compare two players, A and B. Suppose that A's offensive efficiency (points per possession) is greater than B's, with less than 24 seconds on the clock and the team tied or down no more than three points. Does that mean that A is clutchier than B?

Not at all. If B has stiffs for teammates, compared to A, then he's likely going to be faced with tighter individual defense than A, and likely earn a lower offensive efficiency than A. That's a couple of instances of "likely" in there, but the point doesn't have to be ironclad, it just has to be plausible, even probable. We just don't know enough to conclude with anything approaching certainty that A is clutchier, because we haven't conditioned on the teammates. (Or the defense, for that matter.)

Observe that this is mostly independent of what statistic you use to measure clutchiness. Suppose, instead, that you decide to use win probability increment. A player's ability to increase his team's likelihood of winning is still going to be affected by his teammates: If he passes, they will have a lower probability of scoring; if he doesn't, the defense can afford to defend him more tightly.

Of course, maybe you're OK with this kind of quality vacillating with things like which teammates a player has. But personally, I think such a measure has a certain ephemeral aspect that we don't usually associate with clutchiness.

The problem is, how can you possibly condition on the kind of teammates that a player has? Players don't change teammates the way they change their clothes (or at least they shouldn't). So what do you do?

Here's my gentle suggestion: Stop trying to answer these abstract questions statistically. I've been using outlandish forms of the word "clutch" to underscore this, in case you haven't noticed, but my point is serious. Use statistics to answer the questions they can. As the field advances, we'll be able to answer more of these questions, but in the meantime, use the same method we've been using all along: subjective observation. Western civilization didn't break down before we had PER. Nothing hinges on who people outside the game think is clutch. And mostly, stop pretending to any degree of certainty in the matter, just because a number is attached to it.

EDIT: Since I'm a fan of Kobe Bryant, one might reasonably wonder whether or not I've got a built-in bias against crunch-time statistics, since almost all of them (except perhaps a raw count of shots made in crunch time, as opposed to efficiency) point to quite a few players as being superior in the clutch. Obviously, I can't deny said bias. Quite possibly I would not be making these same arguments, or making them with quite the same degree of vehemence, if those statistics showed Bryant in a better light.

That being said, however, I don't think the question of using statistics to examine clutchitude should be predicated on how well they accord with conventional wisdom (where Bryant is, indeed, king of clutch). In my opinion, there are quite compelling fundamental arguments that straightforward linear classifiers such as PER or offensive efficiency or wins produced, conditioned on crunch time or not, are simply not reliable indicators of individual performance, and those arguments would remain valid regardless of whether I espoused them, or of whom they revealed to be the top performers, in crunch time or in the game overall.

Voter Mixing Equals Criterion Mixing

2011-01-05T16:06:00.000-08:00

I'm going to talk about basketball and probability again. Wasn't that obvious from the title of this post?

It's apparently never too early to talk about the MVP award for the NBA. We're coming up on the halfway point of the season, and writers have been tracking the MVP candidates for, oh, about half a season. Nobody takes them seriously until about now, though.

One side effect of the question being taken seriously is that some wag will point out that the MVP is not—and has never been—defined precisely. In fact, I can't find anywhere where it's been defined at all by the NBA, precisely or otherwise. That leaves the voters (sportswriters and broadcasters, mostly, plus a single vote from NBA fans collectively) to make up their own definition, a situation that said wag invariably finds ludicrous.

Well, here's one wag that finds this situation perfectly acceptable. Desirable, even.

Listen: There is no way that everybody will ever agree on a single criterion for being the "most valuable player." Most valuable to whom? The team? The league? The fans? Himself? (I can think of a few players who certainly aim to be most valuable to themselves.) And what kind of value? Wins? Titles? Highlights? Basketball is entertainment, after all. There are just too many different ways to evaluate players.

Instead, we might imagine that some writers would get together at some point and define MVP as a mixture of criteria. For instance, the title of MVP could be based in equal parts—or inequal parts, for that matter—on individual output, contributions to team success, and entertainment value.

Except, I'd argue that that is exactly what we've been doing for all these years. We have all these voters, all of whom have differing ideas of what the MVP does (or should) stand for. Some people think it should be based on individual statistics (Hollinger's Player Effectiveness Rating, or PER, is a current favorite). Some people think it should be based, at least in part, on team success, so team wins are an input to the decision (a 50-win minimum is a popular threshold). Still others dispense with explicit criteria altogether and vote based on reputation or flash.

Well, if exactly the same number of voters take each of those different perspectives on MVP, then we will have an MVP based in equal parts on individual output, contributions to team success, and entertainment value. And if more voters lean on individual output than on entertainment value, then the MVP make-up will show that same leaning. Voter mixing equals criterion mixing!

What's more, this criterion mixing is automatic. No committee needs to be formed, and the exact mixture evolves as the voter population evolves. If someday team success becomes more important to the basketball cognoscenti, then it'll automatically have a larger impact on MVP selection. No redefinition is necessary.

Can this equivalence be demonstrated on any kind of formal level? In something as complex as basketball, my guess is not. But it's close enough, and intuitive enough, that I think it just doesn't make sense to gripe about the MVP lacking a precise definition. As long as each voter comes to their own decision about what it stands for, we'll get the mix that we should.

Too Many Damned Monkeys

2010-11-24T16:31:00.000-08:00

What do you need more monkeys to do: (a) guarantee the writing of all of Shakespeare's plays, or (b) be able to sink an infinite number of basketball shots in a row? OK, I realize that this is entirely inconsequential, but it actually came up a couple of days ago in what would otherwise have been fairly ordinary coffeehouse conversation, so let me bring you up to speed.

The anchor point is the notion that by having an infinite number of monkeys, each of them sitting in front of a typewriter, randomly typing away, you could guarantee that one of them would surely generate a perfect typescript of Hamlet. Or Macbeth. On the other hand, you'd also guarantee that one of them would generate a "perfect" version of Astrology for Dummies.

What this is really about (since few of us are likely to corral together an infinite number of monkeys) is the so-called cardinality of possible books of arbitrary (but finite) length. Now what's cardinality? The cardinality of a finite set is simply the number of things in the set. So, for example, the cardinality of the U.S. Supreme Court justices is nine, usually. The cardinality of the English alphabet is 26. And the cardinality of the sand grains on the Earth is some almost unimaginably large number. But it's still finite.

Infinite sets are a whole 'nother kettle of fish. Maybe the simplest example of an infinite set is ℕ, the set of natural numbers: 0, 1, 2, ... We use the ellipsis (...) to indicate that the natural numbers go on, forever, without end. There is no last number; in other words, infinity is not really a number in the usual sense. Nonetheless, we might say that the cardinality of ℕ is infinity, which is conventionally denoted ∞.

But in so doing, we would be ambiguous, for as it turns out, there are different varieties of infinity. The infinity of ℕ is the smallest possible infinity, but there are larger infinities. That sounds kind of paradoxical: How could a set go on longer than forever?

Well, let's see if we can construct an infinity that's larger than the cardinality of ℕ. The first thing we might do is add some more numbers to ℕ and see if that yields a set with larger cardinality: we might add in all the negative whole numbers, to get ℤ, the set of all integers. Shouldn't ℤ, which is (naively) almost twice as big as ℕ, have nearly twice as large a cardinality?

No, and here we run into one of the fundamental differences between finite sets and infinite sets. Suppose we divide ℕ into two mutually distinct subsets: O (1, 3, 5, ...) and E (0, 2, 4, ...). Intuitively, both O and E are infinite sets. But if ℕ is the union—the sum set, so to speak—of O and E, is ℕ then doubly infinite?

Mathematicians decided that was too much. So cardinality is defined, less intuitively but more consistently, as follows. We say that the cardinality of the English alphabet is 26, because there are 26 letters in the alphabet. Another way of saying the same thing is that the letters of the alphabet can be placed into a one-to-one correspondence with the set of numbers from 1 through 26: 1-A, 2-B, 3-C, and so on, up to 26-Z. You can try a similar exercise with the U.S. Supreme Court justices.

If we define the notion of cardinality this way, then it follows that two sets have the same cardinality if there exists a one-to-one correspondence between the sets. Somewhat amazingly, then, the set of odd numbers O has exactly the same cardinality as ℕ, because one can define a one-to-one correspondence that matches each number in ℕ with a number in O, and vice versa: 0-1, 1-3, 2-5, 3-7, ..., in each case pairing a number n from ℕ with the number 2n+1 from O. It doesn't matter that one can define a correspondence in which the two sets don't match one-to-one; all that matters it that there exists at least one correspondence where they do match.

Pretty clearly, we can do the same thing with E, matching n from ℕ with 2n from E. So all three sets—ℕ, O, and E—have the same cardinality, even though O and E combine to make up ℕ. The question then arises: Are there infinite sets that can't be matched up one-to-one with ℕ, no matter how you try? We can certainly do that for ℤ, matching up all odd numbers m in ℕ with (-1-m)/2 from ℤ, and all even numbers n in ℕ with n/2 from ℤ.

Well then, what about ℚ, the set of rational numbers—all possible fractions involving only whole numbers in the numerator and denominator? Surely that is a bigger set. But as it turns out, ℚ also has the same cardinality as ℕ, even though there are an infinite number of possible numerators and an infinite number of denominators. This state of affairs has led people to write such semi-sensical equations as

∞ + ∞ = ∞

since O and E combine to make ℕ, and

∞ x ∞ = ∞

since all the infinite pairings of ℕ make up ℚ. (By the way, in case you're wondering, ℕ stands for Natural Numbers, of course; ℤ stands for Zahlen, the German word for number; and ℚ stands for Quotient.)

All right, what about ℝ, the set of real numbers? Can that set be placed into a one-to-one correspondence with ℕ? Based on the way things have been going, you might suppose that they could, but in 1891, the German mathematician Georg Cantor (1845-1918) showed that in fact they could not, that ℝ was a strictly larger set than ℕ.

His argument was clever one, employing proof by contradiction. Suppose, Cantor said, that you could find such a one-to-one correspondence. You could write out a catalogue of real numbers then, as follows:

1 - 0.14159265...
2 - 0.71828182...
3 - 0.41421356...

and so forth. Now, suppose you construct a new number g, using the following process: The first digit of g will be the first digit of the first number in your catalogue, plus one; the second digit of g will be the second digit of the second number, plus one; the third digit of g will be the third digit of the third number, plus one; and so on. We could read out g along a diagonal in our catalogue of real numbers, like this:

1 - 0.24159265...
2 - 0.72828182...
3 - 0.41521356...

So g would be the number 0.225... This number g has an amazing property—it cannot appear anywhere in our catalogue of real numbers. Why not? Because it differs from the first number at the first digit, it differs from the second number at the second digit, it differs from the third number at the third digit, ... in short, it differs from every single number in the catalogue.

We have a contradiction: Either g is not a real number, or our catalogue is not complete as we thought it was. Well, g is clearly a real number, so the problem must lie with the other part—our catalogue is not complete. After all, we only assumed we could create such a catalogue. Since it seems we cannot, no one-to-one correspondence exists between ℝ and ℕ.

You might think that there's a simple way around this, if we simply add g to our catalogue, or rearrange it in some way. But Cantor's diagonalization argument, as it is usually called, would apply just as well to this new catalogue. No matter what catalogue you attempt to compile and amend, there's no way to avoid the construction of a real number that's nowhere in the list. Those two sets fundamentally have different cardinalities. And because of that, we can't use the single symbol ∞ to denote their cardinalities. Instead, mathematicians use the aleph-numbers: The cardinality of ℕ is ℵ0 (pronounced "aleph-null"), and under certain commonly held assumptions, that of ℝ is ℵ1 (pronounced "aleph-one").

So what about all those scripts for Shakespeare? Each of them can clearly be entered into a computer document, which is represented by a finite string of digits in the computer. We can therefore place the set of possible scripts into a one-to-one correspondence with the integers in ℕ, meaning that the set of scripts has cardinality ℵ0, so ℵ0 monkeys would be enough for at least one monkey to write any given script. (In fact, ℵ0 monkeys would write that script.)

But what about the infinite string of makes in a basketball game? These are infinitely long strings of basketball shots (each one with ℵ0 shots), so there would be a one-to-one correspondence between those strings and infinitely long sequences of digits—i.e., ℝ, the reals. So it would take ℵ1 monkeys to guarantee that at least one monkey would shoot any given sequence (in particular, the one sequence consisting of all makes).
I don't even want to know about the bananas.

An Unusual Series

2010-09-17T17:29:00.000-07:00

Which may not be all that interesting to you, unless you're interested in recreational math. For lots of you, that may be sort of an oxymoron. (Although, I'm hoping it's less likely among readers of my blog than it would be among the general population.)

Here's the idea. Start with an integer. Add its digits together. If that sum is even, halve the number (not the sum of digits) to get the next number. If the sum is odd, add one to the number.

For instance, suppose we start with the number 10. Its digits sum to 1+0 = 1, so we add 1 to get 11. Those digits sum to 1+1 = 2, so we halve it to get 11/2 = 5.5. Those digits sum up to 5+5 = 10, so we again halve the number to get 2.75. Those digits sum up to 2+7+5 = 14, so we again halve the number to get 1.375...well, I think you get the idea.

On the other hand, suppose you start out with the number 1. Its one digit sums to 1, so we add 1 to get 2. Its single digit sums to 2, so we halve it to get 1 again. Obviously, this series repeats forever: 1, 2, 1, 2, 1, etc.

The first eight numbers, 1 through 8, all end up at that same repeating sequence. The next number, 9, leads immediately to 10, which starts out as I worked out above, and then goes on indefinitely: Each number has one more digit after the decimal point than the preceding number, so the series never repeats, and it never reaches zero, either.

In my limited trials, every integer I've started out with either ends up with the repeating sequence 1, 2, 1, 2, 1, ..., or else it eventually merges with the same series that you get if you start with 10 (or 9, for that matter). So, two questions for those of you who might like to play with this kind of thing:

Is it true that the series for any integer always either ends with the sequence 1, 2, 1, 2, 1, ..., or else merges with the series that starts with 10?
Consider the series that starts with 10. As we said, it goes on forever, without repeating. What is the average of the numbers in that infinite series?

Neither of these questions can be answered definitively (as far as I can tell) with brute-force computation, although the results might be suggestive. If you do want to try some computations, use an infinite-precision package; our friend Bernie has already tried it with ordinary floating-point numbers (eight-byte doubles, I think), and roundoff error rendered everything after about the 15th number quickly invalid.

P.S. Don't ask me how I got started thinking about the series. It's inspired in part by this guy, but I've already forgotten how I decided to think about this variant.

Grasping at Genius

2010-09-03T16:03:00.001-07:00

No, this isn't about me trying to become a genius. My aim is a lot more modest: trying to draw a bead on what genius is. Partly this is motivated by my last post about music, but mostly it came out of a discussion I had several years ago with a co-worker over whether athletes could be geniuses at their sport. I thought they could, and he thought not. He conceded that they had some outstanding skill, but felt that it would be demeaning the word "genius" to call it that. I was willing to be a bit more expansive with the term. One does have to be a little careful—probably half the parents out there think their precious little ones are geniuses—but limiting genius to a specified list of fields seemed unnecessarily restrictive to me.

The discussion more or less had to end there because we never really grappled with the larger issue of what genius really is, and without that any debate over whether it means anything in sports is putting the cart before the horse. I want to tackle that now, so I can go back and win the original argument.

First of all—because I'm sick and tired of hearing about it, even now—what is genius not? It is not a high IQ, or intelligence quotient. Lots of folks are intimidated by numbers (especially, but not exclusively, those who do not feel comfortable around them), to the point that any description using them feels more objective and unassailable. Well, they might be that, but what's lost when a number is attached to anything is the process by which that number was derived. If you don't know and understand that process, the number—while not exactly meaningless—is not as reliable as it sounds.

In the case of IQ, the formula is generally straightforward; what's not so clear are the principles on which questions are selected for IQ tests. If you've ever taken one, you know that questions on such tests are fairly narrowly circumscribed: which one of these things doesn't belong, how many blocks are there, numerical or word analogies, etc. The only thing that we can be sure IQ tests measure is how well someone takes IQ tests. Beyond that patently circular assertion, it gets hazy. Does it measure intelligence? How about genius? There are lots of folks who have very high IQs (Marilyn vos Savant—really? that kind of name?—comes to mind) who nonetheless evince no obvious signs of genius. To her credit, vos Savant doesn't make any claims of genius for herself.

If we can't rely on a test to identify genius, we are back to Potter Stewart's famous dictum (in his concurring opinion in Jacobellis v. Ohio regarding hard-core pornography): "I know it when I see it." So where do we see it?

If we start with the so-called hard sciences (physics and chemistry), plus mathematics, I think you'll find little argument that folks like Archimedes, Isaac Newton, Carl Friedrich Gauss, and Albert Einstein were geniuses. Expand that to all of letters and sciences, and you embrace other noted geniuses, such as Charles Darwin, Louis Pasteur, and B.F. Skinner. But maybe these get a little dicier. These are great scientists, to be sure, but what about them promotes them beyond the ordinary rabble?

You might expect that things would get dicier still when we go to the fine arts, but at least in my experience I find less argument about ascribing genius to artists like Leonardo da Vinci (also an engineer), William Shakespeare, Auguste Rodin. How about musicians? Ludwig van Beethoven, Richard Wagner, and Igor Stravinsky all wear the mantle of genius, and wear it rather comfortably at that. (Yes, I realize these are all dead white dudes. I'll get to that in a moment.)

Let's pause a while and take stock of what we have. Accepting for the sake of discussion that these people are all geniuses, what makes them so? They don't just do what ordinary people in their professions do, only better—although by and large, they do do those things better. They also don't just do what ordinary people can't do—although, again, they do do that, too. What sets them apart is that they do things that ordinary people in their profession could never even conceive of, before the geniuses did. Arthur Schopenhauer put it this way:

"Talent hits a target no one else can hit; genius hits a target no one else can see."

I must emphasize that innovation is a vital part of this. One of Newton's most important contributions to physics was a mathematical demonstration of the law of universal gravitation (the so-called "inverse square law" of gravitation) from Kepler's observations and laws of planetary orbits. That same law is derived countless times over by students in undergraduate physics classes around the world (albeit using analysis, rather than the essentially geometrical means that Newton employed). That doesn't mean that any of them, let alone each of them, is a budding Newton, for likely none of them, plucked at birth and set down in a pre-Newtonian world, could have done what Newton did. Newton's genius lay in blazing the trail that future scientists and students would follow.

In that context, then, let me add a few other names to the list: Charlie Parker, Miles Davis, Herbie Hancock. Jazz is an art form, among others, that combines composition and performance in a single moment, adding for the first time—to my list, anyway—the element of dynamism. (I don't mean to slight other performance geniuses, such as actors and stand-up comedians, but I'm trying to make a point!) Although jazz tunes are composed to a certain extent, a fundamental aspect of jazz performance is improvisation. No two jazz performances are ever exactly the same—not, at any rate, to the extent that classical music performances are alike. The music is constantly written and rewritten by each new performer that approaches it, and each new performer must contend not only with the structure of the music, but with the performers around him or her, in an endeavor that is, in the best of cases, at once collaborative and competitive. And genius denotes the ability, moment to moment, to conceive and perform what others in that situation could not even imagine.

From that point, how far of a step can it be to arrive at sports? I'm going to talk about basketball, because it's the sport with which I'm most familiar, but similar arguments could be made for other sports. (Imagine, for instance, the shots that Tiger Woods can execute that others would never even attempt, or the sudden volley, deft but fierce, of Pete Sampras.) Basketball, like jazz, requires the constant attention of the athlete to the ever-changing state of the game, from the highest level down to the smallest detail, and the ability to respond to that state, all on the spur of the moment. Where's that pick going to be in five seconds? What are the possible tactical options available to me, given the current score and time remaining? Seeing the passing lane halfway down the court is a geometric exercise in negotiating tangled world-lines in the four dimensions of space and time; to actually complete the pass, when everyone else is watching, one must summon the legerdemain of a practiced conjurer.

We think of sports as an essentially physical activity (which is probably why my co-worker could never attach the genius label to an athlete), but in its own way it is as demanding on the intellect as the most abstruse mathematical theorem, and unlike the mathematicians, who can return now and again to their labors when it suits them, the athlete has only the splittiest of split-seconds to act—or else the instant is gone. Who are we to say that genius could not act here, as well as anywhere else?

We may debate whether or not Wilt Chamberlain, Michael Jordan, or Magic Johnson merit the label of genius, whether or not what they do exceeds the conception of their colleagues. But not, in my opinion, whether the question makes sense. Even we non-geniuses can see that, I think.

The Sound of Music

2010-07-30T18:03:00.000-07:00

I've always been intrigued by music; there's something almost incomprehensible about its appeal, which, nevertheless, you desperately want to comprehend. At least I do. And the best I can do is sort of nibble 'round the edges.

For one thing, it's a temporal art form. Mostly you experience it over time, however long it takes to hear a performance (or a recording thereof). And if you feel its impact, be it sadness, suspense, gladness, or even a kind of horror, that too is felt over the duration of the music. It never happens that a piece of music saves up all of its emotional impact for a single whap in the face, like a painting or a sculpture might. Yes, I'm aware that those art forms have nuances that can take extended or repeated viewings to appreciate. But for those forms, it is possible for the entire gestalt to strike you at a single moment, followed by a sustained decay of gradual discovery.

To be sure, trained musicians can look at a musical score and apprehend it. But even then—unless they are familiar with the music, and sometimes even then—they hear the music in their head, once again over time.

And the emotions you feel—oh! Music seems to speak to us in a language that is uniquely suited, not for communication, but only for emotional transference. A strain of music can connote hope or despair, struggle or triumph, seemingly no matter your roots or background. You almost think that if only somehow that universality could be harnessed, you could solve the world's problems in a single swoop—but then, that sounds like a travesty to be visited on music. At times I feel as though it should be protected from that kind of directed use.

Music stays in us. We have a tune stuck in our head. As much as we may appreciate the Mona Lisa or the David, how often do we complain that one of those (or their modern counterparts) are stuck in the same way? Maybe music gets a leg up from being a primarily auditory art form. We get so much of our information about the world from our eyes; our ears are generally accompanists, not the featured performer. As a result, though, it works its magic subliminally, providing a soundtrack for us. Seeing a visual art form may put us in an ecstatic trance of exploration, but rarely does it pull something directly out of us, something we recognize. Whereas surely all of us have songs that invariably draw forth some sharp memory. Music makes us aware that we have a story.

None of which brings me much closer to being able to comprehend its appeal in any meaningful way.

A Tale to Tell

2010-07-15T15:07:00.000-07:00

People love to tell stories. It's something that I think is fundamentally built into the human psyche. Having others' attention and entertaining them with a good story is as strong a rush as there is. I've heard that the vast majority of criminals, when arrested, will simply confess because the urge to tell their story to a captive audience is just too strong.

This tendency manifests itself even when there is, quite literally, no story to tell. The clustering illusion denotes the human impulse to see significance in random patterns. Suppose a series of ten coin flips goes as follows: T, H, H, H, T, T, T, T, T, T. A lot of people (but hopefully not too many of my own readers) would see the coin as streaky, though how they would react to that perception might vary: Some might conclude that the coin was "due" for heads and bet that way, while others might conclude that it was on a "tails" streak and bet that way. (For what it's worth, I flipped a quarter ten times and that's exactly the way they came out.)

This has major implications for how we watch and remember sporting events. Maybe the most obvious example of this is the so-called "hot hand" in basketball: the idea that a shooter is "in the zone," and more likely than normal to hit any given shot. Various studies have looked for and failed to find evidence for the hot hand. It's entirely possible that the hot hand is wholly illusory, that it's just the clustering illusion in play. However, as Carl Sagan was wont to say, absence of evidence is not evidence of absence. Except for free throws, in which shot selection and defense have no play, shooting accuracy is highly contextual. Some shots are wide open, while others are tightly contested. They are shot from all over the field. Some are shot on the run, others are shot on the step back, while still others are spot up shots. What's more, players are intensely aware that they're hot, and as a result may shoot any hot hand they have in the foot (as it were). All these factors conspire to make the hot hand difficult indeed to discern. (For free throws, there is apparently a moderate hot hand; see this paper (or at least its abstract) by Jeremy Arkes.)

But a more basic example is in how we all remember and talk about the game afterward. We talk about the shooting struggles of such and such a player, and how (if our team won) he overcame that adversity and pushed through to get the win. We look back in our memory and find events that, although they seemed minor at the time, turned out to have momentous impact on the outcome of the game. Consider this account of Game 7 of the 2010 NBA Finals:

With 8:24 left in the third quarter, Celtics point guard Rajon Rondo picked up a loose rebound off Paul Pierce's miss from 19 feet, and pushed it back in to put the Celtics up 49-36. And through 28 minutes of play, Kobe Bryant had had an abysmally poor night on the offensive end. He had shot three of 17 from the field and one of three from the free throw line for seven points and a true shooting percentage of only 19 percent. Largely as a result of his terrible performance, the Lakers found themselves down by 13. To be sure, Bryant had eight rebounds (four of them on the offensive end), but that hardly put a dent in his overall play.

On the play, however, Pierce injured his shoulder and had to sit out for a spell. Bryant thought he saw something that he could exploit as a result, and went to work. On the very next play, he drove into the lane and drew a shooting foul on forward Rasheed Wallace. He only made one of his two free throws, but from then on his performance surged abruptly upward. Starting with that play and for the rest of the game, Bryant gathered seven more rebounds and shot three of seven from the field and 10 of 12 from the free throw line for 16 points and a true shooting percentage of 65 percent, leading his team to a 83-79 win for the title.

Sounds pretty interesting, doesn't it? Makes you wonder what it was that Kobe saw that he could take advantage of. I would wonder, too, except that I just now made it up. Everything else is true, but the sentence in bold is conjured out of whole cloth. Actually, Kobe simply tossed his hands in frustration for a second before taking the inbounds pass and dribbling it upcourt. In trying this narrative out on a couple of folks, though, I found that it was compelling because once people see the remarkable contrast between Kobe's play before that moment and his play after it, they assume that something equally remarkable must have happened to precipitate it. We will latch onto any little thing as an explanation, even if it had no more to do in fact with the game than any other little thing. Right place, right time.

As far as I can tell, though, there was nothing in that game that happened to Kobe. Aside from a trio of truly horrible shots that he took with the shot clock running down, his shot selection was not noticeably worse while the Lakers were falling behind than it was during their comeback. Sometimes, you know, a cigar really is just a cigar.

Points on the Board

2010-07-02T15:18:00.000-07:00

In the wake of the Lakers' mud-slogging Game 7 win in the NBA Finals over the Boston Celtics by the score of 83-79, some fans were incredulous that a team could shoot 32.5 percent (27 of 83) and still win. In fact, many of them felt that the Celtics lost the game, rather than the Lakers winning it. To me, that sounds a little silly, inasmuch as basketball is a head-to-head sport. If the Lakers were shooting that poorly, presumably the Celtics had something to do with that, and just as presumably, the Lakers were doing something else to win the game.

So what was that something? I'll give you a little hint. It begins with "offensive," and it rhymes with "rebounding."

In the unlikely event you haven't caught on, a major key to the Lakers' victory was their offensive rebounding; they won that battle 23-8 over the Celtics. To be sure, gathering 23 offensive rebounds is usually a dubious feat, for it requires the team to miss far in excess of 23 shots. So to a large extent, the dominance of the Lakers on the offensive boards was a reflection of their miserable 32.5 percent shooting clip.

However, the Celtics only gathered 32 defensive rebounds, meaning that of the 55 rebounds available after Lakers misses, the Lakers collected almost 42 percent of them. So not only did the Lakers get a lot of offensive rebounds, they got them at an stunning rate, and that doesn't depend on how many shots they missed. To give you an idea of just how stunning that is, the NBA league average is about 26 percent. The Lakers were more than half again as effective at getting offensive rebounds. By contrast, there were 38 rebounds available on the Celtics' offensive end, and they got only 8 of them, for an offensive rebounding rate of 21 percent, a bit lower than average.

That suggested the following little puzzle: All those offensive rebounds increased the Lakers' overall efficiency at the offensive end, by giving them extra shots at the basket on each possession. Can we express that increased efficiency in terms of shooting percentage—in effect, collapsing the two figures into one?

I believe we can. Suppose for the moment that we don't care about free throws, three-point shots, and all those aspects of scoring that in truth are rather important. We only care about the raw shooting percentage. The Lakers hit 0.325 of their shots. If their offensive rebounding rate was 0 percent, then the fraction of their shooting possessions (as opposed to possessions that end with a turnover, say) that they score on is 0.325.

However, in truth, they rebounded 0.42 of their misses. They miss 1 - 0.325 = 0.675 of the time, so out of all their shooting possessions, they end up with the ball again on 0.42 × 0.675 = 0.28 of the time. Then they'll score 0.28 × 0.325 = 0.09 of the time, and so on. If they miss, they can rebound again, which they'll do 0.28 × 0.675 × 0.42 of the time. And so on.

It's much more concise to put this symbolically, as follows:

Fraction of shooting possessions ending with a score = 0.325 + 0.675 × 0.42 × 0.325 + 0.675 × 0.42 × 0.675 × 0.42 × 0.325 + ...

Each time, there's an extra factor of 0.675 × 0.42, representing the Lakers missing and then picking up the rebound. Since this can happen an arbitrary number of times in a given possession, this equation can have an infinite number of terms (well, limited only by the length of the game). This is called a geometric series, and fortunately, there's a simple formula that allows you to calculate the sum without adding and multiplying an infinite number of terms. Therefore,

Fraction of shooting possessions ending with a score = 0.325 ÷ (1 - 0.675 × 0.42) = 0.45

That is to say, 45 percent of shooting possessions end in a score for the Lakers. Not to put too fine a point on it, that's still fairly awful. But not as awful as the original shooting percentage suggested.

But now, as I said, I'm going to try to combine the offensive rebounding and the shooting percentage into a single composite figure, by asking this question: Suppose the Lakers gathered only 26 percent of their misses as rebounds (the league average), instead of the 42 percent they actually gathered. How much better would their shooting have had to be in order to match that 45 percent per-possession efficiency? In symbolic terms, solve for x:

x ÷ (1 - (1- x) × 0.26) = 0.45

I'm not going to make you do that for homework; I'll just give you the answer: It turns out that x = 0.38. In other words, if the Lakers had crashed the offensive boards like an average team, they would have had to shoot 38 percent in order to score on 45 percent of their shooting possessions. Like I said: Bad, but not historically bad—not bad like 32.5 percent bad.

To put it another way, their tremendous offensive rebounding was worth 5.5 percentage points on their shooting. That's huge: 5.5 percentage points is usually worth about 10 points on the scoreboard by the end of the game.

We can turn this approach to the Celtics, too. They shot 41 percent (29 of 71), and picked up 21 percent of their offensive rebounds. That means that they ended 47 percent of their shooting possessions with scores:

0.41 ÷ (1 - 0.59 × 0.21) = 0.47

However, if they had just rebounded like an average team on their offensive end, they could have shot a bit worse and still matched that per-possession efficiency. Solve for y:

y ÷ (1 - (1- y) × 0.26) = 0.47

Again, I'll save you the algebra and give you a peek in the back of the book: y = 0.395. That is, if the Celtics were an average rebounding team, they would have achieved that efficiency by shooting just 39.5 percent. A bit better than the Lakers, but I think you'll agree that 39.5 to 38 is a lot closer than 41 to 32.5. Almost six times closer, even.

Now, the Lakers actually won, which means they must have done other things as well to get the win. For one, they turned the ball over somewhat less often, even with all the extra cracks at their offensive end: just 11 turnovers to Boston's 14. And the Lakers also visited the foul line more often (although some of those free throws were toward the end of the game, when the Celtics were fouling to stop the clock, and the Lakers shot poorly on their extra free throws just the same). Those two factors were enough to put the Lakers over the top. But the dominant factor in overcoming an awful shooting performance was their persistence in rebounding on the offensive end.

Say It, You Know You Want To

2010-06-04T15:43:00.000-07:00

I think it's safe to say the time has arrived.

You should be able to get a somewhat larger version by clicking on the above image.

Bending Over Backwards

2010-04-30T16:28:00.001-07:00

One of my favorite science bedtime stories (didn't you have those when you were a kid? or now, if you're still one?) involves the French physicist Prosper-René Blondlot (1849-1930), whose principal claim to fame, sadly, was a non-discovery.

In this particular case, Blondlot was working in his laboratory in the wake of a flush of discoveries concerning radioactivity and X-rays. Apparently, he was trying to polarize X-rays (a tricky task owing to their high frequency and short wavelength), and as part of his attempt he placed a spark gap in front of an X-ray beam. After a few experiments with this set-up, it seemed to him that the spark was brighter when the beam was on than when it was off.

He attributed this to a new form of radiation, which he called N-rays after his home town and university of Nancy. He may have been influenced by all the work on radioactivity and X-rays then going on, but at any rate, he set about immediately to investigate attributes of the new radiation. It appeared, he said, to be emanated by many objects, including the human body. It was refracted by prisms made from various metals, although these had to be specially treated in order to prevent them from radiating N-rays themselves.

It was all very interesting, and for some time, there was a burst of scientific activity on N-rays. The problem was, the N-rays themselves were very shy and retiring, and many physicists had trouble reproducing the results obtained by Blondlot and his staff. But Blondlot always maintained either that they had inferior equipment, or inferior perception.

You see, there was no objective recording of N-rays. All one had was a subtle brightening of a spark, which Blondlot and his colleagues were already prepared to see. To lend at least some notion of objectivity to the research, Blondlot took photographs of the sparks and other N-ray phenomena, but this merely replaced subjective judgment of a live spark with subjective judgment of a photograph. Means for measuring the light output were not sufficiently reliable or accurate yet to resolve the matter.

What did resolve the matter in the end was a visit to Nancy by the American physicist Robert Wood (1868-1955). Wood had tried himself to detect N-rays and had failed signally. Frustrated at his wasted efforts, and curious as to the differences between Blondlot's staff and equipment and his own, he travelled across the ocean to see for himself.

Wood had by this time in his career established himself as something of a debunker, a sort of turn-of-the-century James Randi. But Blondlot was no charlatan; on the contrary, he was firmly convinced of his own discovery. So he had no misgivings about demonstrating his N-rays before Wood and others. He darkened the laboratory (the better to see the increase in brightness). He set his aluminum prism on a platform to refract the N-rays, made some measurements, rotated the platform a bit, made some more measurements, and so on, all the while casually detecting the N-rays. For his own part, Wood could see nothing of what Blondlot was describing. But he kept quiet, waiting for the experiments to conclude.

When they did, and the lights were turned back on, there was general astonishment, for despite all the careful measurements on the refraction of N-rays, there was no aluminum prism sitting on the platform. Wood had, it turned out, pocketed the prism early on in the experiment. The entire time, Blondlot and his staff had been obtaining gradually changing measurements of an unchanging experimental set-up. That spelled the end, for all intents and purposes, of N-rays.

What happened? Intentional deception can be ruled out rather easily, since Blondlot would have known that careful experimentation would eventually disprove N-rays; it would have been a most temporary fame. Nor was he a shoddy scientist. Before the N-ray affair, he was known for having measured both the speed of light and the speed of electricity through wires, a task that had stymied others, and which established that the two were very close (though not quite the same).

Consensus today is that Blondlot had simply wanted to believe in N-rays, expected and wanted to see the predicted brightening, so much that he really did see it, sincerely. It has been suggested that he may have been motivated by nationalism; X-rays were discovered by the German physicist Wilhelm Roentgen, and Germany had recently taken a sizable chunk of France, so that Nancy was now uncomfortably close to the French-German border. But my own feeling is that it almost doesn't matter. At some point, the desire to see his discovery of N-rays vindicated became its own driving force.

The N-ray affair is often cited in support of what is, in my opinion, a central—perhaps even the central—insight of scientific discovery: The easiest person to fool is yourself. And fooling yourself is a necessary prelude to fooling others; charlatanry would have been easier to expose. Exhibit A in support of this position is the sad fact that although N-rays essentially died a hard death in 1904, Blondlot lived on for another quarter century, continued to be productive in science, and took his belief in the existence of N-rays to his death.

It is because it is so easy to fool oneself that science is, and must be, an essentially social activity. It is often said that in science, experimental data rules the day. That's overstating it a bit. Experimental data is indeed necessary for science to progress, but that data means little without scientific theory to organize it (and vice versa). It's not that the data is more important than the theory, but that it validates it, makes it less likely to fool yourself or anyone else. And there's a strong social pressure, within the scientific community, for one to bend over backwards in an attempt to subject one's theories to as much scrutiny as possible. It's that intense examination, which eliminates many theories but marks the ones that survive with an imprimatur of robustness, that distinguishes science from so many other human activities (ahem, politics?) and has made it one of the most successful endeavors of all.

Back, Slash, Back!

2010-04-14T10:01:00.001-07:00

Remember this post? Probably not, but now I have some company and/or vindication. Check out this xkcd comic, drawn by the redoubtable Randall Munroe.

Observe: Friends don't let friends say "backslash" in their URLs.

A Beginning, a Middle, and an End

2010-03-23T16:56:00.000-07:00

One thing I alluded to in my previous post, but never made entirely explicit, is the notion that there are distinct phases to a basketball game (and indeed to many sports competitions), which we might call—by analogy to chess—the opening, the midgame, and the endgame. The difference between the opening and the midgame is pretty ill-defined, and in my conception is based on the feeling that teams like to start games by trying out the various things they've worked on in practice, but within a general framework, and by the time they've gotten some ways into the game (after the first set of substitutions, say), they've got an idea for what's going to work in this game, and put it into practice in earnest. As I say, it's not a clear-cut distinction and we could argue endlessly (and, I think, pointlessly, though I'd be happy to be proved wrong) about where the exact division is.

But in my opinion, from a stats geek point of view, there is a clear-cut distinction between the midgame and the endgame. And the strategies are, empirically, different in the two parts of the game.

The whole objective of a basketball game (and in most games that involve points) is to outscore your opponent. And as basketball consists primarily of a sequence of alternating possessions, the goal should be to score more in each possession than your opponent does, by and large. That's why statistics such as points per possession are supplanting others like points per game, and rightly so. The former accounts for the fact that a game consists of a rather arbitrary but evenly matched number of possessions for each team, and the latter doesn't.

In fact, I'd argue that that objective—outscoring your opponent on a per-possession basis—is exactly the definition of the midgame. During this phase, which lasts for most of the game, you are trying to be as efficient as you can on the offensive end, while preventing your opponent from doing the same. Makes sense, doesn't it?

The question that you might be asking, though, is why this isn't your objective the entire game, why this is only the goal for the midgame. And the answer to that (you knew I had one coming, didn't you?) is that during practically any game, there comes a point where the actual scoring margin outweighs average efficiency.

Perhaps the simplest example is the decision about whether or not to shoot a two-point shot (a "deuce") or a three-point shot (a "trey"). Suppose the shooting percentage on the former is x percent, and on the latter is y. In the midgame, where all you're concerned about is the average number of points scored on the shot, you prefer the deuce if 2x > 3y, and you prefer the trey otherwise (ignoring offensive rebounding and the like, which we shouldn't do in a more extensive example).

In the endgame, however, it can be quite different. Suppose you're down two, and you have the ball with the shot clock off. You're going to hold for the final shot. The question is, what shot should that be?

If you shoot the deuce and you make it, you'll tie the game and go into overtime, where you'll win about half the time (studies apparently show that any apparent "skill" at winning overtime games is just a matter of small sample size). The winning probability is therefore x/2. On the other hand, if you shoot the trey and make it, you'll win the game outright, with probability y. So in this case, in the endgame, you prefer the deuce only if x > 2y (a strictly stronger condition than in the midgame), and you prefer the trey otherwise. (And as the defensive team, you probably want to shift more of your attention to the three-point line than you would during the midgame.) The point of this little example is that your objective is shifted, from efficiency in the midgame, to winning probability in the endgame.

The next question: When does this shift take place?

There's no one right answer, but I think one place to start is one I mentioned in connection with a rule of thumb I came up with for determining when a game is mostly out of reach. (Not to put too fine a point on it, a fellow by the name of Bill James also came up with the same rule.) To first order, I think, that same epoch in the game is where the switch between midgame and endgame happens (or "ought" to happen). After that point, the team that's trailing tries tactics that are not the most efficient (and therefore wouldn't be used during the midgame) but nevertheless maximize one's chances of winning the game; the team that's ahead plays to prevent their opponents from utilizing their preferred endgame tactics.

There's a bit of a catch, though, in that my rule (OK, Bill James's and my rule), strictly speaking, applies only to evenly matched teams. For the most part, that's not a stretch in the NBA, but you could imagine a game between an NBA team and a college team, even a very good college team. If both teams just try to be as efficient as they can, the NBA team will blow out the college team. In order to win, the college team would have to play their endgame practically from the opening jump, by employing some kind of gimmick, such as a non-stop trapping defense. Lest you think this is some kind of merely theoretical possibility, such a ploy has been tried in some circles, to some success.

And it likely has some statistical validity, for inferior teams can generally win only by introducing more chaos into the game (in the non-technical sense), which increases scoring variance. And there's no question gimmicks usually do that. Most of the time, they still won't work, but they'll give you a puncher's chance.

What's the point, in the end? As a kind of pie-in-the-sky proposal, since the objectives in the various phases are different, analyze them differently. Collect or synthesize different statistics for them. And maybe, as a result, you learn something new about why some teams can finish, and others can't.

Unifying Statistics

2010-03-11T15:33:00.000-08:00

As a sometime scientist, I love to unify things—that is, discover that two things that look completely different are actually intimately related at some abstract level. Without unification, science is largely stamp collecting, to paraphrase Ernest Rutherford. (Actually, he said that all science is either physics or stamp collecting, but I like to think that by "physics," he really meant unification, so it's all the same.)

The state of basketball statistics is one of substantial disunion. The box score is a hodgepodge of parameters with little or nothing tying them together. Points, rebounds, assists, steals, blocks, turnovers, fouls, etc.: These all clearly have some role to play in a team's overall goal—to outscore its opponent—but comparing one to another is impossible from those statistics alone. It would be useful if all of these aspects of performance could be put on equal footing. That would enable a proper assessment of the relative importance of the box score statistics.

Maybe, even, it would enable something else: That "equal footing" might just be able to stand on its own two feet as an independent statistic.

This thought grew out of a couple of recent posts I found on ESPN's TrueHoop blog. One was Henry Abbott's take on Kobe Bryant's crunch-time performance, which by subjective standards has been through the roof this year, but certainly (one would think) well above the average in any year, given his long history of hitting game winners. By most objective quantifiers thus far, however, Kobe is human—a good, but by no means great, clutch player. Abbott has a fair point to make against these quantifiers: His pedestrian shooting percentage at the ends of games might not be an indicator of substandard crunch-time shooting, but that his skill allows him to fight his way to shots that lesser players would never even be able to take. The same shots that lower his endgame shooting percentage (but which give his team a puncher's chance to win) are ones that never end up in the box score at all for other players.

Abbott's solution to this statistical problem is to find video of any situation where big-time players have the ball in crunch time, whether they hit, miss, or even fail to get a shot off at all, and watch it all. That certainly would give a better visceral idea of how stars perform at the ends of games, but it doesn't quite help in quantifying endgame performance.

The second post was an examination on Hardwood Paroxysm of a new way to view assists. In the box score, all assists are created equal, whether they lead to a highly contested three that just happened to swish through, or to an automatic, wide open dunk. Tom Haberstroh's suggestion is to weight those assists based on the expected scoring from the shot. So an assist to a dunk that scores 60 percent of the time would be worth 1.2, while one to a long deuce that scores 40 percent of the time would be worth 0.8, and one that goes to a wide open trey that scores 35 percent of the time would be worth 1.05. And so on.

My immediate thought on this proposal was that it sort of leaves unsuccessful attempted assists out in the cold. Suppose Chris Paul puts the ball on a dime to David West at the rim ten times throughout the course of a game, and West scores four times on those passes. (We'll assume for the sake of simplicity that he never gets fouled on these.) By the traditional count, CP3 gets 4 assists. By Haberstroh's count, he gets 4 times 1.2, or 4.8 adjusted assists. He gets a boost for having made West's job easier; West just didn't make very many of them. But why should Paul get penalized for West's misses? There was, plausibly, no real difference between the passes that led to scores and the ones that led to misses. Shouldn't they all count the same?

My not-so-immediate thought was that one could unify all this by putting it on a consistent statistical foundation. The foundation? Expected scoring at the beginning of any usage, where a usage is the period of time during which the ball is in a player's possession. Put aside, for the moment, all notions of personal points, assists, rebounds, etc. Define a usage to start when a player gains possession of a ball. He can optionally dribble it for some period of time. That usage ends when he releases the ball, which is either a shot (and goes in or it doesn't, in which case it ends with either defensive or offensive possession), a pass to a teammate, or a turnover. There are some interesting corner cases to deal with, but let's ignore that for the sake of discussion.

The statistic I'm proposing is, what is the expected points scored on this possession when a player starts his usage, and what is the expected points scored on the possession when he ends it? The difference between those two is a measure of his offensive value for that usage.

Example: Chris Paul dribbles the ball up court, with everybody already set in a halfcourt stance. In this scenario, the Hornets score, let's say, 0.8 points per possession on average. (Lower than their typical points per possession because all the high-value transition points are eliminated.) He dribbles around, and locates David West open underneath the basket, and gets the ball to him, whereupon the Hornets expected scoring at this juncture is 1.5 points. (Not exactly 2.0 because maybe he geeks the dunk, gets fouled, or whatever.) Let's suppose West actually does score the basket. The ledger for this possession is as follows:

Initial expected scoring: 0.8

Increment by Chris Paul: +0.7

Increment by David West: +0.5

Actual score: 2.0

Let's take another, somewhat more complicated case. Jason Williams comes up the floor in semi-transition. The Magic's expected score in this situation is, let's say, 1.1 points per possession. He dribbles around for a few seconds, however, and doesn't locate anything easy, so he pulls the ball back out and passes it to Vince Carter on the left wing with 16 seconds left on the shot clock. Williams hasn't done anything terribly negative with the ball (no turnover), but he hasn't broken anyone down, and in the meantime he's frittered away 8 seconds, and that lowers the expected score for the possession to 0.7 points. Vince shot fakes a few times, then takes it toward the baseline, drawing a few defenders to him, and then passes to Dwight Howard in the lane. Doing so increases the Magic's expected score up to 1.2 points. Howard dribbles left, fakes, goes back to his right, then tosses up a right hand hook that bounces off the rim and is rebounded by the other team. Final score on this possession is, of course, 0.0 points. So the ledger looks like this:

Initial expected scoring: 1.1

Increment by Jason Williams: -0.4

Increment by Vince Carter: +0.5

Increment by Dwight Howard: -1.2

Actual score: 0.0

On average, the initial expected scoring equals the actual score, so the typical player would score an average increment of 0.0. (For instance, suppose that 60 percent of the time, Howard makes that shot and scores an increment of 0.8; then, 40 percent of the time, he misses it and scores an increment of -1.2. Those two balance each other out exactly.) Higher is better, naturally, and lower is worse. This approach dispenses with the coarse categorization of basketball actions into scores, turnovers, assists, rebounds, and non-box-score actions, and assesses every single usage in terms of its contribution to the final score. I think it would be much more representative of everybody's activity. (One thing that is left out: screens.) One could also rate defense this way, to a certain extent, although zone defenses and double teams definitely make things challenging.

The drawback is that it's tremendously more work to encode all this information about the game. But diagnostically it might be worth it for teams to pay someone to do it; if you could figure out what a player is doing when his increment is 0.4 lower than average, that'd be very useful information. One benefit to this approach is that it only cares about what happens when the ball changes hands. Whatever a player does throughout his usage can be discarded as far as this statistic is concerned, so that would reduce the burden of encoding information.

The application to crunch-time shooting? I think it's pretty obvious. You've got 3.4 seconds left, down two, inbounding the ball 40 feet from the basket. In this case, you're in the endgame, not the midgame, so your objective is not to maximize scoring, but to maximize chance of winning. (A two-pointer is better than a three-pointer in midgame if it succeeds more than one and a half times as often, but it's only better in a two-point endgame if it succeeds about twice as often.) When you start this possession, your probability of winning is, let's say, 0.15. You get the ball, and you can the trey. Your actual winning probability is 1.0 (you won the game). Your win increment is therefore +0.85. If you had missed it, it'd been -0.15. So, when the situation looks dire, success is rewarded much more than failure is penalized.

Now, on the other hand, suppose you went for the deuce. If you miss it, the winning probability still goes to 0.0 and the increment is -0.15, but if you make it, the increment is only +0.35 (assuming you have a 50 percent chance of winning in OT). You've improved matters significantly, but you still haven't won the game. By this analysis, the cold-blooded assassin quality that Kobe Bryant supposedly personifies is not only bravado, but potentially sound tactical thinking, and this aspect would be captured by compiling expected win increments.

You could even go so far as to assess the impact on winning the title (much as Hollinger's playoff calculator does). By that metric, LeBron's fadeaway three against Hedo Turkoglu in Game 2 of last season's ECF was an absolute monster. Assuming that the Cavaliers would have been even money against the Lakers in the NBA Finals, that shot (which took the Cavaliers from at best a 0.1 win to a 1.0 win) was worth in the neighborhood of 0.1 to 0.2 of a title, an incredible value for a pre-Finals make. The fact that the Cavaliers did not go on to even make the Finals is immaterial in this valuation, as it couldn't have been known at the time. On the other side of the balance sheet would be Frank Selvy's miss at the end of regulation in Game 7 of the 1962 Finals, which ended up being worth an increment of about -0.2 or -0.3 of a title, as instead of winning the title outright on the shot, the Lakers had to go on to play OT, where they eventually lost.

The Suspension of Belief

2010-01-22T14:02:00.000-08:00

You may think I've mistitled that, but no, not really. Suppose I put to you two ways to say a common sentiment:

All that glitters is not gold.
Not all that glitters is gold.

Now, put aside all notions of poetic rhythm or provenance. (Or that the original version in Shakespeare's Merchant of Venice had "glisters" instead of "glitters." The former comes from Dutch, while the latter comes from Norse. In our day, the Norse version has entirely displaced the Dutch version, but in Shakespeare's day, they both had currency. Or at least so Shakespeare would have us believe.) Does either of these seem "righter" to you than the other?

I've put little quizlets like this to various people and they seem to fall mostly into two groups. One group of people can't see anything at all to recommend one over the other. Moreover, when the particular distinguishing feature is pointed out, they either don't see it or can't see why anyone would care. (You might, if you fall into this group, see if you can figure out before reading on what this distinguishing feature is, if you don't already know.)

The second group, of course, sees a logical distinction between the two and what's more, they're irritated that there's a mismatch between intent and wording. What's still more, they're irritated that the first group doesn't acknowledge this. To this group, the above two sentences are logically equivalent to the following:

All glittery things are non-gold.
Some glittery things are non-gold.

A quick glance at the script for Merchant of Venice indicates that Shakespeare chose the first wording ("All that glitters...") but his meaning is clearly the second. Does this bother you?

OK, that's not really all that important, as we all know what Shakespeare meant. Here's another one:

I don't believe we have a coherent plan for the Middle East.
I believe we don't have a coherent plan for the Middle East.

Obviously, when it's presented this baldly, it's clear what the difference between this two (especially, I hope, in light of the previous example), but I can't count the number of times that people have interpreted #1 (or minor variations thereof) as #2. And honestly, I don't think it's because they can't think logically. I think it's because they're impatient with disbelief.

Nowhere is this more evident than in politics. It's practically a cliché to demand politicians give their position on some issue or another, to the point that it's considered a weakness if they can't immediately spit one out. While I'm all for politicians being prepared for new situations (and as a by-product, for questions from the press), is having a response for all such questions really preferable to being able to suspend belief when the situation warrants? We've seen the dangers that feigned certainty can bring. And it's not as though suspension of belief necessarily means suspension of action. We can act rationally on uncertainty just as well as we can act on strong belief.

As prominent as it is in politics, though, this rejection of uncertainty permeates our whole world, including science, where it has no business. Political truths may last for a generation or two (think about how long the Democratic party has been on the side of civil rights), but scientific truths, once verified, last essentially for eternity, subject only to occasional refinement. Given that, what's the rush to judgment? Why not suspend belief until we know for sure? Impatience with uncertainty is fine as long as it motivates us to reduce it, but not if it forces belief before we're ready.

Cutting Your Losses

2010-01-11T15:24:00.001-08:00

I was standing at the vending machine at work today, buying some chips with lots of small coins (nickels and dimes). And as I often do, I carefully inserted the nickels first, then the dimes; if I had used any quarters, they'd have come last.

You may—assuming you've read this far—wondered why this is. To be fair, having done this for a long time, I wondered myself for a moment. And then I remembered.

See, when I first started doing this, I was in college. I was living in the dorms. The dorms had vending machines, which were balky, much like anything in the dorms. They would, occasionally, find something objectionable about your change. They were even particular about the way you inserted your change; sometimes, it would take six or seven tries for you to get it to accept a specific dime. I would bring extra change just in case, if I had any, but sometimes even that would run out. So there I would be standing, with 45 cents that the machine was refusing to take, and more money back in the dorm room that I could try out on the Keeper of the Fizzies. But in order to get that money, I'd actually have to back to the dorm room. Away from the vending machine.

I'd run downstairs, get the change, run back upstairs, and hope that in the meantime, no dormitory Grinch had decided to get a 30-cent discount on his Coke.

Because, as it happens, sometimes they would. I'd get back and there would be no credit at all in the vending machine. You might suppose that Whoever It Was would at least leave the credit they had benefited from in change on the side, but noooooo.

That's when this business with inserting change in ascending order of value started. It was a way of cutting my losses. You might think that it would be simpler for me to just push the coin return and withdraw my change before heading downstairs, but in the first place, the coin return lever was balky, like everything else, and in the second place, it had often taken me lots of effort to get those coins in and I was reluctant to relinquish those hard-won gains.

Eventually, I managed to obtain a small dorm fridge and thereafter bought my drinks at the market. But this was before all that. Just the same, I continued my coin-sorting practice even to the present day, where (I daresay) my co-workers are far less likely to stiff me out of a handful of change than my dormmates were.

You know me, always looking for something mathy about the situation, so here's the question: Suppose that I only used n nickels and d dimes (no quarters), that I foolishly brought exact change, and that the vending machine refuses to take exactly one coin, randomly and uniformly selected from all the coins. On average, how much less money did I place at risk going nickels first than I did going dimes first?

The answer: The average reduction in risk was equal to the value of the nickels multiplied by the fraction of coins that were dimes.

I had thought to try to tie this story to something deeper, but I just can't bring myself to do it.

Square Roots, Lasers, and Mobilization

2009-12-11T10:18:00.000-08:00

I promised (threatened) that I would say more about square roots, and so I am. This is me, talking about square roots again. In typical fashion, though, I'm going to start with something else that will seem, for a time, completely unrelated.

Galileo, he of the telescope, the balls rolling down inclined planes (and probably not in actuality from the Tower of Pisa), the sotto voce thumbing of the nose at the Inquisition—Galileo also discovered, or more likely rediscovered, that pendulums mark out roughly even time, no matter how far they swing. It isn't perfectly even time, owing to friction and to the circular track of the pendulum bob (although both of those can be—and were—accounted for, starting with Huygens's employment of cycloid guides). But it's pretty close.

Since the pendulum keeps fairly even time, that must mean that if the pendulum swings in twice as big an arc, it must also be moving twice as fast, in order to keep beating out even time. Now, as it's defined in Newtonian physics, the kinetic energy of the pendulum bob—that is, the energy of the bob due to its motion—goes as the square of its velocity:

KE = ½ mv²

So, twice the arc, twice the velocity, four times the kinetic energy; three times the arc, three times the velocity, nine times the kinetic energy. And so on.

That swinging motion of the pendulum bob is an example of periodic or wave motion, so called by virtue of it swinging back and forth as a water wave swings up and down, if you were to watch it passing by a buoy. Wave motion is primarily characterized by two parameters: its frequency, which is how often it returns to its starting point; and its amplitude, which is how wide it swings. So the arc through which the pendulum bob swings is essentially its amplitude. (Actually, for historical reasons, the amplitude is defined as half of that arc, from the center point of the swing to either of its extremes, but this won't affect our discussion.) So we can say that the pendulum's energy is proportional to the square of its amplitude.

This turns out to be common to many different kinds of waves—including light waves. Light is a wave. (It's also a particle, in many ways, but we'll ignore that for now.) And being a wave, it has an amplitude, which is the extent to which the light oscillates. What is it that's oscillating, anyway? In the case of water waves, it's water, and in the case of sound, it's the molecules in the air. You can't have water waves without the water, and you can't have sound waves without the air; that's why sound doesn't travel in a vacuum. But light does travel in a vacuum, so what's waving the light, so to speak? Well, the answer is that the light itself is waving, or less opaquely (heh heh), the electromagnetic fields that permeate space are waving.

In any event, like other waves, light waves also carry energy that is proportional to the square of the light's amplitude. If you double the amplitude, you quadruple the energy; triple the amplitude, and the energy goes up nine-fold. And so on.

How would light's amplitude be doubled, though? You might imagine that if you put two flashlights, the amplitude of the two together would be twice that of each individual flashlight, and the combined light output—the energy of the two together—would be four times that of each flashlight. But I think, intuitively, we know this to be false, that the combination is only twice as bright as each flashlight. And if you measure the light carefully, in a dark room, this turns out to be perfectly true.

What happened? Light waves, like other waves, have a secondary property, called phase. Two waves of the same frequency are said to be in phase if they swing in the same "direction" (in some not altogether well-defined sense); imagine two pendulums swinging in unison, so that when one swings left, the other does, too. They are out of phase if when one swings left, the other swings right, and vice versa. Or, they may be partly in phase, partly out of phase.

When you combine two light waves of the same frequency and the same amplitude, you get for all intents and purposes a single wave that is the two original waves added together. If they're in phase, the peaks get peakier and the valleys get, err, valleyier, and the amplitude of the waves is in fact doubled. On the other hand, if they're out of phase, the peaks of one get cancelled out by the valleys of the other (and vice versa), and the resultant wave has no amplitude at all.

More typically, though, the two waves are partly in phase and partly out of phase, and the resulting wave's amplitude is somewhere in between zero and two times the original. On average, one can show that the amplitude is the original times √2 . What's more, if you add three waves together at random phases, the amplitude of the sum is the original times √3 . And so on. Aha, the square root!

And since the energy of the final wave is the square of the amplitude, what comes out has two, three, or whatever times the original energy. Which is, of course, exactly what you'd expect. And good thing, too, because if it came out otherwise, we'd have a violation of the conservation of energy. Clearly, it takes n times as much energy to run n flashlights as it does to run one, and if their combined output were something other than n times the original, we'd have to seriously rethink our physics.

You might wonder if there isn't way to get the waves to line up properly in phase so that the amplitudes do add up in the normal way, and you get a dramatic ramp up in energy. And there is; it's called a laser. A laser essentially gets n individual photons to line up in phase so that what comes out is a sort of super-photon (or super-wave, equivalently) with n² times the energy of any of the input photons. The physics-saving catch is that it takes more energy to line up, or lase, the light than you get as a result.

Nevertheless, that single photon or wave, coordinated as it is, can do things that you couldn't do with the individual photons separately. You can shine a bunch of flashlights at your eye and nothing will happen, other than a rather annoying afterimage and perhaps a headache. But even a modest laser can be used to reshape your cornea and render your eyeglasses superfluous. Of course, it should go without saying that it's not such a great idea to randomly shine lasers into your eye!

Or out, for that matter.

I see in this a kind of metaphor for human nature, and I hasten to say it's only that; as far as I know, one can't really take this and apply it rigorously in any scientific sense. But I think it's a useful metaphor all the same. I like to say that religion, among other things, is a laser of people. What on earth do I mean by that? A single human being can do a certain amount of work (in physics, work is defined as energy applied in furtherance of a force). What happens if you get two human beings together? Well, if they work against each other—if they're out of phase, in other words—less work gets done. Maybe none, if they spend all their time squabbling. Even if they're not exactly out of phase, if they're not particularly coordinated, their combined output is rather less than you might think, like the drunkard making slow and halting progress homeward because he can't put one foot directly in front of the other.

On the other hand, if they cooperate—if they're in phase—they can do twice the work. In fact, maybe they can get even more done, for there's no arguing that a coordinated combination of two people can do things that each individual person couldn't do, even adding their results together. Two people can erect a wall, for instance, that neither person could individually. Maybe, in some sense, those two people can do what it would take four people, working randomly, to achieve. And perhaps three coordinated people can do what it would take nine randomly working people to. And so on.

But it's pretty straightforward to get two or three people to work together, if they're of a mind to. But what about a hundred, or a thousand, or a million? That's where ideologies can be enormously effective; through them, a thousand can achieve what would otherwise require a million. And there may be no ideology better suited for the purpose than religion, although other ideologies—sociological, fiscal, even autocratical—may suffice. That's not to say that all that these various ideologies achieve is beneficial: for every great liberation, there may be a dozen pogroms. But they are part and parcel of a society's capacity for achievement; without them, we get only as far as a drunkard's walk will take us.

Square Roots and Great Comebacks

2009-12-07T15:29:00.000-08:00

From the time I learned about them, I've been fascinated (probably to an unseemly amount) by the square root. I remember reading about a method for calculating square roots by long hand. There's no point, really; we have calculators to do that for us. (If you have some spare time and you enjoy this sort of thing, see if you can figure out the algorithm from the example at left.)

What use are square roots, anyway, aside from solving math problems about the diameters of circular lawns? (Have you ever seen any of those? They must encircle those conical swimming pools we dealt with in calculus class.) Here's one use: They can tell you when how big a lead your favorite basketball team needs to be secure in a win.

A few years ago, I derived a rule for determining when a lead was safe in a basketball game—specifically, an NBA game. (It matters, because the shot clock is different between an NBA game and a WNBA game and a NCAA men's game and a NCAA women's game.) You take the square root of the number of seconds left, and add three. For instance, if there's 3:45 left in the game, that's 225 seconds. Square root of 225 is 15, and you add 3, so an 18-point lead is pretty darned safe with 3:45 left. The "add 3" is for a trey at the buzzer. Go ask the Miami Heat about that 'round about now.

Pretty keen, huh? Although—not to put too fine a point on it—well-known sports statistician Bill James also came up with this very same rule. We'll call it independent discovery, at least on my part. I have no idea whether James stole it from me. Give him the benefit of the doubt, though.

But why? Why should this rule work? Why isn't it just the time remaining divided by some rate at which the team that's behind catches up? If a team can make up a 15 points in 225 seconds and then cap that with a trey to make up the 18, why can't it make up 33 points in 7:30? Or 63 points in 15:00?

And the sort-of answer to that is, it can. It's just terribly unlikely. Of course, it's already unlikely that a team can make up 15 points in 3:45, but it's still in the realm of possibility. Asking a team to do that twice in a row is just too much. If it was 100 to 1 against doing it once, doing it twice in a row would be 10,000 to 1 against. On the other hand, making up the same 15 points in twice the time is obviously easier. So in twice the time (7:30, natch), you should be able to make up some deficit in between. According to both me and Bill James—and honestly, are you going to go against both of us?—that deficit is 15 times the square root of 2. That's about 21, and if you add the 3 at the end it makes it 24.

Where on earth does this come from? One place is the drunkard's walk, otherwise known as the random walk (but I think "drunkard's walk" is more evocative). In this mathematical scenario, the eponymous drunkard starts off at some placemark—a lamppost, say. Each moment in time, he takes a step, but in a completely random direction. Might be in the same direction as the last step, might be in the opposite direction, might be anything. So after a bunch of steps, he might end up back at the lamppost where he started...or he might be home.

Odds are, though, he'll be at some intermediate distance. How far from the lamppost? Well, the first step is going to take him one step away for sure. We'll represent this by saying that d(1) = 1, where d(t) is the distance of the drunkard from the lamppost at time t. OK, now what about d(2)? Before that second step, he's one step away from the lamppost. His second step might take him two steps away, if he walks in the same direction, or zero steps away, if he walks in the opposite direction (back toward the lamppost). On average, though, he'll walk in some intermediate direction: let's say, perpendicular to his current progress from the lamppost. The Pythagorean theorem says then that

[d(2)]² = [d(1)]² + 1² = 1² + 1² = 1 + 1 = 2

or, in other words, d(2) = √2. We can go further. We've already got two examples where d(t) = √t and we'd like to get more. To do that, we'll use a process called induction. Suppose that you have a value of t for which d(t) = √t ; we'll now try to show that d(t) = √(t + 1) . Using the same argument as before—that the drunkard walks in some intermediate direction—we get

[d(t + 1)]² = [d(t)]² + 1² = [√t]² + 1² = t + 1

and then we directly get d(t + 1) = √(t + 1) . So as long as we can find a t where d(t) = √t , we're set; it's true for all greater values of t. But we already have such a value: t = 1! (And t = 2, for that matter.) It turns out, then, that the drunkard's walk, after time t, takes him a distance √t away from the lamppost.

Now, a couple of things. First, this isn't anything like a rigorous demonstration of the square root property of the drunkard's walk. You can look that up if you like. But if you work at it a little, it gives you an inkling of the intuition behind it. Secondly, though, and here we're back on track a bit: What has all this got to do with basketball games?

A basketball game is an alternating sequence of possessions. In each possession, the team with the ball is of course trying to score, and the other team is of course trying to prevent it from scoring. When the ball changes hands, the roles are reversed. In each individual possession, the effect on the score is biased: Only the team with the ball can score, usually. But in each pair of possessions, that bias cancels out, since both teams get a chance with the ball. The margin in the game can move in any direction—just like the drunkard's walk.

If the drunkard starts off 50 steps from home, he could conceivably get home in just 50 steps. But it's ridiculously unlikely: Each of those 50 steps would have to be in exactly the right direction. The square root property tells us he'll probably be just a bit over 7 steps from the lamppost; it would take 2500 steps to get him, on average, 50 steps from his starting point. After those 2500 steps, is he guaranteed to be home? Nope. He still has to be walking in the right direction. But it's at least plausible now.

In the same way, a basketball team that's down 18 points could conceivably make that up by scoring six three-pointers in a row while holding their opponents scoreless. If they did that by fouling and their opponents obliged by missing all of their free throws, the whole deficit could be made up in half a minute or so. But that's as unlikely as the drunkard walking 50 steps in exactly the right direction. Instead, a team will make up its deficit in halting fashion, sometimes making up three points, but other times giving up a point, or staying even, in any particular pair of possessions. The drunkard's walk, in other words, and that's why the square root rules great combacks.

I was going to follow this up with a discussion of sociology and mobilizing people, but this post is getting long (see, I do notice it!) and I'll defer that till next time.

Basketball Math Fail

2009-12-02T11:11:00.000-08:00

Today's miscreant: The Boston Globe's Celtics Notebook. The offending paragraph reads:

Rasheed Wallace has eight technical fouls in 18 games, which would equate to 36 over a full season. That number is astronomical, of course, especially since the NBA suspends players one game for each technical after the 16th.

First of all, the NBA does no such thing. It does suspend players one game for every other technical, starting with the 16th. (See the NBA Rule Book, Rule 12, Section VII.) But that's not the math fail in this instance. The math fail is figuring that Wallace would get 36 technicals over a full 82-game season, when by their own admission, he wouldn't even play 82 games because of the suspensions for all those technicals.

So how many technicals would he get, if he were to get them at the same rate for the rest of the season, and he didn't miss any games to injury or other reasons besides the suspensions from the technicals?

At the current rate, Wallace would pick up his 16th technical in his 36th game, meaning he would be suspended for the team's 37th game. (We'll assume that Wallace doesn't appeal any of his technicals or suspensions.) He would then pick up four technicals in every nine games he played in thereafter. For the sake of argument, let's say he picks up the even-numbered technicals (16th, 18th, 20th, etc.) in the fifth and ninth game of every cycle of nine games he plays in. Since each of those technicals would carry with them a one-game suspension, these cycles would actually span 11 games for the Celtics.

As a result, Wallace would pick up those even-numbered technicals in the Celtics' 42nd, 47th, 53rd, 58th, 64th, 69th, 75th, and 80th games, in each case being suspended for the next game. That technical in the 80th game—his 32nd—would be his last, since he'd only be able to play in one additional game, and we'll charitably assume that he wouldn't get called for a technical in that one. So he'd get suspended for nine games in all, drawing 32 technicals in 73 games.

Analogies for Better or for Worse

2009-12-01T15:57:00.000-08:00

Douglas Hofstadter wrote about the relationship between analogies and intelligence in the September 1981 installment of his Scientific American column series Metamagical Themas, entitled "Analogies and Roles in Human and Machine Thinking." His central point is that being able to see similarities between different situations and to capitalize on those similarities to make predictions is core to the nature of human intelligence (and by extension, to fruitful research on machine intelligence as well). "Being attuned to vague resemblances," he writes, "is the hallmark of intelligence, for better or for worse."

As if to highlight the "worse" side of the ledger, somewhere toward the middle of the column, he discusses the pitfalls of taking analogies too far. Ultimately, situations don't map perfectly onto each other, and the greater the demands placed on any given analogy, the more likely it will stretch so far until it snaps.

Analogies are particularly useful for teaching purposes. Students seem often to learn something better when it is explained in terms of something they already know. We might learn about electrons orbiting an atomic nucleus by analogy with planets orbiting the Sun, for instance. To the extent that principles in one domain apply to the other, we can understand and explain behaviors in the new, unfamiliar domain in terms of the old, familiar one.

There are dangers to this path to learning, though. The famed Caltech physicist Richard Feynman—surely one of the great physics teachers of all time—was extremely conscientious when it came to teaching by analogy. He avoided analogies that he found misleading or circular. It might be natural to think of electromagnetism as being mediated by imaginary "rubber bands," he said, but in the first place, rubber bands draw things together more the further apart they get, whereas electromagnetism gets weaker with distance, and secondly, rubber bands themselves work through electromagnetism interactions at the molecular level, so any understanding students derived through this analogy must needs be circular.

Care must be taken, too, not to stretch the analogy beyond its limits. The fact is that electrons don't orbit the nucleus in neat circles (or even ellipses) like planets orbiting the Sun. If we study further, we find that although planets can apparently orbit the Sun at any distance whatsoever, electrons are constrained to orbit the nucleus only at specific distances, which we can characterize at those distances which allow an integral number of electron waves to circle the nucleus. If we study still further, we find that electrons don't travel in any kind of orbit at all, but instead can be found at any location around the nucleus according to a probability distribution (or, equivalently, are simultaneously at all different points according to that distribution—at least prior to observation).

The problem is that analogies are so darned appealing. The good ones yield correct answers to our questions so often that we lose track of where the limits of the analogies are, or even that there are any. We simply trust the analogies, often to our detriment. It's tempting to understand the budgetary situation of, say, the United States in relation to our personal budget; after all, there are many similar concepts and relationships: income, expenses, debt, balance, and so forth. It's tempting, but it's often misleading. But because we do understand many things correctly using that analogy, we become overconfident in areas where the analogy was never going to hold water.

My pet peeve in this regard is the rubber sheet analogy for general relativity. Given that general relativity was one of the major developments of 20th-century physics, you'd expect that there'd be significant time spent in explaining it to the lay public. I mean, even people who only vaguely have a notion of what physics is about have heard of Albert Einstein and "warped space."

Gravity is everywhere; we feel its effects all the time. And we've sort of internalized the Newtonian theory of gravity, which is that any two particles exert a gravitational force on each other, no matter how far apart they are; although the degree of force drops off quite rapidly with distance, it never quite shrinks down to zero. We've internalized it so well that we hardly ever wonder how that force is mediated. How does that force get exerted across all that distance? By the Newtonian theory, I wiggle my finger here, and my finger's gravitational influence on the most distant galaxy, however faint, oscillates with the same frequency as my wiggling finger. Newton himself felt this conundrum most keenly, never mind his insistence that he did not "feign hypotheses."

Einstein's general theory of relativity ostensibly resolves all of that. It posits space not simply as a theater in which gravitational interactions take place, but a physical, almost tangible thing that is affected by masses and in turn affects them. The usual term for this is curved space—a term that is justified in a mathematical sense but which is almost certain to mean nothing directly to anyone who isn't already a physicist. I imagine that the most common response is mute incomprehension.

So we explain what we mean by "curved space" by analogy. First of all, we should really be calling it "curved space-time," since in Einstein's theory time and space are interwoven almost irrevocably. With three dimensions of space and one of time—well, that's a lot of dimensions. People don't visualize four dimensions very well. So we abstract away two of them: one of the spatial dimensions, and the one time dimension, leaving two spatial dimensions. The one spatial dimension is OK, probably, but already there are problems. You've lost the one temporal dimension you have; it's possible that you might lose something essential there!

But we're pressing on. We lay down an infinite rubber sheet, typically marked with grid lines. We plop down a big heavy ball, like a bowling ball. This is the Sun, we are told. It bends or curves or warps space. Sure enough, the rubber sheet is seen to dimple significantly. Then, we roll a smaller ball around the bowling ball, and because of the warping caused by the bowling ball—err, Sun—the smaller ball (representing the Earth, say) sweeps around in a neat circular or elliptical orbit. Just like the real planets.

This is an enormously popular representation of general relativity; even Carl Sagan's Cosmos, my favorite science documentary series of all time, uses it. And yet, in my opinion, it's fatally flawed. In the first place, it's circular, just like Feynman's rubber bands. We're told that the effect of the Sun's gravity can be interpreted in terms of the Sun's warping of nearby space, by analogy with the warping of the rubber sheet caused by the bowling ball. But what is it that causes the bowling ball to warp the rubber sheet? Gravity itself! We can't rightly claim to understand gravity if gravity is involved in the explanation as well.

Even that would be excusable for pedagogical purposes if the analogy were actually accurate. But it's not. In all of the rubber-sheet depictions of general relativity I've seen, and I've seen quite a few, only one includes a disclaimer that demonstrates what's wrong with it—a little-known primer on relativity written by Lewis Carroll Epstein called, appropriately enough, Relativity Visualized. (I heartily recommend it.) He makes the following point: In space, there is no universally preferred direction up or down; those directions are only understood in reference to some gravitational field. So the rubber sheet analogy, if it's really right, should work just as well if you flip the rubber sheet upside down, so that the warp goes upward (like a volcano) rather than downward. After all, it's not supposed to be the bowling ball itself that makes the other ball go 'round, but the warp. But if you roll the smaller ball toward the volcano, what happens? As any miniature golfer knows, it certainly doesn't orbit the volcano; instead, it either goes into the volcano, or it veers away from it, never to return.

But even that's not the worst of it. The irony of this analogy is that even though it's not a very accurate depiction of general relativity, it's a dead-on match for Newtonian potential energy wells. That's right: This immensely popular analogy, which is supposed to highlight how general relativity differs from Newtonian gravity, is instead a much better illustration of the very theory general relativity was intended to supplant! I was so struck by this that I wrote up an exposition of general relativity for my astronomy Web site, which (on the off chance you've actually read this far) you can find here. In it, you'll find an analogy to general relativity which is hopefully understandable but hits much closer to the mark. (I even asked a physicist!)

But does anyone care? Nooooo, I'm sure we'll continue to see the rubber-sheet analogy trotted out at regular intervals on the Discovery Channel, with no disclaimer regarding its appropriateness.

Something to Do With Math, Right?

2009-10-22T16:18:00.000-07:00

In my last post, I mentioned that scoring differential has been shown to be a better predictor of future wins than even past wins are. What this referred to, specifically, is the so-called Pythagorean expectation (PE), a creation of baseball statistics guru Bill James. It's called that because of the form of the PE formula: If you let RS be runs scored by the team, and RA be runs scored against the team, then a good estimator for the winning percentage—at least in baseball—is

WP = RS² / (RS² + RA²)

So, for instance, if over the course of a season a team scores 800 runs, but only gives up 600, then the PE formula predicts that their winning percentage will be about 800² / (800² + 600²) = 0.640.

Actually, there's nothing magical about the exponent 2 in this formula; as it turns out, an exponent of 1.81 matches actual winning percentage better than 2 does. What I'd like to do in this post is say a few words (well, who are we kidding here, more than a few words) about where this exponent comes from, and an interesting correlation.

Baseball, like any sport, can be treated like a combination of strategy, tactics, and random events. The strategy and tactics represent those things that are under the control of the two teams, while the random events are things that are out of their control, such as where the baseball hits the bat, how it bounces off the grass, and so forth. Technically, as I've said before, these aren't actually random, but they happen so quickly that they're essentially random for our purposes; we can't perfectly predict how they'll go. All we can do is assign probabilities: e.g., such-and-such a player will hit it up in the air 57 percent of the time, on the ground 43 percent of the time, stuff like that.

As a result, the outcome of games aren't perfectly predictable, either; as they say, that's why they play the games. Again, we can assign probabilities—probabilities that a team scores so many runs, or gives up so many runs, or that they win or lose a particular game. The PE formula is an attempt to relate the probability distribution of runs scored and runs given up, to the probability distribution of winning and losing.

The probability distribution can only be specified mathematically, but we can get an inkling of how it works by sketching it out schematically.

In the diagram above, the horizontal axis measures runs given up, and the vertical axis measures runs scored. The diagonal dotted line represents the positions along which the two measures are equal, so if you're above that line, you win the game, and if you're below it, you lose the game.

The red blob depicts the probability distribution of runs scored and given up for a hypothetical team. Each point within the blob represents a possible game outcome. Games in the lower left are pitcher's duels, while those in the upper right are shootouts. Those in the other corners are games in which the team either blew out their opponent or were blown out themselves. Any outcome within the red blob is possible, but they're more likely to be clustered in the center of the blob, where it's a darker red. The particular way in which the games are clustered around that middle is known as the normal or Gaussian distribution. Such a distribution is predicted by something called the central limit theorem, and is also borne out by empirical studies.

From this diagram, we can estimate what the team's winning percentage is: It should be the fraction of all the red ink that shows up above the diagonal dotted line. Since the team scores, on average, a bit more than it gives up, more of the blob is above that line than below it, and their winning percentage should be somewhat above 0.500—say, 0.580, maybe. What Bill James found out was that if you compute the "red ink fraction" for a variety of different values of runs scored and runs given up, the results were essentially the same as those yielded by the formula given above.

Now, as it so happens, if you try to apply the same formula to, say, basketball, it doesn't work very well at all. Practically any team will end up with a predicted winning percentage between 0.450 and 0.550, and we know very well that isn't so: Usually there's at least one team over 0.750, and often times one over 0.800 (Cleveland did that this past season). The reason can be seen if we take a look at the corresponding "red ink" diagram for basketball.

Baseball scores runs, and basketball scores points, but the principle is the same. What isn't the same, however, is the degree of variation in the scores, relative to the total score. Basketball teams show much less variation in the number of points they score than baseball teams do. Basketball teams rarely score twice as much in one game as they do in any other; by comparison, baseball teams are occasionally shut out and occasionally score 10+ runs.

In consequence, a baseball team that scores 10 percent more runs than it gives up will still lose a fair number of games, because the variation in scores is much more than 10 percent a lot of the time. In contrast, a basketball team that scores 10 percent more points than it gives up will win a huge fraction of the time, because the variation in scoring is so much less. As you can see above, the red blob is in approximately the same place in both diagrams, but because the blob is smaller (less variation), practically all of the blob is now above the diagonal line, corresponding to a winning percentage of, oh, let's say 0.850.

This property can be addressed by using James's PE formula, but with a much higher exponent. Estimates vary as to how much higher, but the differences are relatively minor: Dean Oliver suggests using 14, whereas John Hollinger uses 16.5. Either of them will give a good prediction of the winning percentage of the applicable team.

It would be nice not to have to guess at the right exponent, though. So, since there seems to be a pretty obvious correlation between the size of the blob and the size of the exponent, I decided to investigate exactly what that correlation was. It seems likely that someone else has done it before, but a Web search didn't turn up any obvious results, so I'm sharing mine here.

To begin with, there's something else in statistics called the coefficient of variation, which basically gives in this case the size of the blob, relative to how far it is from either axis. In case you're following along on your own paper, it's defined as the ratio of the standard deviation of the distribution to the mean. So, in baseball, the c.v. is relatively large; and in basketball, it's relatively small.

What I did was to figure out, from numerical computations, what the "red ink" fraction was for various c.v.'s and scoring differentials, and to see if a formula of James's basic structure, with the right exponent, would fit those fractions. (My tool of choice was the free and open-source wxmaxima, in case you're interested.) They did, very well. In fact, I found it startling how well they fit, assuming that scoring was normally distributed. In most cases, the right exponent would fit winning percentages to within a tenth of a percent.

For instance, for a c.v. of 0.5, an exponent of 2.26 fit best. The numerical computation showed that a team that scored 20 percent more than it gave up would win 60.1 percent of the time; so did the formula. As the c.v. went down, the exponent went up, just as you would expect. The actual values:

c.v. = 0.5, exp = 2.26

c.v. = 0.3, exp = 3.78
c.v. = 0.2, exp = 5.67
c.v. = 0.1, exp = 11.7

I found these results startling: the product of c.v. and exp is almost constant, at about 1.134. (I propose calling this the Hell relation.) In other words, the right exponent is almost exactly inversely proportional to the c.v. of the scoring distribution. Therefore, we would predict that the c.v. of baseball games is 1.134/1.82, or 0.623; that of basketball would be 0.081 or 0.069, depending on whether you trust Oliver or Hollinger. I've heard that Houston Rockets GM Daryl Morey once determined an exponent of 2.34 for the NFL, which would correspond to a c.v. of 0.485.

Obviously, this is a consequence of the particular scoring model I used, but the normal distribution is broadly applicable to a lot of sports, most of which have games that are long enough to allow normalcy to show up. Given how well the basic structure of James's formula holds up, I suspect the underlying assumptions are fairly valid, although it would be interesting to see that verified.

EDIT: Here's an article from a statistics professor on just this very topic, with a rigorous derivation of the various formulae.

Adjusted Plus or Minus (More or Less)

2009-10-19T15:21:00.000-07:00

I spent some time a while back discussing PER and its limitations. Today I'll take a similar look at adjusted plus-minus, or APM.

One of the weaknesses of PER is that it's a rather arbitrary linear combination of basketball statistics. As I pointed out, one can come up with alternate combinations that put any number of players on top of the PER list. In math nerd terms, any player on the convex hull of the statistics space can end up on top, given the right PER formula. With as many dimensions in that space as there are component statistics, that could end up being a lot of players.

And anyway, the bottom line of the game is winning, and there's no clear evidence that maximizing team PER (however you define that) maximizes your chances of winning. (It must be emphasized, by the way, that that's all any statistical approach can do: maximize chances. Basketball may be played on the floor, not on a piece of paper, but the small contingencies that lead to winning or losing are so complex and so numerous that the only thing we can do with them is treat them as essentially random events. Nothing is ever really certain in any practical sense.)

APM is a completely different approach to player assessment that attempts to remedy this weakness. Its purpose is to determine how much a player contributes to his team's scoring margin versus the opponents, which has been shown, to varying degrees of certainty, to be a good predictor of future winning percentage—better even than past winning percentage. It does this by calculating how much the team outscores its opponents with that player on the court. There's a few ways we could do this (just as there are multiple ways to define PER); I'll just be discussing one of them.

As its name implies, APM is an adjusted form of raw plus-minus, which we can call RPM for the moment. The difference between the two can best be illustrated using a simplified example. Suppose some Lakers players (Kobe, Pau, and Lamar) are participating in a two-on-two tournament, with substitutes allowed. Games are 48 minutes long. Let's say that in a particular game, Kobe and Pau open the game and play for 16 minutes, outscoring the opponents by 8. Pau and Lamar play the next 16 minutes, outscoring the opponents by just 2. Finally, Kobe and Lamar close the last 16 minutes, and outscore the opponents by 4. For the sake of simplicity, let's assume for now that the opponents have no sub and play the entire game with the same two players.

During the 32 minutes that Kobe's on the floor, his team outscores the opponents by a total of 12 points. Over a full 48-minute game, that would work out to a RPM of +18 (a 48-minute game is half again as long as Kobe's 32 minutes). Similarly, Pau's 48-minute RPM is +15, and Lamar's is +9.

However, you might ask, for instance, how much of Pau's RPM is due to his own contribution, and how much is due to sharing the court with Kobe? This is the question that APM seeks to answer. It attempts to account for the teammates one plays with, as well as the opponents one plays against (though we're keeping those constant for now).

One might compute the APMs of the three players as follows: Let Kobe's, Pau's, and Lamar's APM be represented by k, p, and l, respectively. From the first 16 minutes, we extrapolate that if Kobe and Pau played the entire game, they'd have outscored the opponents by 24 points. That could mean that both players have APMs of +24, or perhaps Kobe's is +28 and Pau's is +20, or maybe vice versa. There's not enough information to determine for sure. However, at any rate, they add up to 48:

k + p = 48

Similarly, we can write for the other two 16-minute segments

p + l = 12
k + l = 24

I'm not going to go through the gory algebra (I'm assuming you can do that yourself if you've read this far), but these three equations in three variables yield a unique solution: k = +30, p = +18, l = - 6. By way of interpretation, if you had two Kobes play against two average players for an entire game, the Kobes would win by 30 points. (Various versions of APM scale this so that you can just add up the APMs to determine the expected final winning margin. There's no significant difference between this and what we derived; they would just differ by a constant factor—the number of players—so that the scaled APMs would be +15, +9, and - 3, respectively.)

Note that nowhere in all of this computing did we say anything about scoring, rebounds, assists, steals, blocks, fouls, etc.—any of the statistics that make up aggregate parameters like PER. APM is entirely agnostic about what makes players valuable to their team; it simply measures that value. In a way, this is useful, because it completely short-circuits any assumptions about what makes players valuable in general; on the other hand, it sure would help if you knew why your player was valuable. APM can't really answer that. It is, in a very real sense, the holistic yin to PER's reductionistic yang.

Incidentally: What happens if the opponents do use different line-ups? Suppose the Lakers are playing the Magic, with Dwight Howard, Vince Carter, and Rashard Lewis. We'd use d, v, and r to represent their APMs, and assuming they played those line-ups in the same 16-minute segments as the Lakers did, we'd write out something like the following equations:

(k + p) - (d + v) = 48
(p + l) - (v + r) = 12
(k + l) - (d + r) = 24

Note that we now have three equations in six variables, which means that the scenario is said to be underdetermined: there won't be a unique solution to the equations, but multiple solutions (an infinite number, in fact). In general, there will be some kind of mathematical mismatch like this: There are as many variables as players, but as many equations as there are matchups, and those usually won't be equal. Since the number of matchups is larger than the number of players, though, you'll typically have overdetermined scenarios: there won't be any exact solutions at all; any combination of numbers will violate one equation or another.

That sounds bad, but in a sense, it's better than being underdetermined, because we can use statistical methods to determine the best near-solution to the equations—"best" in this case defined by how little the equations are violated as a whole. We can justify this by observing that players aren't robots—their performance varies up and down over the course of a game or a season—so some error in the equations is expected. Typically, the statistical method used is some form of linear regression, which is the same method used to identify likely correlations in all manner of scientific studies. In general, such methods work very well indeed.

I am, however, going to go off the reservation a little: I'm claiming that it might not work so well for basketball.

The key sticking point is hinted at by that name, linear regression, but it's present even in the deterministic case we worked out when Kobe, Pau, and Lamar were taking out their aggression on some hapless two-man team with a constant line-up. I said, for instance, that if Kobe and Pau both had APMs of +24, then they'd outscore the opponents, over an entire game, by those 24 points. Not so earthshattering; if they had in fact played the whole game, that's exactly the APM they'd have ended up with.

But then I also suggested that their APMs might be different: Kobe's could be higher and Pau's lower, or the other way around. And most crucially, I suggested that if one was higher, then the other must be lower by the same amount, so that they always add up to 48. In technical terms, we assume that APM combines linearly. That hidden assumption is part and parcel of the APM calculation; it is what allows us to make the determination that although Kobe's APM and Pau's could be any values individually, they must add up to 48. Without the linearity assumption, we can't write any equations at all; we can't compute APM, statistically or otherwise.

If you think about it, though, what justifies this addition of APMs? What makes us think that we can just add players willy-nilly, like numbers? I personally can't think of a thing that justifies that in anything close to a rigorous way. On the contrary, there's every possibility that they don't always add that way. If two players are both offensive powerhouses but defensive milquetoasts, they might both have good APMs because they spend all of their time playing with teammates that cover for their defensive weaknesses. Put them together, though, and since there's only one ball to score with, their collectively miserable defense might make them a net minus. (EDIT: Wayne Winston's version of APM, at the very least, tries to account for this. Look closely at Winston's answer to Question 5 here, and you'll see that his model includes an "interaction" factor that is a function of a pair of players. As a result, you have an affine relation instead of a linear one, and at least some of the first-order issues with linearity are taken care of.)

The linearity assumption is so seductive because it seems natural and jibes with lots of our experience. If I can grade 20 exams per hour, and you can too, then together we can grade 40 exams per hour. But in any endeavor that requires lots of teamwork and collaboration, the assumption becomes more tenuous. That doesn't unfortunately make it any less critical to the validity of things like APM. It simply has to be demonstrated for us to have any legitimate confidence in the value of APM; it isn't incumbent on anyone else to show that the linearity assumption doesn't hold, but for APM proponents to show that it does.

More insidiously, because linearity seems so natural, we are likely to miss its pivotal role in statistical measures like APM. Perhaps someone somewhere has done a study to validate the linearity assumption for APM. But if so, I haven't seen it, and I bet neither have most APM adherents. If you have, please share it!

Inconsequence (A Jazz Tune)

2009-10-01T10:41:00.000-07:00

Something a little different. A test of the video embedding, I guess. (Could it have picked a more objectionable thumbnail?)

An original composition. In my Walter Mitty fantasy world, this is part of a stage musical and is performed twice; the reprise has slightly different lyrics. For my own nefarious purposes, I have Frankensteined the two into one.

Here we are, you and I,
Face to face, eye to eye.
Shouldn't time give a soul
Who while wondering was blundering
A chance to be whole...?

...Hold that thought, just a mo,
Never mind, let it go.
Doesn't matter what we do
From here on, from here on I'll smile
In consequence of you.

This song is Copyright © 2009 by Brian Tung. All rights reserved. Product may have settled during shipping. Do not incinerate. Objects in mirror may be closer than they appear. Operate in a well-ventilated environment. Handle with care. Do not taunt Happy Fun Ball. Contents under pressure. Do not inhale.

Stardust Memories

2009-08-27T22:17:00.000-07:00

When I was ten, my dad took a couple of friends and me to see a movie. My friends and I had the choice of watching Rollercoaster, which was about a terrorist attempting to extort money from amusement parks by blowing up sections of rollercoaster track just as the coaster gets to them, or this new science fiction film that had recently opened and was getting good notices. As you've no doubt guessed, we chose poorly, while my dad went to the other film, which was (as you've probably also guessed) Star Wars. Meanwhile, one of my friends threw up on the car ride home.

I saw Star Wars in the theater four times, which to this date remains the last time I ever saw a film multiple times in the theater. Early in the film, right after the text crawl, but before the rebel ship comes on screen, you're treated to a view of a star field. In fact, here it is (click to enlarge):

When I saw the film again recently, there was something vaguely unsettling and unnatural about the look of the stars in this scene. For the sake of comparison, here's a real star field, with roughly the same level of detail (again, click to enlarge):

What strikes me now (although I was oblivious to it back in 1977, at least consciously) is how much more regular the star field is in the Star Wars frame than it is in the real photograph. There isn't much variation in the stars in the movie frame, with the top fifty or so being about the same brightness; in contrast (no pun intended), there are many more dim stars in the real photograph, and they fade out gradually, suggesting that there are plenty of stars that are in the field of view, but just beyond the limits of detectability, in this photograph at least. And there are, in fact. For some reason, that sense of infinity, which isn't in the movie frame, appeals to me greatly.

You can sort of see the reasoning behind this if you imagine for the moment that all stars are of the same intrinsic brightness, and that the only reason that some appear brighter and some appear dimmer is that they're closer or further away. (Sort of the way that most adults are of about the same height, but appear to be different sizes because they're at different distances.) And because there is more space far away than there is close up, there are more stars that are far away and therefore dim than there are stars that are close up and therefore bright.

Now, as it happens, stars do vary in actual brightness—sometimes dramatically—but the basic explanation still holds, and is supported by actual counts of bright stars versus dim stars. And I think that through long association with the night sky, we gain an appreciation for that kind of aesthetic. Once upon a time, every human on the planet with reasonably good vision had that association. Nowadays, it's less common. But the potential is still there within each of us, and in my case, it expressed itself in, among other things, my preference for the real star image rather than the Star Wars movie frame.

And this set me to wondering whether a sense for this kind of aesthetic could be mechanized in any way. In a very naïve way, it surely could. The way that the star counts vary by brightness follow a fairly well-understood formula, and a star field could easily be scanned for how well it matches that formula. But I think it's a common feeling that that would fall well short of a genuine sense of aesthetics. There would have to be a larger framework for that kind of aesthetic sense.

Could such a framework lie in fractals? Fractals are, generally speaking, patterns that are self-similar; that is, the appearance of the whole at a large scale is repeated in small parts of the pattern at smaller scales. Examples of fractals range from prosaic snowflake patterns:

to the sublime Mandelbrot set:

Fractals have been used to describe natural patterns as varied as the sound of wind through trees and the coastline of Great Britain. And they can be used to describe the appearance of star fields as well. A star field looks quite the same if you zoom in and increase the brightness. The details are different, so in that sense it is not quite like the snowflake fractal or even the Mandelbrot set. But statistically, the close-up shot and the wide-angle shot are essentially identical.

I cannot say exactly what it is about the "fractality" of these patterns that is appealing. And it does seem as though a certain sense of variation (absent in the snowflake, present to an extent in the Mandelbrot set, and rampant in real star fields) is vital to maintaining visual interest. But I can't escape the notion that self-similarity is something that people generally find captivating and inviting, once they recognize it, and is a large part of why looking up at the night sky is such a natural thing to do.

Seventh Night

2009-08-27T17:08:00.001-07:00

Last night was Seventh Night (七夕), the seventh night of the seventh month in the lunisolar calendar followed traditionally by the Chinese. Because the Chinese calendar usually starts with the second new moon after the winter solstice, Seventh Night usually falls sometime in August in the western calendar.

Seventh Night is associated in Chinese tradition with the story of the Cowherd and the Weaver Girl. In one common telling of the story, a young cowherd by the name of Niulang (牛郎) came across a fairy girl bathing in a lake—a girl named Zhinü (織女). Fascinated by her beauty, and emboldened by his companion, an ox, he stole her clothes and waited by the side of the lake. When she came out looking for her clothes, Niulang swept her up and took her back home. In time, they were happily married with two children. But when the Goddess of Heaven found out that a fairy girl had married a mere mortal, she grew furious and sent Zhinü into the sky, where she became the bright star Vega, in the constellation of Lyra the Lyre. (Watercolor by Robin Street-Morris, 2007.)

When Niulang discovered that his wife had disappeared, he searched high and low for her, but was unable to find her. Eventually, the ox told Niulang that if he killed him and wore his hide, he would be able to ascend the heavens to find Zhinü. Niulang did as the ox suggested, and took his two children with him to find his wife, becoming as he did the star Altair. Find her he did, but the Goddess of Heaven, angered once more by Niulang's impertinence, drew a river of stars—the Milky Way—forever separating Niulang (the star Altair) from Zhinü. Their two children became Tarazed and Alshain, the two dimmer (but still bright) stars that flank Altair in the constellation of Aquila the Eagle. But apparently the Goddess of Heaven was not entirely heartless, for once a year, on the seventh night of the seventh month, she sends a bridge of magpies (鵲橋) to connect the two lovers, for just one evening. And so Seventh Night is associated with romance (and also, interestingly, with domestic skills).

The celestial setting for the entire tale can be found in the Summer Triangle, which is bounded by three stars: Altair, Vega, and Deneb (in the constellation of Cygnus the Swan, also known as the Northern Cross). The Summer Triangle can be found in the night sky throughout summer and autumn; at this time of year, it passes nearly overhead at about ten in the evening. (Photograph by Bill Rogers of the Sa-sa-na Loft Astronomical Society, 2009; click to enlarge.)

How Random is Random?

2009-08-26T09:44:00.001-07:00

We all think that we know when something is random. But how random is random?

Part of the aim of mathematics is to unify concepts. It's what makes mathematics more than just a collection of ways to figure things out. As a side effect, though, mathematics definitions tend to be a bit counterintuitive. For example, I think we all know what the difference between a rectangle and a square is: A square has all four sides of equal length, and a rectangle doesn't.

Except that a mathematician says that squares are rectangles, because to a mathematician, it's inefficient and non-unifying to say that a rectangle is a four-sided figure with four right angles, except when all four sides have the same length. It makes more sense, from a mathematical perspective, to make squares a special case of rectangles.

So hopefully it won't come as too much of a surprise if I say that a completely deterministic process, such as flipping a coin that always comes up heads, is still considered a random process to mathematicians who study that sort of thing. So is a coin that comes up heads 90 percent of the time. Or 70 percent. Or—and maybe this is the surprise, now—50 percent. The cheat coin is simply a special case of a random process. To a mathematician, none of these processes is "more random" than the others. They just have different parameters.

What we think of as randomness, mathematicians call entropy. This is related to, but not the same thing as, the thermodynamic entropy that governs the direction of chemical reactions and is supposed to characterize the eventual fate of the universe. (Another post, another time, perhaps.) It turns out that this "information-theoretic" notion of entropy corresponds pretty well to what the rest of us call randomness. For those of you who are even the slightest bit curious, the definition of entropy for a flipped coin is

S = - ( p_H lg p_H + p_T lg p_T )

where p_H and p_T are the probabilities for heads and tails, respectively, and lg is logarithm to the base 2. For a 50-50 coin, the entropy is S = 1; for a completely deterministic coin (a two-headed one, for instance), the entropy is S = 0. For something in between—say, one that comes up heads 70 percent of the time—the entropy is something intermediate: in this case, S = 0.88 approximately.

So, all right, how entropic is a real coin? The answer is that it's probably less entropic—less random, that is—than you think it is, especially if you spin it. A paper by researchers from Stanford University and UC Santa Cruz (via Bruce Schneier, in turn via Coding the Wheel) has seven basic conclusions about coin flips:

If the coin is tossed and caught, it has about a 51 percent chance of landing on the same face it was launched. (If it starts out as heads, for instance, there's a 51 percent chance it will end as heads.)
If the coin is spun, rather than tossed, it can have a much larger than 50 percent chance of ending with the heavier side down. Spun coins can exhibit huge bias (some spun coins will fall tails up 80 percent of the time).
If the coin is tossed and allowed to clatter to the floor, this probably adds randomness.
If the coin is tossed and allowed to clatter to the floor where it spins, as will sometimes happen, the above spinning bias probably comes into play.
A coin will land on its edge around 1 in 6000 throws.
The same initial coin-flipping conditions produce the same coin flip result. That is, there's a certain amount of determinism to the coin flip.
A more robust coin toss (more revolutions) decreases the bias.

Somewhat along the same lines, Ian Stewart, who for a while wrote a column on recreational mathematics for Scientific American, mentioned a study in one of his columns by an amateur mathematician (and professional journalist) named Robert Matthews. Matthews had watched a program in which the producers had asked people to toss buttered toast into the air, in a test of Murphy's Law as it applies to buttered toast. Somewhat to their surprise, the toast landed buttered side up about as often as it landed buttered side down.

Matthews decided that was not quite kosher. People, he thought, don't usually toss buttered toast into the air; they accidentally slide it off the plate or table. That ought to be taken into account when analyzing Murphy's Law of Buttered Toast. And when he did take it into account, he found something rather unusual. A process that you might have thought was fairly entropic turned out to be almost wholly deterministic, given some not-so-unusual assumptions about how fast the toast slides off the table. Unless you flick the toast off the table with significant speed, the buttered side lands face down almost all of the time. And it has nothing to do with the butter making that side heavier; it's that the rotation put on the toast as it creeps off the table is just enough to give it a half spin. Since the toast starts out buttered side up (one presumes), it ends up buttered side down. Stewart recommends that if you do see the toast beginning to slide off the table, and you can't catch it, to give it that fast flick, so that it isn't able to make a half flip, and lands buttered side up. You won't save the toast, unless you keep your floor fastidiously clean, but you might save yourself the mess of cleaning up the butter.

On the other hand, maybe there's another solution.

Lines for Fries (and Fry's)

2009-08-21T15:59:00.001-07:00

Thoughts about (and while) waiting in line at the neighborhood McD's. Mmm...fries.

The Most Evil Being mocked the last queueing theory post, but he actually read the whole thing to mock it. Apparently, so did Squishy. I approve, of course. But Squishy noticed, with some temerity, that I had tagged that post with "queueing theory," indicating a potential for future posts about same. Well, the future is now. That post dealt with queueing theory itself as little as I could manage, which, OK, is still quite a bit, I guess. Fair warning: There's a bit more of it in this one.

If you were so foolhardy as to look in a queueing theory textbook, you'd probably see a representation of a queue as something like this:

The thing at the left is the queue, or waiting line; the colored blocks inside the queue represent customers; and the circle at the right is the server. Nowadays, in the computer world, we think of servers as big honking machines, but in general, it could be anything that provides a service. Say, an order taker at a fast food restaurant.

That diagram up there represents only one potential way to hook customers up with servers: a single queue with a single server. Lots of places, like McDonald's, or the supermarket, or the bank, have multiple servers available at a given time. How do they connect their customers to their servers? Here are two diagrams representing two options, without any explanation. Before reading on, see if you can figure out what queueing systems they represent, and how the lines you wait in everyday are arranged.

System (a), on the left, represents a queueing system in which each server (e.g., checkout clerk, bank teller, etc.) has his or her own line. System (b), on the right, represents a system in which all the servers together share a single line. Where I live, in Los Angeles, McDonald's and the supermarket use (a), and the bank and some other fast food places use (b). Your mileage may vary, of course.

OK, that wasn't too hard, probably. Now think about this one: All other things remaining equal, which system is better? And by better, I mean that it improves the time that you have to wait, on average, before getting service. We'll assume, to make things easier, that once you enter a queue or line, you stay in it; you don't give up, and you don't defect to another line. We'll also assume, furthermore, that all the servers are equally fast (or slow, depending on your point of view). Would you prefer to wait in (a), or (b)?

Before I answer that, let me first define some terms. All queueing systems have what's called an arrival rate, which is the rate, on average, at which new customers enter the queueing system. All servers have a service rate, which is the rate, on average, at which they can serve customers, assuming they have any customers to serve. One of the things I mentioned in that last queueing theory post was that a system is stable (that is, it doesn't jam up) if the arrival rate doesn't exceed the service rate. With me so far?

All right, one last term: The utilization of a server, or a group of servers, is the arrival rate divided by the service rate. So, pretty obviously, if the utilization of a server or servers is less than one, it's stable, and if it's greater than one, it's unstable—the line or lines get longer and longer. Somewhat less obviously, the utilization of a server is also the fraction of time that it spends actually serving customers, rather than sitting idle (which is why it's called the utilization in the first place).

Suppose Store A is using queueing system (a). It's got, let's say, six servers, each capable of serving one customer a minute. Customers come into the store at a rate of three customers a minute. Since each server gets one-sixth of all the customers, on average, each server's customer arrival rate is half a customer a minute, and each server's utilization is 1/2 divided by 1, or 1/2.

Store B, on the other hand, is using queueing system (b). It also has six servers, each of which also serve at one customer a minute. Because they all get fed from the same line, it's convenient to think of them as together serving customers at a rate of six customers per minute. If the arrival rate to Store B is the same, three customers per minute, then the utilization of the six servers, combined, is 3 divided by 6, or again, 1/2. So far, it seems like the two systems are pretty equivalent.

However, Store A has a problem that Store B doesn't. Consider the situation diagrammed below:

Five of the servers are busy, and they even have customers waiting in line behind them. The sixth server, however, is entirely idle, but because we've assumed that customers don't switch lines, it has nobody to serve. (Lest you think this is entirely unrealistic, I see it all the time at the supermarket, possibly because the idle server is a few counters away from the busy ones.) This is bound to happen from time to time, since the utilization is less than one. Servers are going to be idle every now and then, and if that happens when some other customers are waiting to be served, Store A is going to be inefficient at those times.

Note that this never happens to Store B. Certainly, servers are going to be idle from time to time, and customers are going to have to wait from time to time. But they never both happen at the same time. Any time a server comes idle, if there's any customer waiting for service, it can go straight to that server. As a result, Store B, and queueing system (b), is better for the customers: They wait for a shorter time, on average, than customers at Store A.

What's more, queueing system (b) is fairer, in the sense that customers that arrive first are served first. That doesn't always happen with queueing system (a). In the situation depicted above, if a customer now arrives to that sixth, idle server, it gets served immediately, without having to wait, even though customers that arrived previously to other lines are still waiting. So (b) is doubly better than (a).

In light of this, it shouldn't come as any surprise that Fry's Electronics, essentially the store for übergeeks, uses system (b) in every one of its stores I've been in. It even takes advantage of the longer single line (as opposed to an array of shorter lines) by snaking it between and amongst a panoply of impulse buys. One could argue that supermarkets can't really take proper advantage of system (b), because people usually have carts, and these take up a lot of room, which would obstruct other supermarket traffic. (I also haven't considered the effect of the 12-items-or-less express lanes.)

But a place like McDonald's has no such excuse. Even if you make the point that people switch lines when there's nobody waiting at a server (because the service counter is not so large), it's still unfair, in that it's not first-come-first-served. And other fast food places are perfectly willing to arrange a single line for all servers.

Coke, Currency, and Contagion

2009-08-20T18:07:00.000-07:00

Recently, there was a report, from the American Chemical Society, that about 90 percent of U.S. currency in circulation has detectable traces of cocaine on it. Apparently, the middle currencies—from Lincoln on up through Jackson—are the most susceptible. I guess Washington and Franklin don't rate. Also, not surprisingly, the percentage varies according to the community. Rural areas are less hit by cocaine-laden dollar bills, but in major metropolitan centers, essentially every piece of currency has coke on it. What's more, the percentage appears to be rising. In 1985, a study found that anywhere from a third to a half of bills had cocaine on them; in 1995, the proportion was three in four; and in 1997, it rose to four in five. Now it's nine in ten.

No need to panic, though. First of all, the traces are generally tiny, much smaller than a grain of sand, and not enough to get any kind of buzz from. And secondly, probably much, though apparently not all, of this increase has to do with the improved sensitivity of the cocaine sniffing tools.

The question is, how does cocaine get on all these bills? Certainly not all of the bills get cocaine on them because they were directly around the stuff, either during deals or during use. A small number do, of course, but the vast majority get them through contamination. But is that really plausible? Can so many bills be contaminated so quickly?

Well, let's take a look at that. Suppose that, initially, some small fraction of all the dollar bills have detectable cocaine on them; these are the initial set that get cocaine on them through direct contact with bulk quantities of the drug. Let's call this proportion p. The money isn't discarded, generally; it's put back into circulation (let's not get into how they get put back into circulation). Once that happens, those bills come into contact with other bills, which pick up some proportion of the drug. Apparently, there's an attraction between the drug particles and the green ink used to print U.S. currency.

When I use a bill, and it goes somewhere else, it now comes into contact with, let's say, one new bill. If a contaminated bill comes into contact with another contaminated bill, nothing happens to p, of course; both bills were already contaminated. Same thing holds true if an uncontaminated bill comes into contact with another uncontaminated bill.

But if the bill I had was contaminated and its new companion wasn't, or vice versa, then one new bill gets contaminated. The probability of this happening depends on the current value of p; specifically, it must be proportional to p (1 - p), since we need a contaminated bill and an uncontaminated one. We can put this in terms of a differential equation:

dp / dt = kp (1 - p)

The constant of proportionality k indicates how quickly bills come into contact with one another, and can be eliminated by setting the unit of time equal to the mean time it takes for a bill to be used (and therefore find a new neighbor). I don't have any hard figures, but from my own, non-cocaine-related currency use, it seems to be about a week or so. We can then set k = 1 and solve this equation fairly straightforwardly to yield the formula

p = C e^t / (1 + C e^t)

where C is closely related to the initial proportion of contaminated bills. (To be exact, C = q / (1 - q), where q is the initial proportion. Where q is very small, as in most cases, the two are almost exactly the same.) As t increases, C e^t gets large pretty quickly, and p very quickly approaches 1. If, for instance, q = 0.000001—that is, one bill in a million is contaminated directly by the drug—then it takes a bit more than three months for the fraction of contaminated bills to exceed one-half. But because of the rapid growth of the exponential function, it takes only one more week for the proportion to exceed three-fourths. By the end of the fourth month, the fraction of uncontaminated bills is less than one percent. (Click to enlarge.)

That exceeds even the ACS's report. Why? Well, for one thing, even today's instruments are not perfectly sensitive; there still remain bills with undetectable traces of cocaine, surely. And after a while, there just isn't enough cocaine to go around (for the bills, that is). If, for the sake of argument, we assume that the initial fraction is one in a million, then the ACS's estimate of 90 percent contamination indicates that that first direct contamination can only be split about twenty times before it drops below undetectability.

But a second reason is that bills don't stay in circulation forever. According to the U.S. Treasury, currency stays in circulation, on average, for about 20 months—about 85 to 90 weeks. This makes the dynamical solution to the differential equation a bit more complicated. Let's simplify matters and only look at the equilibrium solution. At equilibrium, the contaminated dollar bills being taken out of circulation each week equal those being contaminated by new contact each week. That is,

p (1 - p) = rp

which yields an equilibrium solution of p = 1 - r, where r is the fraction of bills being taken out of circulation each week (about 1/85 to 1/90). So even with this new influx of bills, if detection tools were perfect, they'd detect traces of cocaine on about 99 percent of bills. Apparently, we still have a few rounds of "alarming" reports about cocaine contamination of currency to look forward to.

OK, here's a less overblown concern. The same model can essentially be used to analyze long-lived infections (such as oral herpes, which infects about 60 to 70 percent of all people worldwide). Such infections are removed from the population only when a person dies. As the above models show, if people were immortal, they'd eventually all be infected with such diseases (and in fairly short order, too). Of course, such diseases couldn't incapacitate their hosts too much, because otherwise they'd fail to be transmitted.

Queueing Theory and You

2009-08-13T15:13:00.001-07:00

Some thoughts on traffic—the automobile kind, not the network kind—while there's maintenance work going on in the office across the hall.

So the other day I'm driving into work, and I encounter not one but two traffic jams. Neither, as it turns out, was due to particularly heavy traffic loads. Rubbernecking (a.k.a. looky-looing) was the culprit in both cases. In both cases, the accident/attraction was off to the side of the road but managed to clog up the roads all the same.

I think it's generally underappreciated how much rubbernecking contributes to traffic jams. No one disputes that the accident itself can precipitate the jam. But a moment's satisfaction of curiosity? On the surface, it seems innocuous, right? As one of the drivers stuck in the jam yourself, you've already spent 10, 15, 25 minutes waiting behind this long line of cars—what could it possibly hurt to glance over for a second or two? But it's precisely that kind of glance that keeps the jam going. The reason for this lies in queueing theory, the study of waiting in lines, and comes about from the interplay between the level of traffic applied to a road, and the carrying capacity of the road.

Roads, like any other conduit, have a certain capacity, which is related to the size of the road but is also determined in large part by driving habits. You're taught, when you're driving, to leave at least three seconds of space between you and the car in front of you—more if it's dark or rainy or whatever. In the Los Angeles area, where I live, it's essentially impossible to do this; if you try, someone will invariably slide into the space, cutting yours down to a second or two, after which your options are to either to stay up close, or to back off until you're three seconds behind the new car, in which case the process repeats.

But actually the exact time is not all that important; what's important is that there is a characteristic following time, which determines the carrying capacity of the road. If the following time is two seconds, then the road can carry half a car per second (per lane). Note that this capacity is roughly accurate no matter how fast the traffic is going—whether traffic is flowing at the speed limit or crawling at 15 mph—as long as the following time is roughly the same. Only when traffic slows so much that cars take a significant time to travel their own body length (the following distance isn't head-to-head, but tail-to-head) does this rule break down.

Provided that that doesn't happen (and we'll get to that in a moment), we can now apply the most basic rule of queueing theory: If the amount of traffic going onto the road is more than the road's carrying capacity, traffic will come to a standstill. Hardly earthshattering news. If the amount of traffic is less than the capacity, traffic can flow. It might, however, flow incredibly slowly.

At first flush, this might sound kind of strange. If a road can carry a car every two seconds, and one car comes down the road only every three seconds on average, shouldn't there be enough room for cars to drive smoothly down the road, with quite a bit to spare? The perhaps surprising answer is that there might not be, and the fault lies in that phrase "on average."

If cars all scrupulously observed at least a two-second following time, and entered the road exactly three seconds after the previous car, then in fact, the cars would be able to flow at the speed limit. They would continue to do so even as you increased the rate of cars entering the road, up until the exact moment when that rate exceeded the capacity. At that point, the cars would start backing up and you'd get a traffic jam. And if you've ever been in a large traffic jam, it might seem that that's exactly what happened.

But that isn't in fact what happens. Generally speaking, the capacity of the road is not exceeded for long stretches. It's just very close. So why doesn't traffic flow smoothly, if the traffic load is less than the capacity? There are a few reasons, but the predominant one is that cars do not observe consistent following time, and don't enter the road at a constant rate. In queueing theory, variation kills.

Suppose that the following time is always at least two seconds, but that cars enter the road every three seconds only on average. Sometimes it's less, sometimes it's more. If it's less—let's say it's a second and a half—the new car now has to wait a half a second before it can proceed, because it's trying to maintain a minimum two-second following distance. On the other hand, if it's more, it doesn't have to wait at all. But it also doesn't try to speed up to catch up to the previous car; it's not trying to maintain exactly a two-second following distance, just a minimum of two seconds.

In short, if the time between successive cars is low enough, it slows traffic down, but no amount of time between cars will speed the traffic up. What's more, the closer the traffic rate gets to capacity, the more often a cluster of cars will arrive to slow down traffic, while the gaps between the clusters still fail to speed it up. We can express this effect graphically, by plotting traffic waiting time (a measure of the intensity of the traffic jam) as a function of the traffic rate R.

The exact shape of this graph depends on how following time and the time between cars entering the road vary randomly, but the basic effect is consistent: Instead of the waiting time (the blue curve) being constant at zero until R reaches the road's capacity C, it actually begins ramping up immediately, slowly at first but with increasing intensity until it spikes upward just as it approaches C (the dotted red line). And when you get close enough to C, the waiting time T gets large enough that you notice it as a honest-to-goodness traffic jam.

So what happens when people rubberneck? Yes, it's true, you might have been waiting for a long time, and you're only looking for a second or two. And you're still kind of driving at the time. But you slow down, just for a split second, and increase your following time. Instead of maintaining a minimum two-second following time, you increase it, maybe to two-and-a-half seconds. And if most everybody does this, the capacity of the road is effectively decreased, by 20 percent. It would have the same effect as closing one lane of a five-lane highway.

You might expect this to increase the waiting time T by 20 percent, but actually, what effect this has depends on how high R is compared to C. If it's relatively low—if we're on the left side of the curve—then moving C down by 20 percent, while keeping R the same, doesn't really affect T very much. But if it's already kind of high (and in Los Angeles, at least, it's that high about 24 hours every day), then moving C down by 20 percent can move you catastrophically high up that blue curve, increasing T many-fold and changing a mild nuisance into a dinner-delaying, or even dinner-cancelling, jam.

But that's OK. You just go ahead and look at that upside-down pickup. What could it hurt?

Slashed Back

2009-08-03T21:47:00.001-07:00

I'd like to call your attention to our latest scourge: Well-meaning radio announcers who, while reading out URLs in commercials, refer to the ordinary slash (/) as a "backslash." Why they feel compelled to even use the word "backslash," goodness only knows, since most people only ever come into contact with the ordinary slash; the backslash is almost exclusively used by DOS and LaTeX heads.

We now return you to your regularly scheduled rant.

The Chinese Script Is, or Is Not, Phonetic

2009-07-28T14:34:00.000-07:00

A new post to satisfy the likes of the most evil being...in the universe.

My favorite Chinese app (whose continued absence from the iTunes Store is delaying my inevitable purchase of an iPhone) is headlined by a dictionary edited by the recently departed John DeFrancis, who taught Chinese for years at the University of Hawai'i. Aside from his 12-volume Chinese language textbook series, he's probably best known for The Chinese Language: Fact and Fantasy, an accessible deconstruction of several myths regarding the Chinese language: that the Chinese script is ideographic, for instance, or that it is specially tailored to facilitate communication between speakers of mutually unintelligible dialects.

At one point, DeFrancis goes even further and suggests that the Chinese script is not even logographic, with each character signifying a morpheme, but simply phonetic, with each character signifying a phoneme—albeit a tremendously inefficient phonetic script, since in many cases it has dozens of characters representing a single phoneme. He's talking here about what most people would consider homophones: characters like 出 to exit, and 初 the first or opening of a series, both of which are pronounced chū in Mandarin (the most widely spoken dialect). This idea is prelude to a discussion of various proposals to do away with Chinese characters entirely, using in their place a properly designed phonetic script.

DeFrancis's interpretation isn't as crazy as it might sound at first. In at least one limited case—the transliteration of foreign terms—the Chinese script is exactly an inefficient phonetic script. Lacking an official alphabet or syllabary, Chinese represents foreign terms using a sequence of characters, such as 巴巴多斯 bābāduōsī for Barbados. The inefficiency lies in that unless you've previously looked this term up, you'd have no good idea which four characters ought to be used to write out Barbados. There are lots of equally effective ways to write out Barbados using Chinese characters...but only one way that is considered "correct."

More generally, even though the distinction between the written forms 出 and 初 might seem vital to Chinese readers, since they mean different things, the morphemes represented by those characters are used all the time in speech, where there is nothing but context to distinguish them. Apparently, "nothing but context" works pretty darned well. There are written passages that consist of nothing but a long string of homophones—a sort of extended pun—but the fact that these are elaborately conceived literary jokes is actually an indication that the semantic disambiguation the different characters provide isn't strictly speaking necessary.

But I think that last point is an indication of why the Chinese script is not a phonetic script, or at least not only a phonetic script. Because even though the semantic disambiguation isn't necessary, I think it's pretty hard to argue that it doesn't help. One can read a passage in Chinese that is rendered only phonetically, but to someone who's literate, it's a lot faster with ordinary characters. Whether we're talking about an alphabetic language like English or a logographic one like Chinese, people typically read a lot faster than they can speak. To me, that indicates that there's something going on in the reading process other than just reproducing the sounds of speech. Robert Ramsey wrote in his book The Languages of China that there's still a lot we don't understand about the way Chinese people read. Without understanding more about that, it's premature to conclude that Chinese characters are a poor stand-in for a syllabary.

Do Not Pass Text Design, Do Not Collect $200

2009-06-27T21:14:00.000-07:00

A long time ago (I won't say just how long, but it'll soon become fairly obvious), I worked my first summer job at a computer-controlled font engraving place in Mountain View called Xybergraphics. Lots of stories from that summer, which I'll eventually get to when I want to talk about what happens when a 95-pounder drinks 42 ounces of caffeine-laden soda pop, or the beginnings of my fascination with the Police, or what a cubic spline is.

At any rate, my job, which paid me the princely sum of $2.85 each hour (workers under 16 could be paid somewhat under minimum wage), required me to encode fonts for the aforementioned computer-controlled engraver. Typically, the engraved letters would be in the neighborhood of an inch or two in height, but for better precision, the letterforms I worked from, designed by my immediate supervisor, were about 14 inches tall and drafted in pencil on vellum. My job was to take a mouse (this was before the Mac, mind you), and trace along the letterforms, clicking at appropriately spaced points, until the letter was entirely traced. A simple letter, like a capital I, might require 50 points; a more involved letter, such as lower-case m, might require as many as 150. This was a tedious and time-consuming job, you could well imagine.

But I'm nothing if not efficient at boring tasks, especially if I've got my tunes in the background, and in the meantime, I learned quite a lot about fonts—what they should look like, how features are shared in common by various letters, what design rules not to break, and so forth. All this that I learned is both fascinating (to me) and almost entirely useless, which means that it has lodged tight in my memory banks and won't budge.

As a result, I'm exceptionally sensitive to bad typography. For example, I was walking one day in the Denver airport, waiting for my next flight, when a store sign caught my eye. Bad Typography Alert! The capital A in the sign was reversed; its broad stroke went down to the left, rather than to the right, as it should in traditional serifed fonts. And this on the sign for a stationery store! I think you will properly apprehend the depth of my mania when I tell you that I actually reported it to the clerk. She seemed quite receptive to my complaint, although she probably promptly forgot about/rejected it as soon as I turned my head.

There's a similar problem with one of the signs where I work—the name of the building has a capital V in it, with its broad stroke, again, going down to the left. Alas, this is welded on and probably unmulliganable.

But my encounter with the very bottom, the absolute worst, came when I began frequenting a supermarket that opened near our house. There was little wrong with the typography at the supermarket, but across the street was this abomination:

This signage is simply staggering in its wrongness. It's hard for me to convey just how staggering, but the fact that you're reading this post is some indication. Click on the image to see an enlarged version and look at this tangle of thorns.

Practically any letter that could have been misplaced, was, and those that weren't, seem to have been correctly placed to highlight how wrong their identical partners were. To wit: The E in GERMAN is upside-down. The M and A in the same word are backwards. Unbelievably, the N is correctly placed. A mistake must have been made.

In CAR, the C is upside-down. The A is backwards. After the bad E in GERMAN, the E's in SERVICE are right, but the S and the C (again!) are upside-down, and the V is backwards.

The M in BMW is backwards. The W is very strange, it seems to have been made with a pair of leftover V's, both with one stroke broken in half. If so, they should have broken the other stroke on the right half, but I'll give them credit for showing some resourcefulness.

The horror continues in Volkswagen. This V is correct (what happened to the V in SERVICE?), but the l is backwards, the s is upside-down, and the w and even the g (!) are backwards again. How do you screw up the g? The lower-case s is upside-down again in Porsche, and the lower-case c joins its capital brethren in being inverted also.

The Audi is correct, but is set in a narrower font. Must have been added on later.

Considering that many of the letters couldn't have been placed incorrectly (either because they're totally symmetrical or totally unsymmetrical), the percentage of letters placed incorrectly runs at about 15/22 = 68 percent, by my reckoning. A dolphin without opposable thumbs flinging letters up randomly with its tail could have done better.

OK, I realize that I'm probably clinically unhinged on this point, but can we agree that someone screwed up royally here? I mean, please.

Game Theory and the Wing-Block Dynamic

2009-06-23T16:12:00.000-07:00

In 2004, when the Lakers played the Pistons in the NBA Finals, a lot was made of Kobe Bryant continually jacking up outside jumper after outside jumper—none too efficiently, most of the time—while monster center Shaquille O'Neal was taking fewer shots, but making them much more efficiently. On the surface, it sure seemed as though Shaq should have been getting more shots, and of course Shaq, never a wallflower at the quietest of times, was not loathe to point this out.

In 2009, when the Lakers played the Magic in the NBA Finals, a lot was made of Kobe Bryant continually taking jumper after jumper—somewhat more efficiently than before—while his "newly tough" post player Pau Gasol was taking far fewer shots, but making them more efficiently. On the surface, it sure seemed as though Pau should have been getting more shots, and surprisingly Pau, generally a quiet fellow, pointed this out with a certain degree of mordacity.

Obviously, in retrospect, the two series turned out rather differently for the Lakers, which is why the former case was judged by many as the reason the Lakers lost the series, and the latter is just a footnote. Bryant's reputation as a ballhog, already in force before the 2004 Finals, was substantially bolstered by that series, and has only just faded within the last year or two. But is that fair? Is that the only possible interpretation for Kobe's shot-taking? Or could ballhoggery conceivably help a team?

Let me be clear here. There's no question in my mind that Kobe could stand to take fewer shots than he does (unless he's just red hot). The question isn't whether he should take as many shots as he does, but whether he should take shots even when he's shooting them at a lower percentage than the post players. And this really goes for any wing player who dominates the ball (e.g., LeBron, Wade, etc.). I just mention Kobe because I watch all the Lakers games.

I'm going to look at this from a game theory standpoint. Put into elementary game theory terms, Kobe and the Lakers have a set of tactical options, and the defenders have a set of tactical options. If each side optimizes its strategy with respect to the other side, then in the end, the game will reach what's called a Nash equilibrium: Neither side can improve its result by changing its strategy unless its opponent changes it too. (The equilibrium is not named after award-winning point guard Steve Nash of the Phoenix Suns, but John Nash, award-winning mathematician and subject of the award-winning book/movie, A Beautiful Mind.)

Suppose we simplify matters by assuming that the Lakers have just two options: Kobe shoots, or Kobe passes to the post, which then shoots. And the opponents likewise have just two options: double Kobe, or play man-to-man. And naturally, we assume that Kobe shoots a better percentage over man defense than over a double team, and the post shoots better when Kobe draws a double team than when the defense plays man-to-man.

The conditions of the game do not require either side to do the same thing each time. Strategies can be mixed. So Kobe can shoot 60 percent of the time, and pass 40 percent of the time. The defense can double 70 percent of the time, and play man 30 percent of the time. The defense can even have partial strategies like a weak double versus a strong double. Under these simple assumptions, it's fairly straightforward to find the Nash equilibrium, where neither side can unilaterally improve their result. What's interesting about this Nash equilibrium is that both Kobe and the post should shoot exactly the same percentage.

Plainly, that doesn't happen very often. Very often, Kobe shoots a lower percentage than the post (even when factors such as free throws and the three-point line are taken into account); by comparison, it's relatively rare that it happens the other way around. Ostensibly, with Kobe shooting the ball so much, he's not adequately punishing the defense for doubling him. He should instead pass the ball into the post more, gradually causing the defense to double less and play more man defense, up to the point where his shooting percentage rises to match that of the post.

[EDIT: The rest of this post is largely different from what it used to be, because what follows totally swamps in significance what used to be here.]

Having said all that, I'm going to go back and suggest that that strategy actually isn't optimal. How can it be sub-optimal, if it's at the Nash equilibrium? Because the game doesn't stop when the ball hits the rim, so the game theory shouldn't, either.

When players shoot the ball against straight-up defense, the defense has the advantage on rebounding any misses, because they're usually between their man and the basket. However, when a perimeter player shoots against a double team, the rest of the players have a man advantage. In our scenario, this advantage plays out in the post, which means that (a) the chances are much improved for an offensive rebound, and (b) if an offensive rebound is gained, it usually leads to a high-percentage shot.

What effect does that have? Suppose that the man advantage on rebounding leads to an increase of 15 percent in the offensive rebound rate; for example, if the offensive used to get 20 percent of the rebounds, they now get 35 percent. And suppose also that this leads to a successful shot 60 percent of the time. If the wing player misses, let's say, 60 percent of his shots against a double team, and he faces a double team 50 percent of the time, the offensive rebounds effectively amount to an increase in shooting percentage of 0.5 × 0.6 × 0.6 × 0.15, or 2.7 percent. That doesn't sound like much, perhaps, but it's about a standard deviation's worth, the difference between a top-10 guard and a middle-of-the-road guard. And it's how much worse the wing should shoot than the post at the true optimal strategy.

Again, I'm not suggesting that this is how Kobe thinks (although I'm pretty sure he does think that his misses can lead to easy baskets for his team), or that Kobe shoots exactly as much as he ought to. But it might explain why, even if he's shooting a lower (true) percentage than his post players are, he shouldn't necessarily shoot it less.

Inconsistent Bracketology and Non-Transitivity

2009-06-18T17:40:00.000-07:00

Does anyone who routinely does NCAA playoff brackets know the answer to this one? Can you fill in a bracket inconsistently, so that you have (let's say) team A beating team B and team C beating team D in the first round, and yet you have either team B or team D coming out of the second round?

Because it's not hard for the probabilities to come out that way. One simple way is for A and C to be mild favorites over B and D, respectively, but for B and D to be prohibitive favorites over C and A respectively. (Matchups between A and C, or between B and D, can be pick-ems.) So you fill out your bracket to have A and C come out of the first round, but either B or D to come out of the second round. This requires a certain amount of non-transitivity in the teams: For instance, A edges B, which trounces C, which edges D, which trounces A again. But that's hardly unknown in the basketball world, and is usually trotted out as the inevitable "matchup issue" between two teams.

Somewhat more surprising is that it's possible for the same inconsistency to happen without any non-transitivity. Suppose A is a huge favorite over B in the first round, while C is a mild favorite over D in the second round. So you have A and C come out of the first round. But suppose C is also a mild favorite over A, but D is a huge favorite over A. There's no non-transitivity—you can place the teams in the total ordering C > D > A > B—but D is nevertheless the favorite to come out of the second round, despite not being the favorite to come out of the first.

Even though there's no non-transitivity in the second example, it's vaguely unsatisfying because it doesn't match our intuition. We'd like to think that since C beats D, C should be a bigger favorite over A than D is. But the inconsistent bracket result only comes about here because that intuition is violated. So, the semi-open question ("semi-open" because I suspect it won't be that difficult to resolve): Is it possible for a set of tournament contestants to fall under a total ordering in the intuitive way suggested above, and still yield an inconsistent playoff bracket in a binary, single-elimination tournament? It need not be limited to a four-team bracket, but it does have to be 2ⁿ for some n.

Kobe, Once More Unto the Light

2009-06-14T21:46:00.000-07:00

The bare facts: The Los Angeles Lakers dominated the Orlando Magic over the last three quarters to take Game 5 and clinch the NBA title, winning going away, 99-86. After a 16-0 run (capped by a nifty Lamar Odom reverse lay-up) took the Lakers from a 40-36 deficit to a 52-40 lead, the Magic never seriously threatened to take the game back, getting no closer than five points the rest of the way and spending most of the second half down by double digits.

Bryant was, I felt, the clear-cut MVP of this series, and of the playoffs, and even when his game was somewhat off in the middle three games of the series, he cast his enormous shadow over how the games were contested. Whether or not you thought he was over-dominating the ball, whenever he was on the floor, he set the tone for the other nine players.

In some sense, for most of his career, he has cast that same shadow on the NBA. For better or for worse (and there have been no shortage of those who see it for the worse), he has been the top talking point of the league. From his unbelievable moves on the court to his embarrassing personal problems in Colorado, his life trajectory thus far has been an eventful one. His triumphs and travails have galvanized public opinion like no other player, possibly in the history of the league. To Kobe haters, Kobe fans are as thin-skinned as their hero, reacting to any perceived slight as though it were heresy; to Kobe fans, Kobe haters seize any opportunity, twist any circumstance, and trample any logic to put the target of their envy in a negative light. Each group sees the other as the yin to its yang, a state of affairs that would be ludicrous with respect to any other player. But apparently it's de rigeur in Kobe's World.

Through it all, Bryant was insouciant, an outwardly joyous 18-year-old rookie; then a driven talent, rising with center Shaquille O'Neal to dominate a trembling league; and then a fallen hero, commonly considered to have forced O'Neal and then coach Phil Jackson off the team. The haters had a field day watching Kobe try, and fail, to lead a ragtag crew to even the lower echelons of the playoffs, pride going before the fall. Jackson returned the following season, but the next two years were barely an improvement, with the Lakers falling to the Phoenix Suns each year in the first round. His undeniable skills on the court were only further testament, it seemed, to his failure to lead his team off it. Bryant himself appeared to adopt the demeanor of a flawed, secretive superhero with a dark past and a darker future, Batman to O'Neal's Superman. The 2007 off-season was the darkest yet, with Bryant railing to all within hearing range about the front office's inability to provide him with a sufficient supporting cast.

The next season brought a pleasant surprise, however, in the unexpected form of a contending team. And when rising young center Andrew Bynum went down with what turned out to be a season-ending knee injury, the beleaguered Lakers' front office obtained multi-talented Pau Gasol from the Memphis Grizzlies for a song, and the Lakers barely missed a beat. Bryant seemed readier than ever to share the ball with his teammates, making the team less predictable, more formidable. There was a regular season MVP for Bryant, his first, matching O'Neal's award from 2000. Even with Bynum out, the Lakers manhandled the rest of the Western Conference on the way to the NBA Finals. The Batsuit was ready to crack. But the Celtics sunk the Lakers in six games, trouncing L.A. by 39 in the clincher.

Back to the cave. Not alone, not to sulk, but this time with all his teammates, forging something of a defensive identity. Bryant and the Lakers were determined that this time would not be the monstrous disappointment of the previous season. There would be no MVP award this year. That would go to LeBron James, the new King. Bryant had no time for regular season plaudits anyway. He wasn't looking for redemption, either; he never felt he had anything he had to redeem himself for. What he was looking for, I like to imagine, was a lighter Kobe Bryant...

The Power of Flexibility

2009-06-13T08:27:00.000-07:00

With 10.8 seconds left in regulation in Game 4 of the NBA Finals between the Lakers and the Magic, the Lakers had the ball out of a timeout after Dwight Howard had just missed two free throws to leave the Lakers down three, 87-84. Lakers coach Phil Jackson decided to take the ball near their own baseline (where the timeout had been called), rather than advance it to halfcourt. Trevor Ariza inbounded the ball to Kobe Bryant, who was immediately double-teamed. Bryant advanced the ball near halfcourt back to Ariza, who then cross-courted the ball to Derek Fisher. Fisher dribbled the ball up toward the three-point line on the right wing. Since Jameer Nelson was playing so far off Fisher, Fisher decided to hoist up the trey at that point and sank it to tie the game with 4.6 seconds remaining. The Magic failed to score in their final possession of regulation and the Lakers would go on to win the game in overtime to take a 3-1 lead in the series.

Tim Legler of ESPN later suggested that Jackson's decision made it easier on the Magic, because of the extra time that bringing the ball the length of the court would consume. I think this takes a narrow and unnecessarily time-centric view of the play.

In the first place, 10.8 seconds is a lot of time for a "last-second" play. It's nearly half of a full shot clock. The Phoenix Suns could probably run three whole plays in that amount of time. It's unlikely the Magic could delay the Lakers long enough to avoid giving them a decent look. Indeed, Fisher sank the three-pointer with 4.6 seconds left, but he actually released it with 6.2 seconds; the whole play took less than five seconds to execute.

Secondly, Legler underestimates the pressure that having to play full-court defense places on the Magic. If the Lakers had inbounded the ball at halfcourt, they would have had to set their offensive positions for the most part, showing their hand on the playcall and allowing the Magic to set their defense. Whether or not the decision to bring the ball up surprised the Magic, it concealed the Lakers' play from them and required them to cover a multitude of options.

As it happens, the Magic decided to double Kobe, and the Lakers took advantage by quickly advancing the ball out of the double-team to give the Lakers a 4-3 man advantage on the rest of the court. The Lakers had used this ploy, a kind of basketball aikido, several times in the second half of Game 5 of the Western Conference Finals against the Denver Nuggets. In that game, the Nuggets decided to double team Kobe aggressively, pushing him all the way toward the halfcourt line. Kobe obliged them, drawing his two defenders so far away from the basket that by the time Kobe passed out of the double team, they were effectively out of the play, giving the Lakers a man advantage for long enough to get an easy shot. In hindsight, this strategic decision by the Nuggets was a main reason they lost the game and the series.

But even had the Magic chosen not to double Kobe, the Lakers still had a multitude of options to run, starting from the backcourt, and the Magic would have had to anticipate them all. Most options put the Lakers in a kind of semi-transition game, placing the Magic defense in jeopardy. Normally, teams run very unimaginative sets at the end of a period, and the Lakers are no different in this regard, typically putting the ball in Kobe's hands and letting him go 1-on-N. The fantastic play run by the Magic at the end of Game 2, freeing up Courtney Lee for an alley-oop attempt, was very much the exception rather than the rule. And in this game, Fisher still had to make the jumper. But Jackson's decision to bring the ball up the length of the court broke the usual mold and gave the Lakers their best chance at tying the game.

See, Why Would You Do This?

2009-06-10T09:58:00.000-07:00

Look everybody: It's pet peeve time! (I'm writing this because it's against my beliefs to blog about basketball after a Lakers loss, heh. Or, rather, I'm copying this—it's originally from another site of mine. But it's recent.)

I like to bicycle. Not like some friends, who like to bicycle like they like to breathe. (Credit goes to Sandra Boynton for that delectable quote, adapted for the nonce.) But I still like to pedal here and there. So I take it personally when a cyclist every now and then decides they wish to flout one of the basic rules of bicycle behavior—even more that they do it in the name of safety.

They prefer, they say, to ride on the left side of the road.

Note that I'm not talking about people who ride briefly on the left side of the road because their home is on that side, and they're about to make a left turn at the corner, but about people who speak of riding on the left side as though it's some sort of closely guarded Success Secret® of the cycling guild. The justification, apparently, is that it allows them to see oncoming traffic, the better to avoid it. This is wrongheaded on so many levels that it's mindboggling to comprehend. Don't be one of those mindbogglers. Ride on the right side of the road. (Unless you ride in the U.K., or some other of those countries where they drive on the left side of the road.) The reasons are legion:

First of all, the most important: Collisions will happen from time to time, whether you can see them coming or not. But riding toward cars means that your velocity is added to theirs, not subtracted, not to mention the prone-to-flipping head-on collision rather than the more stable rear-ender. Drivers will suffer a somewhat higher repair cost. You will suffer a substantially higher chance of dying, and if not that, of serious injury.
The higher relative velocity has another penalty: Cars are much larger than you, are moving much faster than you, and take much longer (both in distance and in time) to stop than you. As a result, the car has by far the greater part of control over whether there's an accident. Your ability to see oncoming traffic is as nothing, in terms of importance, when compared to the drivers' ability to see you. And by riding toward cars, you give drivers correspondingly less time to see you and avoid an accident.
What's more, most cyclists ride on the right side of the road. Drivers know this, whether they know that they know it, and scan the road accordingly, especially when they're making a right turn onto a major road: At such times, they look back down the road to their left for traffic to avoid. Not traffic to the right, which is where you will be coming if you ride down the left (i.e., wrong) side of the road. A similar comment applies to cars parked on the side of the road pulling out into traffic.
The one time when a car is relatively maneuverable on the road is when it is stopped to make an unprotected left turn (i.e., yielding to oncoming traffic). Drivers are then looking into oncoming traffic, where they will see all potential hazards, except—you guessed it—cyclists riding down the left side of the road.
Pedestrians are advised to walk down the left side of the road, because they can stop on a dime, unlike bicycles. If you ride down the left side of the road, you will be coming up behind such pedestrians, who may chose that moment to turn on a dime and smack right into you (potentially throwing both of you into oncoming traffic).

I hear this less and less, fortunately, but it still crops up from time to time, from people who should know better. (That's approximately everybody, in my opinion.) I think it stems from the natural inclination to think that if you only have all the information you want, you'll be in complete control over your destiny, but whatever the reason, it's just a vastly inferior decision.

EDIT: Since I first wrote this, I've been informed that there is a specific term for this: salmoning. So don't salmon. You know what happens to salmon after they make their way upstream, right? It's not exactly an unalloyed happy ending.

Points are Points, Sort Of

2009-06-08T09:43:00.000-07:00

In the wake of last night's Finals Game 2 between the Lakers and the Magic, which the Lakers won in overtime, 101-96, a lot of attention was focused on various plays that the Lakers made down the stretch and the Magic didn't. Now, obviously, in a game that close, there were plays—even down the stretch, at least in regulation—that the Magic made and the Lakers didn't, and if the game had gone the other way, we'd be talking about those plays. But this just by the way.

Some folks pointed out that although those plays late in the game are magnified in our mind, they aren't worth more on the scoreboard than plays earlier in the game, even in the first quarter. A clutch shot made with the game clock running down is not given more points than an identical shot made in the opening minutes. So undue attention, it is claimed, is placed on, say, Courtney Lee's missed lay-up with 0.6 seconds left in regulation, a shot that would have given Orlando the win and a split in the first two games of the series. (I was watching the game, by the way, and I somehow totally missed the play developing: the jab step, the perfect screen from Rashard Lewis, everything. Thank goodness!) If the Magic simply make one of the two-pointers they missed earlier in the game, it doesn't come down to Lee's make or miss at the end of the game.

It all sounds reasonable, doesn't it? It did to me, too, at first.

Except how do we square this line of thought with the end of Game 1, which the Lakers won going away? At the very end of the game, the Lakers are leading 97-75, and they inbound the ball with the game clock showing ever so slightly more time than the shot clock. If it had been the other way around, it is one of the Great Unbreakable Rules of the game that you are not supposed to shoot, and just let time run out. But for some reason, if the shot clock isn't turned off, you get to shoot with impunity. Never mind that the Magic couldn't possibly have fired off a 22-point shot with only a couple of seconds left in the game. Anyway, with time running out, end-of-the-bench Lakers forward Josh Powell dribbles to his left and hoists up a three-pointer that amazingly goes in. It is the first three-pointer of his entire career, playoffs or regular season.

So, I don't think you'd have any problem convincing anyone that this shot was meaningless. It turned a 97-75 blowout into a 100-75 blowout. It almost certainly didn't mean much in Vegas: I'm sure the Lakers beat the spread, pretty sure that this kept the game in the under.

The problem is, if this shot is meaningless, and three points is three points, then isn't every other shot the Lakers made similarly meaningless? Are we supposed to think this shot was almost meaningless? Perhaps, if we add up enough "almost meaningless" shots, we actually get a meaningful result. Personally, I don't buy that. In terms of the actual game and series (in other words, ignoring Vegas, which probably had some incredibly tangential bet involving Powell and a trey at the end of the game), this shot was not just mostly meaningless, it was entirely meaningless.

What I'm going to propose is a kind of probabilistic importance—the idea being that points matter to the extent that the game is in doubt at the moment, to the extent that they bear on the result of the game. I've seen, as a kind of experimental thing with the NFL on some sports Web sites, a play-by-play measure of the winning probability for the team that makes the play. If the Baltimore Ravens score a touchdown, it increases their chance of winning from, let's say, 43 percent to 59 percent. And so on.

Now, imagine the same gadget being used for basketball. How much do you suppose a two-point basket is worth in the opening moments of the game, when the winning probability for both of two evenly matched teams is 50 percent? Actually, more than you might think. Suppose the standard deviation on scoring difference between the two teams is 10 points, and that teams score about a point per possession, close enough. A two-point basket is an increase of one point over what was expected for that possession, and a single point—0.1 standard deviations—is worth about 3.6 percent. In other words, that two-point basket would increase the winning percentage from 50 percent to 53.6 percent. If, on the other hand, the shot was missed, the winning percentage would drop from 50 percent to 46.4 percent. That shot is a swing of 7.2 percent, believe it or not.

Now let's consider the same shot in the closing seconds of the game. The team with the ball is down one, and is holding for the final shot. Obviously, if they make the shot, their winning probability is 100 percent; if they miss it, it's 0 percent. The percentage swing here is 100 percent, and clearly 100 >> 7.2.

But this huge swing is counteracted by the fact that in most cases, the game doesn't come down to that. Most of that time, that shot would be worth 0.4 percentage points, or 1.1, or something like that. At the very end of the game, it would be worth 0 most of the time. On average, that two-pointer would be worth 7.2 percent, just like the earlier shot was. It's sort of like the lottery: Would you rather have 35 cents, or a lottery ticket that gives you a one in 100,000,000 chance of winning 35 million dollars? On average, they're both worth 35 cents. But I think you'd have a hard time convincing yourself they're exactly the same.

So, I guess, I'm not letting Courtney Lee off the hook. Make the shot, and the winning probability swings from 50 percent (overtime) to 100 percent (game over, Magic win). Two points is two points, but I think people's intuition is right: When the points happen matters, and matters a lot.

EDIT: I corrected some of the above exposition to account for the fact that the hypothetical early-game two-pointer can be missed, which is one point lower than expected for the possession.

Secondly, here's a more self-contained example of this kind of probabilistic importance. Suppose that the two teams are evenly matched—50/50 to win each game, home or away. In a seven-game series, the swing for the series win in a Game 7 is obviously 100 percent: The team that wins Game 7 wins the series. However, Game 7 only gets played when the series goes 3-3, which happens about 31.2 percent of the time. In contrast, Game 1 gets played 100 percent of the time. However, it isn't as pivotal as Game 7: It can be shown that the Game 1 winner's odds of winning the series go from 50 percent to 65.6 percent, and the losing team's odds from 50 percent to 34.4 percent. That's a swing of 31.2 percent. So Game 1 swings the odds by 31.2 percent, 100 percent of the time, whereas Game 7 swings the odds 100 percent, 31.2 percent of the time. They therefore have exactly the same average importance, but Game 7 is obviously more important when it does get played.

The Infamous Fisher "0.4" Shot

2009-06-06T10:02:00.000-07:00

Introduction

Perhaps no playoff shot has been dissected, debated, or deconstructed as much as the "0.4 Shot" made by Derek Fisher in Game 5 of the 2004 Western Conference Semifinals between the Los Angeles Lakers and the San Antonio Spurs. The Lakers did not win the title that year (they went on to be defeated by the Detroit Pistons in five games), but the closeness of the timing and the marquee nature of the two teams, who had combined to win the last five championships, conspired to focus unprecedented attention on the game-ending jumper.

Much speculation centered around whether Fisher could humanly have caught the ball, turned around, and released the ball, all in the 0.4 seconds available to him. Spurs partisans insisted that he couldn't possibly have done all of those things in so short a time; Lakers fans responded that Fisher didn't do all of those things sequentially, but combined them so that he could do them all. My own personal impression (possibly colored by my bias as a Lakers fan) was that the clock started somewhat late, but not substantially so.

Fortunately, there's no need to rely on anything so nebulous as whether Fisher's shot was plausible or not. Missing from all these speculations was an examination of the actual footage. Video from the game captures instants of the game that, for the live angle at least, are equally spaced in time. The video can therefore be used as a kind of "clock" to determine the interval of time that Fisher had possession of the ball. In assembling this particular look at the Fisher shot, I used a video file that was encoded at 25 frames per second (as I determined by stepping through frames at the end of each quarter). Unfortunately, this was not the native frame rate of the original broadcast, and this increases the random error involved in timing intervals between events. It should not, however, produce any systematic bias one way or the other.

By figuring out how many frames pass between the time that Fisher catches the ball and the time he releases it, and dividing by 25 frames per second, the elapsed time can be calculated. The bottom line, for those who are impatient or don't care about analysis: About five to six tenths of a second elapsed between the time that Fisher caught the ball and the time that he released it.

The Game

On May 13, 2004, the San Antonio Spurs played host to the Los Angeles Lakers in Game 5 of the Western Conference Semifinals. After leading most of the game by as many as 16 points, the Lakers went cold from the outside while the Spurs came steadily back, eventually going ahead 71-68 on a layup by Tony Parker with a little more than two minutes left in regulation.

After a timeout, Shaquille O'Neal responded with a turnaround eight-foot jumper in the lane to bring the Lakers to within a point. The teams traded empty possessions until Kobe Bryant sank a 19-footer from the left wing on a screen by Karl Malone, putting the Lakers ahead 72-71 with 11.5 seconds remaining.

After a non-shooting foul by Derek Fisher, the Spurs inbounded the ball in their frontcourt with 5.4 seconds left. Manu Ginobili passed the ball into Tim Duncan, and tried to cut to the basket for a return pass, but got tangled up with O'Neal and was out of the play. With no other clear options, Duncan faked one way, then dribbled the other toward the top of the key, taking a blind fadeaway jumper that touched nothing but net. The clock read just 0.4 seconds.

The Lakers called timeout. Dejected and weary players trudged slowly back to the bench, none wearier than Bryant, who was exhausted not only by the 47 minutes he had played in the game, but also by the constant jetting back and forth between the team and his legal troubles in Colorado. The Lakers' play out of the timeout called for the players to stand in a stack near the top of the key, in an attempt to break out one of their stars, O'Neal or Bryant, for a quick shot or a tip-in. But before the Lakers could inbound the ball, the Spurs called a timeout. They had seen enough, they hoped, in order to defend the play well.

After the timeout, the Lakers came out in a different set, with the players scattered across the halfcourt. Players cut, especially Bryant, but with Robert Horry doubling on Bryant rather than playing Payton inbounding the ball, Payton couldn't find an open teammate and had to call the Lakers' final timeout.

When the ball was brought into play for the final time, the Lakers returned to their original set. Bryant broke out from the stack toward halfcourt, tailed by Horry and Devin Brown. O'Neal curled toward the basket, while Malone drifted toward the top of the key. Finally, Fisher broke toward Payton.

Payton tossed the ball, leading Fisher toward a spot about 18 feet from the basket on the left wing. Fisher began angling his body for the turn before catching the ball in mid-air, then coiled on his legs and prepared to shoot over Ginobili's outstretched arms. At seemingly the same instant, Fisher released the ball, the game horn sounded, and the backboard's red light came on. Nineteen thousand people held their collective breath. The ball arced upward and came down; Fisher thought he had pushed it off too hard, but it was offset just enough by his backward motion from the basket, and the ball fell perfectly through.

A hush fell over the crowd as Fisher ran down the court in celebration, eluding his mobbing teammates and streaming down the tunnel toward the locker rooms. Rasho Nesterovic and Kevin Willis waved their hands to indicate the shot got off late. Duncan stood unmoving, hoping they were right. The referees, who had called the shot good when it happened live, convened at the scorer's table to examine the video of the play from the ABC cameras. A few tense minutes passed before the referees confirmed their initial call was correct: The shot was good. The Lakers had won Game 5, 74-73, and returned home to trounce the Spurs in Game 6 to win the series, 4-2.

The Aftermath

Writing on May 14, the day after Game 5, Dusty Garza, the editor of Spurs Report, relayed news that the Spurs had filed a formal protest with the league office, claiming that the clock started too late after Fisher touched the ball, and that the shot should not have counted. The league denied the protest that same day.

Garza also offered his personal opinion that Fisher's shot did not get off in time—indeed, could not have gotten off in time—based on the notion that human reaction time is, on average, three-fourths of a second (750 milliseconds). Since the clock couldn't have started any faster than that, Garza wrote, Fisher could have had anywhere up to a bit more than a second to shoot the ball.

This seems an unreasonable conclusion. In the first place, Garza contends that the average human reaction time is three-fourths of a second, then says that unless the referees are superhuman, they couldn't possibly have pushed the button less than three-fourths of a second after Fisher touched the ball. Well, if the three-fourths of a second is an average, wouldn't half the human population be able to do it faster (assuming negligible skew in the distribution)? And presumably NBA referees are trained to be a bit faster than average.

Secondly, research indicates (Laming 1968, Welford 1980) that simple reaction time—the time required to do something simple like push a button after a visual stimulus—is more like one-fifth of a second (200 milliseconds), rather than three-fourths.

What's more, it's unclear that human reaction time is involved here at all. Bennett Salvatore once said, speaking to Henry Abbott of ESPN's TrueHoop blog, that NBA referees don't anticipate calls; they only observe the game. However, that can't possibly be literally true all the time. When Payton passes the ball in-bounds, it is immediately evident that the ball will be caught (or at least touched) first by Fisher. It is human nature to anticipate this first contact, and act accordingly. But what does it mean to "act accordingly"?

For years, the clock was operated manually, by the timekeeper, based on the rules of the game and the whistles of the referees. The system worked well most of the time, but placed a lot of reliance on the alertness of the timekeeper.

In 1999, the NBA installed a new system developed by Mike Costabile, an NCAA referee who previously officiated in the NBA. Each referee carries a small transmitter attached to his or her belt, with a button. When the clock is to start, each referee pushes the button at the exact instant at which he or she believes the ball to be in play. The first button push triggers an automatic start to the clock. The system also includes a microphone that is sensitive to the particular frequency of the whistles used by NBA referees, and stops the clock when the whistle is blown.

In order to activate the clock, at least one of the referees must push a button at the instant he or she believes the ball to be first touched. Obviously, this generally doesn't happen right at the moment the clock is "supposed" to start. There are two potential delays here: reaction time, and execution time (the time it takes the finger to actually push the button).

To understand the relationship between these two, and how the actual delay is affected by context, suppose I ask you to clap your hands as soon as one of the following happens:

A basketball I drop from four feet hits the floor.
I clap my hands.
I move my hands at all.

In the first case, it takes about half a second for a basketball released from four feet up to hit the floor. That is enough time for you to react and execute the act of clapping your hands at the precise moment the ball hits the floor. In the second case, both of us need our execution times to clap our hands, but you have to react to the start of my clapping motion. In counting the delay, your execution time is cancelled out by my execution time, leaving just your reaction time. And in the last case, I can move my hands without warning, meaning that the delay is your reaction time plus your execution time.

In the case of the play in question, it took about half a second for the ball to pass from Payton's hands to Fisher's hands. For a referee who is ordinarily alert, this is plenty of time to predict the path of the ball and press the button almost immediately upon contact. Even if we accept that referees do not anticipate events, they must at least be prepared for the potential event of contact between Fisher and the ball; there is no reason, at any rate, for the delay to be anywhere near three-fourths of a second.

But let's not be too hard on Dusty Garza. He was writing in the heat of the moment, and from honest feeling. Besides, let any of us without team favoritism cast the first stone. Let's get down to brass tacks: Garza sincerely believes that the video shows that Fisher took about a second to get the shot off. Did he?

The Video

The video file I used in putting this article together is encoded at 25 frames per second. (I determined this by advancing the video frame by frame, 200 frames, at the end of each of the four quarters, when the clock is counting down at tenths of a second. Each time, 200 frames corresponded to an interval of exactly 8 seconds, so the video must be progressing at 200/8 = 25 frames per second.) Therefore, each frame represents 1/25 = 0.04 second. This is not the frame rate of the original broadcast, which would probably have been 30 frames per second. As part of the re-encoding process, frames would lose some definition, increasing the error involved in estimating precisely when events happen. Since these errors do not accumulate, however, they add a small random error, but they do not systematically bias estimates of interval lengths.

One problem in reviewing this particular game video is that only the live shot actually keeps time accurately. In all subsequent replays, the ABC crew slowed the video at a variable rate, in order to allow Al Michaels and Doc Rivers to comment on it. But the live shot has the camera in line with Fisher and Payton, making the determination of the instant Fisher first caught the ball difficult. Here, for instance, are three successive frames of the live shot, obtained using the snapshot function of the xine video player. Which one of these frames do you think shows Fisher actually catching the ball?

I think it's pretty clear that this angle can't be used to reliably determine when Fisher touched the ball (which would have started the clock). Fortunately, we can still make use of other camera angles. This has precedent in the NFL, in which "composite" video reviews are conducted. This allows the referee (and video replay official) to assess multiple angles in order to come to a firm conclusion, even when no single angle provides all of the information necessary. This isn't to say that the NFL uses fancy three-dimensional visualization tools (à la The Matrix), since they can't do that in time, and neither is it necessary here. We'll just combine the angles mentally.

Here are video stills from the opposite baseline. It shows the ball and Fisher approaching one another. In this first frame, it's not very easy to see the ball, but you can see it superimposed on referee Joe Forte's right foot. At this point, the ball is still a couple of feet from Fisher's outstretched hands.

The frame below shows Fisher and the ball considerably closer to one another, but there still appear to be several inches in between them.

This third frame shows Fisher's hands possibly touching the ball for the first time. They don't clearly touch, but this is the first frame from this angle where contact is plausible.

Note the positions of the other players. The positioning of the left foot of Duncan (guarding Malone at the free-throw line) is especially revealing. That foot covers the free-throw line, as seen from this camera angle. Duncan's foot must therefore be in reality at least as far out as the free-throw line. It could be beyond it, if it's above the floor, but it certainly cannot be between the basket and the free-throw line. This crucial piece of video detail shows that Fisher does not touch the ball until the second of the three live video frames above. Below are successive frames from the live video, starting from the one where Fisher first touches the ball, and running until he releases it.

Frame 1:

Frame 2:

Frame 3:

Frame 4:

Frame 5:

Frame 6:

Frame 7:

Frame 8:

Frame 9:

Frame 10:

Frame 11:

Frame 12:

Frame 13:

Frame 14:

Frame 15:

In my opinion, Frame 14 (which shows the clock switching from 0.1 to 0.0) shows Fisher apparently having released the ball—about as apparently as he touches it in Frame 1. If we take those two frames as the endpoints of Fisher's possession of the ball, then he has the ball for 14 minus 1, or 13 frames in all. At 25 frames per second, that works out to 13/25 = 0.52 seconds. The method I've used to produce our composite review I estimate to have an error of a frame or two in either direction, which works out to plus or minus 0.06 seconds; add another 0.02 seconds for the video re-encoding at 25 frames per second.

In addition, I should account for my status as a Lakers fan. (Who else would go through this much trouble for Fisher's shot?) I remember sitting in bed, having twisted my ankle in my own basketball game earlier that afternoon, and feeling pretty good about the Lakers until the fourth quarter, then anxious, then frustrated, then angry, and finally elated. It is sensible, to account for this possible systematic bias, to add a frame's worth of time to the figure above to yield 0.56 seconds. Note that the ball has clearly left Fisher's hands in Frame 15, which also shows the red light on the backboard going on for the first time.

To summarize, this video shows that Fisher had possession of the ball for about 0.5 to 0.6 seconds. One corollary of this finding is that the referees started the clock approximately 0.1 to 0.2 seconds after he caught the ball. This is entirely typical and in line with usual execution times; it would be unreasonable to claim the clock was started "late." It's certainly shorter than the three-quarters of a second that Garza claimed was necessarily human reaction time; after all, Fisher executed his entire possession in less time than that.

Final Thoughts

Some Lakers fans pointed out, in the aftermath of the series, that prior to Fisher's game-winner, Duncan's shot swished through the hoop with considerably more than 0.4 seconds left on the clock. See, for instance, this video frame:

If so, claimed Lakers fans, the Lakers should have had more time on the clock, possibly rendering the above dispute moot. NBA rules stipulate that the clock should be stopped at the moment the ball exits the bottom of the basket, including the nylon, not when it enters the basket. The frame above shows the ball exiting the basket with the clock switching from 0.8 to 0.7 seconds. By that token, it must have taken somewhere between 0.3 and 0.4 seconds for the referees to whistle the clock stopped after the shot swished through. Why did it take longer for the clock to stop after Duncan's shot than it did for it to start after Fisher's contact?

It's impossible to state for certain, but one possibility is that because it's less predictable that Duncan's shot will exit the bottom of the basket than it is that Fisher will touch the ball, the referees had to wait longer to be sure that the basket was made before whistling the clock stopped. Then, too, it's a whistle blow that stops the clock, as opposed to a button press that starts it again, and those two actions may well have different execution times. But it seems plausible that had perfect timekeeping prevailed in the final seconds, Fisher's shot would have been good by about 0.1 seconds.

Superstars and the PER

2009-06-05T09:05:00.000-07:00

And now, a few words about the Player Efficiency Rating, or PER.

As a statistics guy, I am generally wary of how statistics are used in sports. This is not a matter of not believing in what I do, it's more that I know where the numbers come from, so I know what they can say and what they can't. And it drives me a little batty to see some statisticians—people who I think should know better—take too much stock in their statistics, especially if it's statistics that they had a hand in crafting.

Take, for instance, the PER, which has its roots in the Sabermetric movement in baseball and is the basketball equivalent of OPS (On-Base Percentage plus Slugging Average). Roughly speaking, we can divide all basketball statistics into two broad groups. One group consists of raw observables, such as steals, blocks, minutes played, three-pointers attempted and made, and so forth. PER does not fall into this category.

PER falls in the second category of aggregate statistics, which are combinations (often but not always linear combinations) of other statistics. As a way of accounting for all the various things that a player might do to help his team out, PER combines a slew of raw observables into a formula, which reduces to a single number. There is no unique PER formula, but the most popular one was developed by John Hollinger. Its output is normalized, so that the league average is 15. Hollinger has developed a heuristic for judging players based on PER:

A Year for the Ages: 35.0
Runaway MVP Candidate: 30.0
Strong MVP Candidate: 27.5
Weak MVP Candidate: 25.0
Bona Fide All-Star: 22.5
Borderline All-Star: 20.0
Solid 2nd Option: 18.0
3rd Banana: 16.5
Pretty Good Player: 15.0
In the Rotation: 13.0
Scrounging for Minutes: 11.0
Definitely Renting: 9.0
On the Next Plane to Yakima: 5.0

Forget for the moment the bottom end of the ranking. By definition, the average player has a PER of 15, and since there are so many apparently average players in the NBA, there should be a lot of players around 15, and there are.

What about the top end? We would expect that there would be precious few players beyond a PER of 25 for any given season, and that turns out to be true. Doesn't that on its own mean that PER is a good measure of player performance?

On its own, no. The PER formula is not derived from first principles; it's an individual attempt to capture the effectiveness of a player, and as such is a carrier of the arbitrary priorities of the PER designer. One could also design PER to positively weight turnovers, missed shots, and personal fouls, and still have most of the players in the league around 15, and a precious few above 25. Only now it would be the very worst players who would show up at the top. That's an extreme example, of course—no one would actually design PER that way—but all that means is that the arbitrary nature of PER is more constrained.

To see what I mean, suppose for the sake of simplicity that we're only interested in capturing two raw observables: points and rebounds. Let's look at a few hypothetical players.

Wade James: 30 points, 5 rebounds
Howard Williams: 20 points, 14 rebounds
Chris Bryant: 28 points, 8 rebounds

And suppose also that the league average for points is 10 and the league average for rebounds is 5. So one possible formula for PER would be points + rebounds. It's easy to see that the league average for this PER would be 10 + 5 = 15. By this measure, Wade James has a PER of 35, Howard Williams a PER of 34, and Chris Bryant a PER of 36. So Bryant has the highest PER. But it's close.

It's so close, in fact, that it's almost an incidental consequence of the way we designed PER. If we wanted a higher weighting for rebounds and a lower one for points, we could have another formula for PER: 0.5 × points + 2 × rebounds. In that case, the PERs would be 25 for James, 38 for Williams, and 30 for Bryant. Here, it's a runaway for Williams. Or, we could do the reverse, and make the formula 1.25 × points + 0.5 × rebounds. Then the PERs would be 40 for James, 32 for Williams, and 39 for Bryant, and James now has the highest PER. In all these cases, the league average PER is 15, and yet any of the three superstars could end up on top, depending on which PER formulation you choose.

There is, in mathematics, the notion of vector domination. In these terms, one player dominates another if none of his statistics are lower than the other's, and at least one is higher. For instance, 20 points and 6 rebounds dominates 14 points and 4 rebounds, and is in turn dominated by 25 points and 6 rebounds. None of them dominates, or is dominated by, 28 points and 3 rebounds. It can be shown that with any sensible definition of PER, in our limited context where we're only interested in points and rebounds, if one player dominates another, his PER is guaranteed to be higher. That's not surprising, since there should be no doubt that he's better, if we only care about points and rebounds.

Note that none of our three hypothetical players is dominated by any of the others. That's almost inevitable when you're comparing superstars. Because they're superstars, chances are good that each one does one thing better than all the rest, which means that no superstar can dominate another. Superstars will dominate the majority of players in the league, but not each other. As a result, one can define PER in such a way to put almost any given superstar on top, and which one ends up on top says as much (if not more) about the PER designer's predilections for skills as it does about the top players.

The crazy thing is that PER is probably very good indeed for comparing journeyman players, and Hollinger routinely uses it for that. But most PER fans don't seem to be interested in that. They only want to compare the top players with PER, and as you've just seen if you read this far, I think it's a pretty subjective way to do that. But most people associate statistics with objectivity, and people with subjectivity, with the end result that (a) fans of the player that ends up with the highest PER lord over fans of the other stars, and (b) those fans of the other stars start hurling invectives and accusations of bias at the PER designer (usually Hollinger). I can't count the number of times Hollinger has been called a Lakers hater just because Kobe Bryant doesn't end up with the highest PER.

To be fair, I think Hollinger brings some of that on himself, since he himself uses PER to compare the top players. Although I think he should know better than to do that, I don't really blame him; if I designed a PER, I'd probably use it for that, too.

Which is the very first reason I've never been tempted to design a PER.

EDIT: Here's a graphical representation of the various PER formulas in our hypothetical scenario. (Click to enlarge.) Points are plotted along the vertical axis, and rebounds along the horizontal axis. The green lines represent "iso-PERs": lines along which the initial PER is constant, at either 15, 25, or 35. Red lines represent the rebound-heavy iso-PERs, and blue lines represent the scoring-heavy iso-PERs.

The NBA Finals and the 2/3/2 Format

2009-06-04T11:24:00.000-07:00

I'm not going to talk about these upcoming NBA Finals, actually. I want to, but I also want to avoid the wrath of the Gods of Woof.

Instead, I'm going to say a few words about the so-called "2/3/2 format" of the NBA Finals, which refers to the sequence of venues for the seven games. As in, the first two games and last two games are played at the court of the team with the better record, while the middle three games are played at the court of the team with the worse record. (Usual tiebreakers apply.) This is in contrast to the other 14 series in the NBA playoffs—and, for what it's worth, every series in the NHL playoffs—which use the 2/2/1/1/1 format. You'll excuse me if I don't spell out in gory detail how that format goes.

Each year, at around this time, we read and hear the same time-worn opinion pieces about how the difference in format (which is intended to minimize travel for the teams) affects the chances for the two teams. Some think that the difference favors the underdog, because if home-court advantage holds for the first five games, the favorite has to return to its own court having to win the last two games. Others think the difference favors the favorite, because to get to that point after five games, the underdog has to win three games in a row.

The first thing I want to do is dispense with this notion that the format confers any kind of inherent advantage to either team. Occasionally, one sees it pointed out that the odds for the two teams are unaffected by the difference in format. One relatively simple way to see this is that the result of the series is not changed at all if we somehow force both teams to play the full seven games, even if the series is already decided before that point. (I'm sure Madison Avenue is all for that.) Since the favorite hosts four games and the underdog hosts three, no matter which format is used, the odds should be the same.

It's important to note that this line of reasoning assumes that the game results are independent of each other. If the results of earlier games can statistically affect the results of later games, that argument loses force and it becomes quite possible that the series result could in fact be affected by the format.

In this light, one interesting observation that I haven't seen before (and it might just be that I haven't looked hard enough) is that the two formats are identical except for one small change: The venues for Games 5 and 6 are switched. Otherwise, Games 1, 2, 3, 4, and 7 are played in the same place in both formats. So let's restrict our analysis of the format to just those two games.

What are the possible situations going into Game 5? We can eliminate series sweeps, because in those cases, Game 5 never gets played. So either the series is tied 2-2, or else one team is up 3-1. And let's suppose that we believe in the notion that players tighten up under pressure (which we'll assume is the case if they're playing to stay in the series), lowering their winning percentage. What effect does the difference in format have under these assumptions?

If the series is 3-1, the team that's down is under the gun for both Games 5 and 6, and the difference in format doesn't have much effect at all. If, however, the series is tied 2-2, there's no more pressure on one team than the other in Game 5, but the loser of Game 5 has the pressure on them in Game 6. Now, look at this from the perspective of the underdog. Let h represent their winning percentage at home, v their winning percentage on the road, and µ the effect of pressure. In the 2/2/1/1/1 format, with Game 5 on the road for the underdog, the expected number of wins in Games 5 and 6 is then

E[wins] = v (1 + h + µ) + (1 - v) (h - µ) = v + h + 2µv - µ

In the 2/3/2 format, with Game 5 at home for the underdog, the expected number of wins is

E[wins] = h (1 + v + µ) + (1 - h) (v - µ) = h + v + 2µh - µ

Under these assumptions, the 2/3/2 format is better for the underdog, yielding on average 2µ (h - v) more wins for them when the series is tied 2-2 going into Game 5. In ordinary terms, playing Game 5 at home gives them a better chance of taking advantage of the pressure factor in Game 6, and a lower chance of suffering from pressure themselves. What's more, the win differential goes up in proportion with both the pressure factor µ and the home-court advantage (h - v).

Of course, you should take this with a sizable grain of salt. I'm using a very simplistic model of home-court advantage and pressure. If you like, you can extend the model to let the two teams have different pressure adjustments, or even have lack of pressure (instead of pressure itself) depress winning percentage. The more important thing to take out of this is the set of factors that bear into the difference between the two formats, because under most conditions, they really aren't as different as they're made out to be.

Hello? Is this thing on?

2009-06-04T10:14:00.000-07:00

Someone is always listening, even if it's just the googlebots.

I have no really clear idea what this is going to be about. The name of the blog is supposed to conjure up images of an inoculation of statistical levelheadedness, but beyond that, who knows? I have this vague notion that I'll throw my hat in the ring on my various passions: in no particular order, astronomy, sports (basketball especially), music, poetry, the way people think and act, etc. But of course I feel no compulsion to stay within those bounds.

Deep breath. We'll see how this goes.