Blegg Mode

As part of a series—ah, Sequence—of posts explaining the hidden Bayesian structure of language, Eliezer Yudkowsky discusses a parable about factory workers faced with the task of sorting objects which very strongly tend to either be blue, egg-shaped, furry, flexible, opaque, luminescent, and vanadium-cored (categorized by the workers as "bleggs"), or red, cube-shaped, smooth, hard, translucent, non-luminescent, and palladium-cored (categorized by the workers as "rubes").

I want you to imagine that you're a worker in this factory, and occasionally, an object comes down the conveyor belt that's blue, roughly egg-shaped, and furry, but also hard (unlike the typical blegg, which is slightly flexible to the touch). If such objects are extremely rare, you might not notice them at all—you'd quickly categorize each one as a blegg and toss it in the blegg bin without a second thought. But as these unusual hard bleggs start to become more common, you notice them, get curious, and take the time to examine one.

You make a startling discovery—this object was originally a smooth, hard red cube, of which someone had sanded down the corners to approximate an egg shape, and ironed on a layer of blue faux fur. You show your work to Susan the Senior Sorter.

"Wow," she says, "someone sure has gone to a lot of trouble to make these rubes look like bleggs!"

"Hold on," you say, "I'm not sure we should be disrespecting that effort by calling them rubes. The categories were made for man, not man for the categories: there's no rule of sorting saying that we should call them rubes, and there are plenty of rules of human decency saying that we should call them bleggs. And at a glance, they look like bleggs—I mean, like the more-typical bleggs."

Susan rolls her eyes at you, but apparently doesn't care enough to argue about it, so the two of you agree to call the modified hard objects adapted bleggs and get back to work.

Further investigation reveals that 90% of the adapted bleggs—like 98% of rubes, and like only 2% of non-adapted bleggs—contain fragments of palladium.

As the days go on, you find yourself taking notice of adapted bleggs—now that you're aware of their existence, they're not too hard to spot (although you have no way of knowing how many successfully "passing" adapted bleggs you've missed), and you need to take them to the sorting scanner so that you can put the majority of palladium-containing ones in the palladium bin (formerly known as the rube bin). You notice that—despite having insisted on the neutral-valence adjective adapted to describe the modified objects rather than something pejorative like counterfeit—you don't really put them in the same mental category as bleggs: they seem to occupy a third category in your ontology of sortable objects.

You ponder what this matter has taught you about the nature of categorization: what kind of structure does a population of entities need to exhibit in order for an efficient cognitive architecture to find it profitable to reify it as a distinct category of entity? (This job is so boring that you need to do philosophy of cognitive science to keep your mind occupied while you sort.)

After some thought, you conjecture that it probably has something to do with having cheap-to-detect features that correlate with more-expensive-to-detect features that are decision-relevant with respect to the agent's goals—

A few (non-adapted) bleggs are purple rather than blue, but are very nearly like ordinary bleggs in all other aspects, so it feels more intuitive to think of them as oddly-colored bleggs rather than their own category of object: their easily-observed deviant color doesn't let you make significant inferences about anything you care about. (While "only" 95% of purple bleggs contain vanadium ore, as compared to 98% of standard-color bleggs, the three percentage-points difference doesn't seem like a big deal.)

Likewise, 2% of otherwise-entirely-ordinary bleggs contain palladium, but you have no way of knowing this without taking them to the sorting scanner (which is finicky to start up and takes a minute to run): their metal content is of great practical interest, but seems like a rare, unpredictable fluke, unrelated to any other feature that you might hope to use to distinguish a new category of sortable object.

In contrast, adapted bleggs are both easily identifiable and the difference matters to your decisionmaking: a distinction that makes a difference, something your brain wants to have an efficient representation so that you can attend to it.

You're pleased with the iota of philosophical progress you seem to have made, and will be sure to be on the lookout for more applications of it.