One More Reason to Prefer Simple Models

When building models for classification and regression, the question arises often: Go with a simple model that’s easy to understand but doesn’t have the highest accuracy? Or go with the model that’s impressively complex but much more accurate?

The needs of the situation often force one choice over another. If explainability is important, the simple model may win. If black-boxes are fine and it is all about accuracy, the complex model may be chosen. If the accuracy is roughly the same, Occam’s Razor may point to the simpler model.

I recently came across a different reason for preferring the simpler model.

In Classifier Technology and the Illusion of Progress, David Hand argues that the accuracy advantage of the complex model may not persist for long [note that he refers to the data used to train and validate the model as the “design distribution”]

The performance difference between two classifiers may be irrelevant in the context of the differences arising between the design and future distributions … more sophisticated classifiers, which almost by definition model small idiosyncrasies of the distribution underlying the design set, will be more susceptible to wasting effort in this way: the grosser features of the distributions (modeled by simpler methods) are more likely to persist than the smaller features (modeled by the more elaborate methods).

The apparent superiority of the more sophisticated tree classifier over the very simple linear discriminant classifier is seen to fade when we take into account the fact that the classifiers must necessarily be applied in the future to distributions which are likely to have changed from those which produced the design set … the simple linear classifier captures most of the separation between the classes, the additional distributional subtleties captured by the tree method become less and less relevant when the distributions drift. Only the major aspects are still likely to hold.

Data scientists are often cautioned that future data may be different from the data used for training the model. This advice isn’t new.

What I found interesting was the notion that, even when the data changes in the future, its major features are likely to hold up for longer or change more slowly than its minor features. And this, in turn, favors simpler models since they tend to use the major features.




Can Animals Perceive Human Relationships?

From the thought-provoking Beyond Words by Carl Safina:

When one individual knows another’s relationship to a third, it’s called “understanding third party relationships.”

Primates understand third-party relationships … and so do wolves, hyenas, dolphins, birds of the crow family, and at least some parrots.

A parrot, say, can act jealous of its keeper’s spouse. When the vervet monkeys that are common around camp hear an infant’s distress call, they instantly look to the infant’s mother. They know exactly who they and everyone else are. They understand precisely who is important to whom.

When free-living dolphin mothers want young ones to stop interacting with humans, the mothers sometimes direct a tail slap at the human who has the baby’s attention, signaling, in effect, “End the game; I need my child’s attention.”

When the dawdling youngsters are interacting with dolphin researcher Denise Herzing’s graduate assistants, their mothers occasionally direct these – what should we call them: reprimands?- at Herzing herself. This shows that dolphins understand that Dr. Herzing is the leader of all the humans in the water.

For free-living creatures to perceive rank-order in humans – just astonishing.

I am only about a third through the book but it is already changing the way I interact with my 5-year-old Labrador Retriever Google.

What an Educated Person Should Know – Steven Pinker

[Via Farnam Street, one of  my favorite blogs]

Steven Pinker of Harvard University writes:

It seems to me that educated people should know something about the 13-billion-year prehistory of our species and the basic laws governing the physical and living world, including our bodies and brains. They should grasp the timeline of human history from the dawn of agriculture to the present. They should be exposed to the diversity of human cultures, and the major systems of belief and value with which they have made sense of their lives. They should know about the formative events in human history, including the blunders we can hope not to repeat. They should understand the principles behind democratic governance and the rule of law. They should know how to appreciate works of fiction and art as sources of aesthetic pleasure and as impetuses to reflect on the human condition.

On top of this knowledge, a liberal education should make certain habits of rationality second nature. Educated people should be able to express complex ideas in clear writing and speech. They should appreciate that objective knowledge is a precious commodity, and know how to distinguish vetted fact from superstition, rumor, and unexamined conventional wisdom. They should know how to reason logically and statistically, avoiding the fallacies and biases to which the untutored human mind is vulnerable. They should think causally rather than magically, and know what it takes to distinguish causation from correlation and coincidence. They should be acutely aware of human fallibility, most notably their own, and appreciate that people who disagree with them are not stupid or evil. Accordingly, they should appreciate the value of trying to change minds by persuasion rather than intimidation or demagoguery.

I believe (and believe I can persuade you) that the more deeply a society cultivates this knowledge and mindset, the more it will flourish.

Pretty much sums it up. If every person was educated this way, I have no doubt that the world will be an immeasurably better place.

Imagine if politicians around the world just did these two things!

They should appreciate that objective knowledge is a precious commodity, and know how to distinguish vetted fact from superstition, rumor, and unexamined conventional wisdom.

… they should appreciate the value of trying to change minds by persuasion rather than intimidation or demagoguery.

When I grade my own education against Pinker’s list, I can see some big holes, particularly in areas like history, culture, law and the arts. Definitely need to fix them. And it has never been easier, thanks to MOOCs.


The Taste Of This Meal Is Affected By The Room We Sit In

From The Laws of Simplicity by John Maeda.

I once met a designer friend in a quiet Paris flat with white walls, white surfaces, and white furniture. A lunch of aesthetically prepared sushi was served. Red tuna, pink salmon, white squid, silvery mackerel, and a sliver of green leaf boldly engaged my visual senses as I took the entire scene into my mind. I reached to my chopsticks to begin, when my friend said, “The taste of this meal is affected by the room we sit in.”

True. With everything around me in pure white including the plate upon which the sushi was served, the thin slabs of raw fish atop the fist-sized mass of white rice appeared to float in space. I could imagine the taste to be very different in an environment that was appointed with different dishes, table, overall decorum and even different people.

The quality of our interaction with what’s in the foreground may be heavily affected by what’s in the background.

I can relate to this and can think of many examples from my daily life. When I am eating, if the tablecloth under my plate is wrinkled, it does tend to irritate me, in an under-the-radar, nagging sort of way. When I am working with my laptop, if there’s clutter around it on my desk, same thing. I also think that if my work isn’t going well, the frustration from that combines with the “background irritation” to make it feel worse.

Apart from being aware that this happens, what else can we do?

If given an empty space or an empty room, technologists would invent something to fill the expanse; similarly, businesspeople would not want to pass up a potential lost opportunity.

On the other hand, a designer would choose to do their best to preserve the emptiness because of their perspective that nothing is an important something. The opportunity lost by increasing the amount of blank space is gained back with enhanced attention to what remains.

… When there is less, we appreciate everything much more.

Maeda asks us to surround something with nothing. Make the background (whatever it is) clean, simple, uncluttered, with plenty of “white space” so that there’s nothing to distract us and the contrast between it and the object of our attention is high.

Interesting advice. I will give it a try.