The world loves statistics. Any and every consumer product that we use has at least some part of its time, effort and/or budget allocated to statistical research. Apparently, getting the right balance between minimum quality and maximum payoff from customers is a big deal.
Statistics in itself is a pretty useless thing. What is useful is the inferences that can be drawn from the loads and loads of data that is with us. Archives lying in a big library or in a database on some server need to be analysed in order to find patterns in it. These patterns can be extrapolated to get useful insights, which can be used to improve existing processes for better results etc.
It might seem too complex, but we as individuals do the same thing. When we need to get tickets to some game, we first contact friends who watch too many games and are cooperative. If these ones can't get it, we look for the ones having either one of these qualities. And if it still does not work, we just contact as many friends as possible and hope we get tickets anyhow. By the way, this was just an example, and not the best one. Buying tickets, either online or offline works too :-)
The catch here is that like in life as usual, perspective matters more than actual information. Sometimes we keep worrying about silly, nonexistent things, while other times, ignorance is bliss and it helps us coast through almost anything.
In the same way, in statistics, what you look at and what you ignore will be different from what someone else looks at and ignores. This difference will ensure that what different people learn from the same information and how they use it will be different. (I am trying to shorten my sentences, its a work in progress)
For example, the amount of time that we spend on working or studying etc. is a heuristic. The more it is, the better results we expect to get. But is this heuristic accurate? Mostly: Yes, always: No. For someone trying to uproot weeds off their garden, the amount of time spent on it is a very good indicator of progress. But for someone researching on something, it might not be as good an indicator because it usually takes a few weeks to a few months (sometimes even years) of work just to know that you've been wasting your time on a dead end.
So, for the purpose of this blog post, Moneyball is an approach to look into better heuristics rather than the statistical status quo. The rest of this post is me coming up with some examples on use of better heuristics.
Decisions we make as individuals
It is obvious that the amount of work we put in, or the money we invest, or the things that we do are directly related to the result that we get.
What is not really obvious is something that Paul Graham says: if you are doing what everyone else is doing, you will get what everyone will get. Too bad I was unable to locate the exact place where he wrote that, so no link and not an exact quote.
Going against the wind, or doing something that most people are not doing, is normally thought of as something silly. And it sometimes is. But it is the only way to get unusual results. Had Mark Zuckerberg and his friends joined some silly software company after graduating, instead of continuing on their dorm room project, we would have no facebook. They would be a normal programmers, working for a paycheck and blogging about ruby-on-rails or something on weekends. And I would have saved a lot of time from not having facebook. Not exactly right, because in that case some other social website would have come up.
This is the reason why politicians and advertisement campaign managers always want something different in terms of content. Same old stuff never gets attention.
The need for reinventing ourselves
A big corollary of trying-to-do-different is the need to constantly reinvent ourselves. In most sports, a new player will play their heart out, but get out of form after a few games. Reason: they have now been analysed by other teams/players.
You know something thats really good at reinventing itself? Viruses. Almost everyone has heard that flu has no medication. Some say existing common cold medication is just an attempt at fighting the symptoms and waiting for the virus to subside by itself. You know why this happens? Because every organism has a certain genetic structure. It usually takes some time (a few hundred generations at least) before any organism evolves. But in case of viruses, it happens within a few generations. Add to that a few days' or weeks' lifecycle, and drug research, something that takes at least a few years for a drug to come up, just can't keep up.
Decisions we make when we choose other individuals
Here are two situations when we get the chance to choose other individuals: choosing a suitable mate and choosing a employee,coworker or business partner.
The choosing-a-suitable-mate part makes me seem like a student of anthropology, but all I know about it is the spelling: a-n-t-h-r-o-p-o-l-o-g-y. Apart from that, have you ever noticed how animals, including humans choose mates for reproduction? We all have some heuristic or a set of heuristics that we look for. In humans, the most used factors are beauty and resourcefulness in that order. While these would have been good about two hundred years before now, these are not exactly deal makers or breakers now, at least to me. In this time of technology and knowledge, intelligence and understanding should matter more, but what the hell.
Same goes for situations when we choose people to work with. This does not look like an exact science, more like a work in progress. Some people just want trust to be the sole factor for choosing employees, coworkers and business partners, while some others consider only skill or expertise. Yet some more consider previous reputation of individuals as the most important factor, and there certainly are factors I am not even thinking of. While I don't know what exactly the deciding thing should be, it is funny that people and organisations with very different choosing and hiring ideologies are successful across different fields. So, this is something that needs more thinking, certainly more than the coin toss or dice roll (take your pick) we have been doing upto now, especially when success is an important thing, or just an assumption.
Analyzing police performance
Police and its role in India has been a point of debate since British times. While people might argue about the role of police everywhere, here is something I think about them.
Police in India at least is judged on the basis of the number of cases they get and the number of cases they solve. This means that a police station getting lesser cases is one in a place with good law and order credentials. But what if the police at that place are insistent on not registering any cases just to look good? What if instead of solving cases, the police just focus on getting the opposing parties to a compromise, mostly by the use of force?
What can be a better way to judge the police's work? For one, just like the government tries to audit everything, it should audit police work too. If the crime statistics in some area are just too good to be true, they probably are. If it takes a media report or public pressure to get a case registered, surely there are more such non-cases that need attention at that place.
Funny thing, the Supreme Court's ruling in the 2006 PIL, Prakash Singh vs. the Union of India looks to be full of Moneyball-like points, but has been gathering dust in almost all states. (I can't even think of any state that is implementing it)
Two pointers, both stolen
.. Firstly, it is highly theoretical with no applications in the real world. Even the basic supply demand relationships do not work in the real world. Ask any baniya(businessman) in Delhi and he will give you a simple economics law – “The price of any good is determined by the bargaining (not purchasing) power of the consumer” If the store keeper finds you to be a new one time customer not adept at bargaining, he will charge you exorbitant prices covering his entire margins so that he can provide huge discounts to the not so friendly aunty of the neighborhood (who is also a repeat purchaser btw). This philosophy of the local kirana stores is responsible for kicking ass of all fixed price modern retailers.
Clearly, it is Moneyball at its best.
The second small thing became clear to me while I was sitting with two people who were arguing over something. Now, they were polite and both seemed right as each spoke their points, but the equal or bound-for-a-tie nature of their arguments bought me to something: it is not the right point that wins an argument, it is the louder person. Not something I would like to stress too much, but the older I get, the more office politics and yelling bosses I see, the more I support this one.