The Irrationality of Humans

I find human behavior and decision making wildly fascinating...and mostly comical, especially my own. This is because of the variety of paradoxes, cognitive biases, and irrationalities that are constantly at play in our micro and macro conclusions.

In fact, these irrational behaviors are largely what led me into the career I have today. It all started with economics and statistics (my first graduate degree/love) and I've spent ten years working in understanding human behavior. Maybe the methods and engineering are fancier now than back then but, at my core, I am still fascinated by human behavior, data, and statistical inference—the methods that provide a glimpse of understanding about a person's choices.

What I've found is that humans are just kind of weird in their behaviors and decision making. Some would say down right irrational...I certainly would.

Much of that irrationality comes from all sorts of paradoxes and cognitive biases that are well known in different academic disciplines (particularly behavioral economics and statistics) but that aren't as well known by most folks. So I thought I'd write briefly about them and why they're important.

It's worth emphasizing that even if you are aware of these things, you are still quite likely to be subject to them or make the mistakes they call out—I do all of the time. Regardless, knowing them can—to some degree—help you mitigate them and help you make better decisions.

I would like to over-emphasize that my own decision making is nothing to brag about but I decided to write this because I thought it'd be fun and I do love this topic.

Paradoxes and Biases

I have decided to explicitly order these in the order I felt is the most important. This, too, is ridden with bias but I believe that this is the order most important to me and that I've found the most useful in my personal and professional life.

00. The Dunning-Kruger Effect

The Dunning-Kruger Effect is one of the most important cognitive biases that exist, probably because of how impactful it is to everyone (i.e., we all suffer from the burden of incompetent people and it is quite likely we, too, are someone's burden).

In short, people with low ability at a given task tend to overestimate their ability and those with high ability tend to underestimate it.

If you pay attention, you will see this occurrence often...especially from those without much experience in a given area of expertise (though this is not always true).

You may see this often from people in business...run away.

01. The Double Standard

This is probably my favorite bias because people (like me, for example) commit this often and it's such a subtle yet common thing. I believe this bias to be a consequence of the Dunning-Kruger Efect mentioned above.

Definitionally, a Double Standard is "the application of different sets of principles for situations that are, in principle, the same."

I find this to be very important professionally as people often have unrealistic expectations of others that they wouldn't have of themselves. I find this comes up when folks without domain expertise are frustrated by timelines of for building various things.

Said another way, "Why does this take so long?"

As a manager, I regularly ask myself "If I were doing this, would I expect the same outcome in the same time?" and I find that this helps me better empathize and be more realistic about the outcomes.

More importantly, my colleagues probably find me more tolerable. :)

02. The Curse of Knowledge

I say this non-ironically: something is obvious when you know it, and not if you don't—so too the definition of the Curse of Knowledge.

This is something I experienced a lot in my career because people often forget all of the context they have when referencing something. Business is very jargony so when I onboard people or explain things I very explicitly try to avoid using acronyms or cryptic language. It certainly takes mental effort but it makes it much less frustrating for the other party.

Also, I find that assuming someone knows something or being surprised that someone doesn't know something can sound extremely condescending, so probably it may be best to avoid that.

03. Simpson's Paradox

I could write an entire post about Simpson's Paradox but, to keep it brief, Simpson's Paradox is a statistical phenomenon in which a correlation between two variables can be reversed by the addition of another.

But how??? Time for a graph!

Above variables and are negatively correlated, quite strongly too with a correlation coefficient of -0.74. But what if there was some other group variable which represented 5 groups, we would then be able to see:

Oh no! The exact opposite conclusion! It's worth knowing that in statistics and in life, you may never know of the existance of .

So while this is useful for regression and statistical inference, I find this paradox to be applicable to many more situations.

Simply, I may always be missing a single, critical piece of information that may flip my conclusion. So I tend to calibrate my opinions accordingly.

04. The Sunk Cost Fallacy

As elegantly written by The Decision Lab, "The Sunk Cost Fallacy describes our tendency to follow through on an endeavor if we have already invested time, effort, or money into it, whether or not the current costs outweigh the benefits."

Emotion often clutters our ability to understand the actual expected value/reward of a given thing we are putting effort into but sometimes it is in our best interest to cut our losses rather than see it through.

It rarely feels good but can often be the optimal decision.

06. Loss Aversion

Loss aversion is simply the disproportional weight a person often places on minimizing losses to acquire economic gains.

For example, someone may prefer to take $10 with 100% certainty rather than $20 with 90% certainty because the displeasure from that 10% possibility outweighs the pleasure they'd receive from the additional $10 (or expected $8 = (0.9 * 20) - 10).

This is extremely irrational and puts non-disciplined investors at a mathematical disadvantage.

07. The Gambler's Fallacy

The Gambler's Fallacy refers to the incorrect belief that a given event is more or less likely given a previous sequence of events when the event is not a function of time.

This can be seen through coin flips. If you see 5 heads flipped in a row, you may think that a tails is "due" but this is incorrect (assuming a fair coin) since coin flips are always independent (i.e., one flip doesn't depend on the next).

08. Anchoring

Pulling again from the Decision Lab, "Anchoring is a cognitive bias that causes us to rely too heavily on the first piece of information we are given about a topic. When we are setting plans or making estimates about something, we interpret newer information from the reference point of our anchor, instead of seeing it objectively."

This is often used in marketing and pricing to delude you into thinking something is on sale. :')

09. Sample Bias

Sample bias originates from statistics and is a result of a flawed collection of an intended random sample.

This is particularly common in business and Twitter where people think their customers or Twitter poll-responders are representative of the entire population.

They're not and this can often lead to very poor decision making or conclusions.

10. Assignment Bias

Assignment Bias is similar to Sample Bias in that it is a bias in the sample but is rooted in a broken assignment system. For example, imagine an experimental drug trial where the "random assignment machine" (i.e., a machine that assigns things at random) only treated the young and healthy, while that is a pathological and extreme example it highlights the issue.

By the way, it turns out that a good "random" sample is extremely hard to collect in the real world—ask the Census.

11. Self-Selection Bias

Self-Selection Bias is another form of sample bias but it's caused by the participants choosing whether or not to participate in the experiment or treatment.

Again, in the example above imagine instead that the "random assignment machine" only treated the people who wanted to be treated and not the ones that did.

Similar to the previous case it would ruin the experiment.

12. Decision Fatigue

Decision Fatigue is a phenomenon whereby an individual's decision making quality deteriorates after a long session of decision making.

In short, you get tired of making choices and you start to get sloppy. In the business and investing world, this is extremely consequential because your or your investor's money is on the line.

13. Optimism Bias

To quote Wikipedia, "Optimism Bias is a cognitive bias that causes someone to believe that they themselves are less likely to experience a negative event. It is also known as unrealistic optimism or comparative optimism."

It is good to be optimistic but it is good-er to balance it in reality.

14. Response Bias

Response Bias is both interesting and counterintuitive.

It is catch-all for the frequent tendency of participants to respond inaccurately or falsely to survey questions. This is part of the reason surveys often conflict with reality.

As a statistician, I feel surverys are kind of useful but behaviors reveal the truth. Measure behaviors.

For internet companies, you may find that user-survery metrics conflict with tracking metrics that you have for your customer. What people say and what they do are often wildly different.

15. The Accuracy Paradox

Lastly, The Accuracy Paradox is the paradoxical finding that Accuracy isn't necessarily a good metric for for measuring statistical or machine learning models.

This is because of an imbalance of outcomes.

Suppose I was trying to predict a whether someone had a rare illness (1 out of 10,000 people have it), if I predicted everyone didn't have it I'd still have 99.99% accuracy.

So, accuracy can often be quite useless as a metric for assessing the quality of things in general (though not always).

Managing the Irrational

You probably can't completely stop yourself from irrational decision making but you can possibly manage it.

My approach is simple: acknowledge the biases above, the idiosyncratic ones I have from my life exeriences, and reflect frequently. I find that this results in me changing my mind often.

This can be frustrating but I think that early reactions or understandings are often not the optimal ones so I try to put effort into reflecting so that I can just do better.

Some Advice

(this advice may or may not be useful)

Being objective is nearly impossible when it relates to human judgement, and I'm not even exactly sure what "objective" really means outside of mathematics.

But my advice on being as objective as possible is to write things down: bullet points, a simple pros and cons list, or whatever suits you can highlight flaws in your judgement and reasoning.

I find this brings me mental clarity more than anything else really. I also tend to write things down on paper the good ol' fashion way (maybe it's an old habit grounded in problem sets, who knows).

Managing cognitive biases and mitigating the adverse consequences they may hold is extremely challenging but a well worthy endeavor, as good decisions compound like great investments.

But don't worry too much if you find yourself struggling with it, we are all human after all.

-Francisco