<blockquote class="twitter-tweet">you know, deploying machine learning models is very, very difficult&mdash; Francisco Javier Arceo (@franciscojarceo) <a href="https://twitter.com/franciscojarceo/status/1544110672660807680?ref_src=twsrc%5Etfw">July 5, 2022</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>


# Some History

I've been working in the application of machine learning for nearly 10 years
and it's quite exciting to have experienced firsthand how much machine learning
infrastructure has changed in that time.

I spent the first half of my career as a "modeler" where I focused on building
machine learning and statistical models for a bunch of different use cases. Over
time I found that I migrated more and more to the engineering work to get the model
deployed because I always found this to be a bottleneck in the work I did.

Concretely, I'd build a pretty good model that suggested it would be impactful
(based on expectations of the performance) but I often found it was a rather
extraordinary effort to get it live and interacting with users/customers.

This is quite well cited by the data science / MLOps community today but it was
less obvious back in 2012 and it was very frustrating. The good news is that, for
better or worse, I was rather relentless in getting my models over the finish line.

Which, in many ways, is what led me to the career I have stumbled into today as
an engineering manager working at the intersection of machine learning, data,
and engineering.

All this to say that I have spent a surprising share of the last decade working
on getting models into live product experiences. Since I have some lessons
learned and some potentially useful opinions, I thought I'd write them down.

# Ten Lessons from a decade of Deploying ML

## 0: Create strict contracts with the input source of your features

This is the single most important and most frequently omitted step in the
machine learning pipeline. I say this because making unit tests for schemas is
mostly tedious work but, in my experience, this is 90% of the errors
that come up in production. Whether it be a missing field, new field, or a
change to the type of the field, it just seems to happen to *everyone* at some
point.

Whether your model is batch/offline or real-time/online having data/schema validations
is critical as often ML models are dependent upon data from a wide variety of
different upstream sources where the data producer has no knowledge about the
usage of the data, so a trivial change on their end could make cause a world of
chaos for you and the consumers of your models.

Some limitations with this approach are (1) that you may not always know the
top-level producer of your data so making a test may not be so trivial if you
rely on an intermediate data consumer to provide you the data or if you rely on
a vendor that changes something without your notice (this happens more often
than you think). Regardless of the limitations having some tests here
will certainly provide you more coverage and ensure higher quality.

## 1: Test your feature engineering more than you want to

Feature engineering is code.

When software engineers write code for microservices we write lots of unit
tests to capture schema changes, validate edge cases, and make the code
more readable.

Unfortunately, these software design concepts don't always translate to
machine learning applications. This isn't always the case but often this happens
because the skills needed to build good machine learning models are rarely
the same as the skills needed to write good software.

So the recommendation is simple: write lots of unit tests covering all of
the edge cases that come up during feature engineering.

What are some of the edge cases?

- Return type changed
- division by zero
- incorrect logic
- incorrect imputation
- incomplete if-else
- and others

In many ways this is laborious but it makes sense because the bulk of the work
(and code) in machine learning for tabular data is in the feature engineering
and the pipelines supporting those features, so that is where the bulk of the
testing should be.

## 2: Codify the range and scale of your features

Scale and range checking in practice can be rather challenging as sometimes
it's very hard to know what the upper or lower bound will be but often you can
apply sensible heuristics. For example, you can test that features that are
calculated as ratios or rates to between [0,1] and test large but sensible
extremum values for other features.

## 3: Separate your model execution from your feature engineering

Post-estimation a model is a rather trivial function call applied to a matrix
or tensor for most algorithms. The pipeline is usually as simple as:

```
# lots of stuff above
features = build_features()
predictions = model.fit(features=features)
# lots of stuff below
```

But there are challenges that come up with the `model.fit()`. Specifically,
some models can be very large in size, especially ones using neural networks.
If you do have a large model you have to make sure you think about: right-sizing
the server that will hold the model in memory, where that model will be stored,
how the server will load the model, and what fail-over looks like.

These hardware and software considerations can get complicated on their own,
which is why it's worth not mixing up the model execution with the feature
engineering pipelines or other stuff if you don't have to.

## 4: Separate matrix serialization from model execution

This is another small step in practice but is very consequential as errors come
up and identifying the root cause can often be hairy. So I recommend that you
separate the matrix serialization from your model execution and feature
engineering because it is cleaner and you'll be able to triage inevitable
breaks sooner.

Two specific issues that come up are (1) passing a character to a matrix and (2)
passing a missing value (e.g., `np.nan`). (1) is not a number and in order for
the algorithms to do their fancy stuff they need explicit numbers and (2)
is effectively equivalent but some matrix implementations in python will allow
you to hold a placeholder for a missing value and may even behave with some
basic operations but will ultimately fail at the `model.fit()` step.

Here are two trivial examples that show this problem in action:
```python
# Just showing that this works as normal
xs = np.matrix([[1,2],[3,4]])
print(xs * 1.3)
#matrix([[1.3, 2.6],
# [3.9, 5.2]])

# Example 1: Now let's try that character
xs = np.matrix([[1,2],[3,'c']])
print(xs * 1.3)
# ...it breaks! and eventually you see this confusing error:
# numpy.core._exceptions.UFuncTypeError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U21'), dtype('<U3')) -> None

# Example 2: Now how about that np.nan?
xs = np.matrix([[np.nan, np.nan],[np.nan, np.nan]])
print(xs * 1.3)
# Wild that this works
matrix([[nan, nan],
 [nan, nan]])
```

Trivial in code, chaotic when it blows up a server.

## 5: Avoid mixing business logic with statistical operations

Machine learning systems, even those as simple as what is effectively a
glorified calculator can be surprisingly complicated so I generally recommend
keeping any business logic away from the machine learning or feature engineering
work. It's simply because both business logic and ml code tend to grow
complicated for all sorts of valid reasons but the skills used to debug or
even read and understand either are quite different, so the separation tends to
help with that.

## 6: Precompute what you can

I could write an entire book about feature engineering and the complexity that
it holds but the TL;DR version is that features tend to have all sorts of
different time representations (some feature representing real-time, others
data from yesterday, and others from 3 seconds ago) and this temporal component
ends up creating a lot of engineering complexity.

A general recommendation: precompute and run in batch as much as you possibly
can. You can actually get quite far for many use cases this way without having
to build the complexity that comes with real-time and streaming use cases.

Once you do get to real-time and streaming use cases, there's a whole other
bunch of work to do and I won't cover that here.

## 7: Load your model at service build time

In an ideal world your model is an isolated service or an isolated endpoint with
it's own memory (maybe on kubernetes, maybe not) and you simply load it at build
time. It's pretty simple but this all means you could store your model artifact
in either some [model registry](https://www.mlflow.org/docs/latest/model-registry.html)
or in an S3 buckets.

If you're early in your model building it's absolutely fine to store the model
in GitHub if it's not terribly huge but generally it's preferred to put it in
some other storage meant for larger file sizes.

## 8: Right size your server

Going back again to the challenges that come up when having a large model, you
may not just be encountering issues there. You may also find that some of the
data that you have to process in real-time can get bulky (e.g., if you're
dynamically calculating features based on user history and look back through a
year's worth of data for a bunch of different things).

So it's important to not only right size your server for the model that you will
be using but also for the features that you will be calculating in real-time.
This is usually fairly manageable and if you precompute a lot of features then
you're only doing a lookup and you'll likely have much less memory pressure but
it is worth validating this before deploying your model.

## 9: Monitor your model and features

While this is the last item, it's an extremely important one. A general truth
about models is that if you're launching one it's going to be impactful to some
process or customer experience (woo hoo!) but that means that people make
tradeoffs to get things shipped and monitoring tends to be one of them.

I've typically seen folks leave the monitoring of the model performance as some
afterthought or follow up work to be completed later and usually it does and
it's okay but it really should be top of mind. From a statistical standpoint
this is purely offensive as all sorts of chaotic things can go wrong
when you go from some historical data to a live machine.

It could turn out that your data wasn't from a representative sample, your model
was trained on the wrong metric, you're adversely impacting business metrics
(though this tends to be caught quickly), you have a bug in your code, or that
your model just doesn't work.

You can track most of that from monitoring, so don't omit it.

Most importantly, every applied machine learning model that I've built in
industry has degraded (eventually). It's mostly because things change over time
and that obviously makes sense. Without proper monitoring you will not be able
to observe this phenomena so please don't forget this core component as it could
be very consequential to your business.

It's worth mentioning that model degradation not driven by engineering problems
ends up being a lot of statistical work to understand what is driving the
decay of models. Suffice it to say that it's a complex, high-dimensional
problem.


# Some final thoughts

Reflecting on the list above, I can't help but call out that 9 of these 10 items
are purely focused on the engineering around deploying a model and that the
monitoring is the **last** step.

*As a brief aside, there is a significant amount of pre-work done here to build
these models during the model development lifecycle but I didn't discuss that
here as that's outside the scope of this blog post.*

You don't *actually* have to do anything on that list to get a model in production,
I just recommend it. At an early stage of your service it may not even make sense
to build a bullet-proof system but at a bigger scale these things actually do become
increasingly more important as preventative and defensive measures.

Lastly, I ranked these in order of importance and the most important ones are
all *preventative change controls*, i.e., they can all detect breaks before you
deploy something to production (i.e. in unit tests). Defensive change controls
are great too but one should remember that these will always come second place
to preventative controls simply because you're catching a mistake after it's in
production.


*Have any feedback on this post? Let me know on the [twitter](https://twitter.com/franciscojarceo)!*

10 lessons from a decade of deploying machine learning

The Difficulties of Machine Learning in Production


# The Irrationality of Humans

I find human behavior and decision making wildly fascinating...and mostly comical,
especially my own. This is because of the variety of paradoxes, cognitive biases,
and irrationalities that are constantly at play in our micro and macro conclusions.

In fact, these irrational behaviors are largely what led me into the career I have today.
It all started with *economics and statistics* (my first graduate degree/love)
and I've spent ten years working in understanding human behavior. Maybe the methods
and engineering are fancier now than back then but, at my core, I am still
fascinated by human behavior, data, and statistical inference—the methods that
provide a glimpse of understanding about a person's choices.

What I've found is that humans are just kind of ***weird*** in their
behaviors and decision making. Some would say down right irrational...I certainly
would.

Much of that irrationality comes from all sorts of paradoxes and cognitive
biases that are well known in different academic disciplines (particularly
behavioral economics and statistics) but that aren't as well known by most
folks. So I thought I'd write briefly about them and why they're important.

It's worth emphasizing that even if you are aware of these things, you are still quite
likely to be subject to them or make the mistakes they call out—***I do all of the
time***. Regardless, knowing them can—to some degree—help you mitigate them
and help you make better decisions.

I would like to over-emphasize that my own decision making is nothing to brag
about but I decided to write this because I thought it'd be fun and I do love
this topic.

# Paradoxes and Biases

I have decided to explicitly order these in the order I felt is the most
important. This, too, is ridden with bias but I believe that this is the
order most important to me and that I've found the most useful in my
personal and professional life.

## 00. The Dunning-Kruger Effect

[The Dunning-Kruger Effect](https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)
is one of the most important cognitive biases that exist, probably because of how
impactful it is to everyone (i.e., we all suffer from the burden of incompetent
people and it is quite likely we, too, are someone's burden).

In short, people with low ability at a given task tend to overestimate their
ability and those with high ability tend to underestimate it.

If you pay attention, you will see this occurrence often...especially from
those without much experience in a given area of expertise (though this is not
always true).

You may see this often from people in business...run away.

## 01. The Double Standard

This is probably my favorite bias because people (like me, for example) commit
this often and it's such a subtle yet common thing. I believe this bias to be a
consequence of the Dunning-Kruger Efect mentioned above.

Definitionally, a Double Standard is "the application of different sets of
principles for situations that are, in principle, the same."

I find this to be very important professionally as people often have unrealistic
expectations of others that they wouldn't have of themselves. I find this
comes up when folks without domain expertise are frustrated by timelines of
for building various things.

Said another way, *"Why does this take so long?"*

As a manager, I regularly ask myself "If I were doing this, would I expect the
same outcome in the same time?" and I find that this helps me better empathize
and be more realistic about the outcomes.

More importantly, my colleagues probably find me more tolerable. :)

## 02. The Curse of Knowledge

I say this non-ironically: something is obvious when you know it, and not
if you don't—so too the definition of [the Curse of Knowledge](https://en.wikipedia.org/wiki/Curse_of_knowledge).

This is something I experienced a lot in my career because people often forget
all of the context they have when referencing something. Business is very
jargony so when I onboard people or explain things I very explicitly try to
avoid using acronyms or cryptic language. It certainly takes mental effort but
it makes it much less frustrating for the other party.

Also, I find that assuming someone knows something or being surprised that
someone *doesn't* know something can sound extremely condescending, so probably
it may be best to avoid that.

## 03. Simpson's Paradox

I could write an entire post about [Simpson's Paradox](https://en.wikipedia.org/wiki/Simpson%27s_paradox)
but, to keep it brief, Simpson's Paradox is a statistical phenomenon in which a
correlation between two variables can be reversed by the addition of another.

But how??? Time for a graph!

![Simpson's Paradox: Negative Correlation](simpsons_paradox_1.png)

Above variables $X$ and $Y$ are negatively correlated, quite strongly too with
a correlation coefficient of -0.74. But what if there was some other group
variable $Z$ which represented 5 groups, we would then be able to see:

![Simpson's Paradox: Positive Correlation](simpsons_paradox_2.png)

Oh no! The exact opposite conclusion! It's worth knowing that in statistics and
in life, you may never know of the existance of $Z$.

So while this is useful for regression and statistical inference, I find this
paradox to be applicable to many more situations.

Simply, I may *always* be missing a single, critical piece of information that
may flip my conclusion. So I tend to calibrate my opinions accordingly.

## 04. The Sunk Cost Fallacy

As elegantly written by [The Decision Lab](https://thedecisionlab.com/biases/the-sunk-cost-fallacy),
"The Sunk Cost Fallacy describes our tendency to follow through on an endeavor
if we have already invested time, effort, or money into it, whether or not the
current costs outweigh the benefits."

Emotion often clutters our ability to understand the *actual* expected
value/reward of a given thing we are putting effort into but sometimes it is
in our best interest to cut our losses rather than see it through.

It rarely feels good but can often be the optimal decision.

## 06. Loss Aversion

Loss aversion is simply the disproportional weight a person often places on
minimizing losses to acquire economic gains.

For example, someone may prefer to take \$10 with 100% certainty rather than
\$20 with 90% certainty because the displeasure from that 10\% possibility
outweighs the pleasure they'd receive from the additional \$10 (or expected
\$8 = (0.9 * 20) - 10).

This is extremely irrational and puts non-disciplined investors at a mathematical
disadvantage.

## 07. The Gambler's Fallacy

The [Gambler's Fallacy](https://en.wikipedia.org/wiki/Gambler%27s_fallacy)
refers to the incorrect belief that a given event is more
or less likely given a previous sequence of events when the event is not a
function of time.

This can be seen through coin flips. If you see 5 heads flipped in a row, you
may think that a tails is "due" but this is incorrect (assuming a fair coin) since
coin flips are always independent (i.e., one flip doesn't depend on the next).

## 08. Anchoring

Pulling again from [the Decision Lab](https://thedecisionlab.com/biases/anchoring-bias),
"Anchoring is a cognitive bias that causes us to rely too heavily on the first
piece of information we are given about a topic. When we are setting plans or
making estimates about something, we interpret newer information from the
reference point of our anchor, instead of seeing it objectively."

This is often used in marketing and pricing to delude you into thinking
something is on sale. :')

## 09. Sample Bias

Sample bias originates from statistics and is a result of a flawed collection
of an intended random sample.

This is particularly common in business and Twitter where people think their
customers or Twitter poll-responders are representative of the entire
population.

They're not and this can often lead to very poor decision making or conclusions.

## 10. Assignment Bias

[Assignment Bias](https://www.statisticshowto.com/assignment-bias/#:~:text=Assignment%20bias%20happens%20when%20experimental,people%20who%20are%20significantly%20smarter.)
is similar to Sample Bias in that it is a bias in the sample
but is rooted in a broken assignment system. For example, imagine an
experimental drug trial where the "random assignment machine" (i.e., a machine
that assigns things at random) only treated the young and healthy,
while that is a pathological and extreme example it highlights the issue.

By the way, it turns out that a good "random" sample is extremely hard to
collect in the real world—ask the Census.

## 11. Self-Selection Bias

[Self-Selection Bias](https://en.wikipedia.org/wiki/Self-selection_bias)
is another form of sample bias but it's caused by the participants choosing
whether or not to participate in the experiment or treatment.

Again, in the example above imagine instead that the "random assignment
machine" only treated the people who wanted to be treated and not the
ones that did.

Similar to the previous case it would ruin the experiment.

## 12. Decision Fatigue

[Decision Fatigue](https://en.wikipedia.org/wiki/Decision_fatigue) is a
phenomenon whereby an individual's decision making quality deteriorates after a
long session of decision making.

In short, you get tired of making choices and you start to get sloppy. In the
business and investing world, this is extremely consequential because your or
your investor's money is on the line.

## 13. Optimism Bias

To quote Wikipedia, "[Optimism Bias](https://en.wikipedia.org/wiki/Optimism_bias)
is a cognitive bias that causes someone to believe that they themselves are less
likely to experience a negative event. It is also known as unrealistic optimism
or comparative optimism."

It is good to be optimistic but it is *good-er* to balance it in reality.

## 14. Response Bias

[Response Bias](https://en.wikipedia.org/wiki/Response_bias) is both interesting
and counterintuitive.

It is catch-all for the frequent tendency of participants to respond
inaccurately or falsely to survey questions. This is part of the reason surveys
often conflict with reality.

As a statistician, I feel surverys are *kind of* useful but behaviors reveal the
truth. Measure behaviors.

For internet companies, you may find that user-survery metrics conflict with
tracking metrics that you have for your customer. What people say and what they
do are often wildly different.

## 15. The Accuracy Paradox

Lastly, [The Accuracy Paradox](https://en.wikipedia.org/wiki/Accuracy_paradox)
is the paradoxical finding that Accuracy isn't necessarily a good metric for for
measuring statistical or machine learning models.

This is because of an imbalance of outcomes.

Suppose I was trying to predict a whether someone had a rare illness (1
out of 10,000 people have it), if I predicted everyone didn't have it I'd still
have 99.99% accuracy.

So, accuracy can often be quite useless as a metric for assessing the
quality of things in general (though not always).

# Managing the Irrational

You *probably* can't completely stop yourself from irrational decision making
but you can *possibly* manage it.

My approach is simple: acknowledge the biases above, the idiosyncratic ones I have
from my life exeriences, and reflect frequently. I find that this results in me
changing my mind often.

This can be frustrating but I think that early reactions or understandings
are often not the optimal ones so I try to put effort into reflecting so that I
can just do better.

# Some Advice

(this advice may or may not be useful)

Being objective is nearly impossible when it relates to human judgement, and I'm
not even exactly sure what "objective" *really* means outside of mathematics.

But my advice on being as objective as possible is to write things down:
bullet points, a simple pros and cons list, or whatever suits you can highlight
flaws in your judgement and reasoning.

I find this brings me mental clarity more than anything else really. I also
tend to write things down on paper the good ol' fashion way (maybe it's an
old habit grounded in problem sets, who knows).

Managing cognitive biases and mitigating the adverse consequences they may hold
is extremely challenging but a well worthy endeavor, as good decisions compound
like great investments.

But don't worry too much if you find yourself struggling with it, we are all
human after all.

-Francisco



A brief review of some of my favorite paradoxes and cognitive biases.

Paradoxes and Cognitive Biases


It's always fun to learn some additional tricks in math and computer science and I recently started reading about digital representation of numbers and [Swarthmore](https://www.swarthmore.edu/NatSci/echeeve1/Ref/BinaryMath/NumSys.html#:~:text=In%20summary%3A-,bit,numbers%20from%200%20to%20255.) has some truly great material on this.

I was searching around the internet on how to programatically convert decimal numbers to binary and I found there wasn't *that* great of a resource and stackoverflow had mixed stuff so I thought I'd put it something together quickly.

I'll probably add to this tutorial later but I think the code is the most important part.

# The basics

In short, we want to represent a decimal number (i.e., a number represented in base 10 via the 10 digits we use [0-9]), e.g., 86, as a binary number (i.e., a number represented in base 2 via the 2 digits we are then limited to [0 and 1]).

As outlined in the tutorial above, this results in the two representations:

$$86_{10} = 1*64 + 0*32 + 1*16 + 0*8 + 1*4 + 1*2 + 0*1$$

and

$$86_{10} = 1*2^6 + 0*2^5 + 0*2^3 + 1*2^2 + 1*2^1 + 0*2^0$$

Which is equivalent to saying

$$86_{10} = 1010110_2.$$

The subscript $2$ denotes a binary number. Each digit in a binary number is called a bit. The number 1010110 is represented by 7 bits. Any number can be broken down this way, by finding all of the powers of 2 that add up to the number in question (in this case 26, 24, 22 and 21).

Cool.

So how do we do this with a computer?

# Binary Representation in Python

As we saw before, representing 86 in binary format requires us to deconstruct it from $2^n$, so with computers we can represent 86 by either recursively dividing by 2 (with some additional adjustment) or iteratively (via a while loop) doing pretty much the same thing. Here's the recursive version:

```python
def decimalToBinary(n: int) -> str:
    if n == 0:
        return ""
    else:
        return decimalToBinary(n // 2) + str(n % 2)
```

This is pretty simple and could be trivally mapped to a one-liner but for readability I'll say this is sufficient. Alternatively, we could solve this with a while loop using the following:

```python
def decimalToBinary2(n: int) -> str:
    bs = ''
    while n > 0:
        r = n / 2
        n = n // 2
        bs = str('1' if r % 1 > 0 else '0') + bs

    return bs
```

And this is also pretty simple. It's worth noting that the str() function call is required before concatenating the binary string variable (i.e., bs) because without it python will actually fail to execute the if-else logic and you will not get the behavior you are looking for.

# Decimal Representation from Binary in Python

Okay, so how do we confirm this is behaving correctly? Obviously we can just recover it.


```python
def binaryToDecimal(bs: str) -> int:
    n, r = len(bs), 0
    for i in range(n):
        r+= int(bs[i]) * 2**(n - i - 1)
    return r
```

And that's it! If you stare at this formula for a second you'll see it's just taking each bit in the string and multiplying the $i^{th}$ binary value by $2^{n-i-1}$.

We can verify that all of this works with some simple checks and comparisons:

```python
decimalToBinary(86) == decimalToBinary2(86) # 1010110 == 1010110 --> true
binaryToDecimal(decimalToBinary2(86)) == 86 # true, converted 1010110 -> 86
```

How cool is that?  I'll note that this is only accurate for positive integers but for the general case you can use the first digit to represent the sign of the numbers (with some handling for the 0 edge case).

Hope you found this interesting. As I said I'll elaborate more on this post eventually but I thought I mostly wanted to share the code as I didn't really see good examples providing this back and forth and it was something that helped me understand things more concretely.

Happy typing!

-Francisco


A tutorial on converting numbers from decimal to binary...and back!

Binary Representation of Positive Integers


> I don't know what's going on, and I'm probably not smart enough to understand if somebody was to explain it to me. All I know is we're being tested somehow, by somebody or some thing a whole lot smarter than us, and all I can do is be friendly and keep calm and try and have a nice time till it's over.

-Kurt Vonnegut

As I reflect on hitting my first official year at Fast, I wanted to write about the two most important lessons I learned.

## 1. Fast is better than perfect

There are many adages all referencing how striving for perfection interferes with progress and it's really true.

I spent most of my career at large banks (even some of the "innovative ones") and the most dramatic difference between these two types of organizations is how much Faster startups are able to move. Others have writen before on how deep this runs in the DNA of startups and seeing it first hand is so exciting. Conversely, at established financial institutions, it's just not the case. The incentives aren't there and the bureacracy and over-processing of process creates too many roadblocks to actually get things done.

Often my gripe at those organizations was that we spent more time on strategy, making presentations, meeting, than actualy doing the work. I'm very supportive of being thoughtful and planning your work, but I think something about big organizations creates an incentive to spend more time talking about the work than actually doing the work.

That always frustrated me, again, not because I'm against planning or process but rather I'm against process that interferes with progress, which I think is in place at those organizations more than people want to admit.

So why does that happen? Why do people create processes that interfere with progress?

Because it *feels* like it matters. But what *feels* good and what actually matters aren't necessarily the same thing and I think big organizations simply lose sight of this fact.

Which leads me to the other important lesson I learned...

## 2. Focus on what *really* matters

Time is a finite, precious commodity.

It's a commodity that can't be bought back so I often reflect on what time was wasted "feeling good" and what time actually resulted in change. As I get older, this becomes increasingly more important to me and I only recently realized how much time I've wasted trying to feel good rather than trying to do things that mattered, so I'm trying to course correct these days.

At Fast, I have been so happy with how focused we are on doing work that actually matters. Meetings and work streams are always grounded in action items and next steps, which just makes it all feel like we are working together. And, in truth, that just makes the whole thing fun.

We are all commonly bound by finite time and I feel lucky to get to choose where I spend it. I am grateful to have spent the last year with Fast builders--brilliant, kind, and talented people who focus on what matters.

-Francisco


A Fast Year Indeed

I wrote a little bit about my year in my [last blog post](https://franciscojavierarceo.github.io/post/2021-is-nearly-over) but now that the year is officially ending I wanted to pause for deeper reflection.

So far this year has gone completely differently than I had originally planned but I am very grateful for it. I started this past year [writing about 2020](https://franciscojavierarceo.github.io/post/learning-new-things) and what I learned, so I thought I would do the same as it closed.

The end of 2019 and 2020 was a very big year for my family, as my wife and I moved to South Dakota, changed jobs, and bought a home. So quite a lot happend in the latest trip around the sun.

# Learning

I learned a lot this year, as I always try to do. Most of my focus was on software engineering, in part because of my job, and also because of my interests. So here's the list of things I've learned:

- More Django, JavaScript, and React in general 
- More of the ever-evolving AWS ecosystem (it is both the same and very different than it was back in 2014)
- React Native, mobile development, and deploying to the App Store (this is an exercise on its own)
- The programming language Go
- gRPC (Google Remote Procedure Call)

Learning all of these things was mostly painful but also extremely fun. It takes a lot of patience and tinkering to learn to be effective with programming and I really do enjoy it. I find that I'm happiest most often when tinkering to solve new (at least new to me) computer-related puzzles, so it's been a very enjoyable year for me. In fact, we can see the data in the graph below showing all of the times I committed code:

![That's a lot of commits!](2021-commit-count.png)
A visualization of my daily computering

A fun fact here is that if you compare this to 2020, you'll notice the massive jump in Pull Requests (PRs), which is a fun aside.

# Career

I joined [Fast](https://www.fast.co) last March and it's been an absolutely wild ride, which has been both intellectually fulfilling and exciting, which is a challenging combination to sustain in my experience. 

I have to say that of all of the organizations that I have worked for, Fast has been the most fun, which is unexpected since I work from home in South Dakota. By which I mean that I thought that not being in a physical office would have had a bigger influence on my perception of my enjoyment of my job but as it turns out, while I do miss working in an office, the work that I *actually do* is more important to me than being physically next to people that I'm doing the work with (it's not lost on me that this should be obvious but I'm often surprised how irrational or illogical my brain is about things of this nature). More than anything else, I think it's the people that really do make it great. The people that build the software and services, who are the same people that are kind and fun to work with.

I've learned so much working at an extremely fast (pun intended) growing startup over the last 10 months and I really think it's unlikely I'll ever return to banking after working at an organization that values engineering as much as we do.

I've gotten to the place where I feel confident in saying that I am a full stack engineer, which means I need to figure out what new things I'd like to learn next. 

# Next Year

In 2022, I hope to tackle a few big areas professionally:

1. I still have much to learn as a software engineer, so I hope to continue learning from my amazing colleagues
2. I think we built some really awesome products during my time @ Fast, but there's still so much left so I hope to continue to build and ship impactful products
3. I want to continue iterating on things I've alreaddy built, specifically on my web and mobile app for [Unidos](https://www.getunidos.com)
4. I hope to start learning more hardware as it's an area I have somewhat limited knowledge but lots of curiousity

Let's hope I make some progress on those in 2022 and don't abandon (4). Overall I'm very grateful for this past year and I feel lucky to have spent it with the best person a guy could ask for.

Wishing you and your loved ones a Happy New Year.

-Francisco

2021: A Reflection


It has been quite an extraordinary year and it has gone by terribly quickly, but I suppose that's a natural part of the aging process.

So far this year has gone completely differently than I had originally planned but I am grateful for it, it has been quite fun.

Reflecting briefly at my goals, which as I listed in my [previous blog](https://franciscojavierarceo.github.io/post/2021-goals) post were:

```
1. Be a better husband
2. Talk less, listen more
3. Cook more intentionally
4. Speak more intentionally
5. Help others more effectively
6. Write at least 2x per month
7. Exercise ~5x per week
```

I tried to improve on all of these items over the past year (some significantly more than others), but I'm not where I wanted to be on them.

Regardless, I will say I learned a lot more this year by explicitly trying to improve these things. The goals I found most challengine were (1), (2), (5), and (6). I imagine this is because (1) is hard in general, (2) requires constant mental restraint from speaking up when I want to interject with a response or a thought, (5) doesn't truly realize its effectiveness in short time horizons, and (6) was simply not a priority.

I'm mostly okay with the realities of this except (6), since writing requires being more intentional about my time and thoughts, which disappoints me as it suggests I was more chaotic than I wanted to be. As I grow older I realize how much more important it is to be increasingly more deliberate with the allocation of my time but I always struggle with choosing things as I really do enjoy working on and learning new things.

Anyways, here's to hoping I'm more thoughtful with my time over the next couple of months and years.

Cheers,
Francisco

2021 is nearly over

## I will write less frequently

Unfortunately, I've been writting a little less frequently than I was hoping to this year; I was targetting about once a week but I think that was (1) a little too frequent and (2) a little too ambitious of me to think I'd be able to keep up that pace.

## I will write more quality

I'll aim to write once or a month or so and hope for higher quality posts. 

In the spirit of more interesting content (and more interesting problems for me to tinker with) in the Twitter thread below I decided I'd write a blog post on building a webtool that scans a user's historical tweets and identifies the highest risk ones.

<blockquote class="twitter-tweet">Is there an ML type service that can plow through history for potentially problematic tweets?&mdash; Matt Galligan (@mg) <a href="https://twitter.com/mg/status/1372923621040082944?ref_src=twsrc%5Etfw">March 19, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

I decided to work on this because (1) it's at the intersection of things I really enjoy (nlp, web development, and twittter), (2) I wanted to build a tool that would solve a real problem, and (3) I thought it'd be fun.

## Creating a Toxic Twitter Scanner

I think this tool can be broken down into 3 sub-problems:
1. Creating a web service to handle the front-end user experience (authentication and rendering of data)
2. Creating a machine learning model as an endpoint that handles predicting the toxicity/problematicness of a user's tweets
3. Connecting (1) and (2) and deploying them as a service in an automated way

There's a bunch of stuff in between and technical things to handle, but this is really what I think matters at a high-level, so let's dive in!

### 1. Creating a Web Service
As a web tool, a user should be able to authenticate with Twitter to scan their tweets and consent to the service using their data (this is to access private tweets). As a starting point, I'll probably launch this tool without any authentication and just look at a user's public tweets.

### 2. Creating a Machine Learning model as an API
This is a reasonably straightforward process as the endpoint will be a wrapper around an existing machine learning library and the goal is to be able to send in some data and get some scores back.

### 3. Connecting a Web Service to an ML API
This is an important step. In summary, we want to decouple the user-facing experience from the machine learning because they have very different technical requirements. The compute cluster that'll create predictions for the tweets will use much more memory than the user-facing application, so it's important to separate them so computing doesn't slow down the user experience.

## Progress on Building a Scanner

So far I've been diving into using a pre-trained BERT model developed by [Laura Hanu](https://laurahanu.github.io) at [Unitary](https://www.unitary.ai) and it's been pretty fun. I've been able to score my own tweets and see the behavior of this model and see various implementation details that I'll have to be mindful of (I'll probably write more on this later).

I've been tinkering with [FastAPI](https://fastapi.tiangolo.com) and figuring out how the [cookie-cutter template](https://fastapi.tiangolo.com/project-generation/) works. I don't love the way it behaves and there's a lot of strong opinions in place which is helpful but imperfect; we'll see how my opinion shapes as I spend more time with it.

I'm very excited to learn [Docker Swarm](https://docs.docker.com/get-started/swarm-deploy/) as it seems like a very promising framework for deploying the model API.

You can follow my progress on this [GitHub repository](https://github.com/franciscojavierarceo/twitter-scan). Stay tuned as I'll be updating this blog with my thoughts as I chip away at this problem.

---
*Have some feedback? Feel free to [let me know](https://twitter.com/franciscojarceo)!*

A Data Scientist's Guide to using FastAPI and BERT to Build a twitter scanner of your tweets.

Bert Transformers, FastAPI, and Toxic Twitter

I've written before about my love for [Github Actions](https://franciscojavierarceo.github.io/post/github-actions) and [Docker](https://franciscojavierarceo.github.io/post/docker-for-data-science).

A little over a week ago I tweeted that I was interested in writing a blog post about using Github Actions and Docker to schedule a job to do something interesting, but I didn't have a fun use case. 

Fortunately, my internet friends delivered.

<blockquote class="twitter-tweet">text me with a pic of your dog&mdash; Camilo (@camdotbio) <a href="https://twitter.com/camdotbio/status/1367856131972947972?ref_src=twsrc%5Etfw">March 5, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

So I decided to write a quick script to do this, and that's how I spent my Friday evening as my wife and I watched Avengers. 😂

I've provided a link to the [Github Repository](https://github.com/franciscojavierarceo/twitter-cron-demo) with the full code but here's the short instructions if you're interested.

## 1. Register for a developer account on [Twitter](https://developer.twitter.com/en/apply-for-access) and get your API credentials

This is a pretty easy process and you just sign up and outline what you're doing (it's also free). Note that you'll need to store 4 variables in a `.env` file like below:

 TWITTER_API_KEY=
 TWITTER_API_SECRET=
 TWITTER_ACCESS_TOKEN=
 TWITTER_ACCESS_TOKEN_SECRET=

## 2. Write a Python Script to Tweet a Dog Photo

This was pretty straightforward thanks to the [Tweepy](https://www.tweepy.org) library in Python and this super random [dog photo API](https://dog.ceo/dog-api/) (God bless the internet).

Here's what that Python code looks like:

```python
import os
import json
import requests
import tweepy

def get_random_dog(filename: str='temp') -> None:
 r = requests.get('https://dog.ceo/api/breeds/image/random')
 rd = json.loads(r.content)
 r2 = requests.get(rd['message'])

 with open(filename, 'wb') as image:
 for chunk in r2:
 image.write(chunk)

def main(message: str, filename: str='temp') -> None:
 auth = tweepy.OAuthHandler(
 os.environ.get('TWITTER_API_KEY'), 
 os.environ.get('TWITTER_API_SECRET')
 )
 auth.set_access_token(
 os.environ.get('TWITTER_ACCESS_TOKEN'), 
 os.environ.get("TWITTER_ACCESS_TOKEN_SECRET")
 )
 api = tweepy.API(auth)
 get_random_dog(filename) 

 try:
 api.verify_credentials()
 print("Twitter Authentication Succeeded")
 
 try:
 api.update_with_media(filename, status=message)
 print('Tweet successfully sent!')

 except Exception as e:
 print('Error sending tweet \n %s' % e)
 except:
 print("Twitter Authentication Failed")


if __name__ == '__main__':
 main("Hey, @camdotbio! 👋 \n\nHere's your daily dog photo!")
```

## 3. Create a Docker Container to run the Python Script 

This is a contentious area where some may argue docker is execessive for this but I use different computers with different operating systems so this works for me and I like it. 

In short, I created a [Dockerfile](https://github.com/franciscojavierarceo/twitter-cron-demo/blob/main/Dockerfile) and a [docker-compose](https://github.com/franciscojavierarceo/twitter-cron-demo/blob/main/docker-compose.yml) file where I run the script. The benefit of this is that I don't have to worry about this script not working on my Linux machine or not working on my Mac, it works on both!

## 4. Use a Github Action to Schedule a Cron Job

This was also straightforward and I've copied the code below:

```yml
name: Build and Deploy 🚀

on:
 schedule:
 - cron: '0 15 * * *'

jobs:
 build:
 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v2

 - name: Make envfile
 uses: SpicyPizza/create-envfile@v1
 with:
 envkey_TWITTER_API_KEY: ${{ secrets.TWITTER_API_KEY }}
 envkey_TWITTER_API_SECRET: ${{ secrets.TWITTER_API_SECRET }}
 envkey_TWITTER_ACCESS_TOKEN: ${{ secrets.TWITTER_ACCESS_TOKEN }}
 envkey_TWITTER_ACCESS_TOKEN_SECRET: ${{ secrets.TWITTER_ACCESS_TOKEN_SECRET }}
 file_name: .env

 - name: Build the docker container
 run: docker build .

 - name: Run the script 🚀
 run: docker-compose up
```
Note that you have to create [Action Secrets](https://docs.github.com/en/actions/reference/encrypted-secrets) in your Github Repository (available in the Settings tab) and add the credentials from Step 1.

## Conclusion 

And that's it! 🐶 

Pushing the code to GitHub handles the rest, isn't that wonderful?

Anyways, thanks to my internet friends for the fun idea. I didn't end up sending it to Camilo's phone number because I would have to pay Twilio $0.0075.

---
*Have some feedback? Feel free to [let me know](https://twitter.com/franciscojarceo)!*

Using GitHub Actions to run a Python script inside of a docker container scheduled daily.

Github Actions, Docker, and the Beloved Cron

# Regression

I've written about the wonders of [Linear Regression](https://franciscojavierarceo.github.io/post/ordinary-least-squares) before and one of the things I find most amazing about it is that it allows you to approximate a function.

> But what does that mean?

In short, take data about two things and estimate a relationship between them.

A classic example is Age and Income. 

Suppose you wanted to understand how a person's age *correlates* to their income. You could take a sample of data and store it in a table, spreadsheet, or even a fancy database somewhere so you could analyze it.

You could then draw a scatter plot (like below) and *visualize* the relationship between the two attributes and fit (i.e., estimate) the relationship (i.e., the slope of that line). If the relationship was strictly **linear**, we'd see a scatter plot that looks something like the graph below.

![A scatter plot!](scatterplot.png)
A Scatter Plot of Age and Income with a Linear Relationship

But what if it wasn't linear?

What if we knew Age only increased Income to a degree and that the marginal return was decreasing? Well, maybe we'd see a plot like below.

![A scatter plot!](income_age_squared.jpeg)
A Scatter Plot of Age and Income with a Quadratic Relationship

What if things were a little less intuitive and, after another point, your Income (on average) started to go back up again? We'd see the graph below.

![A scatter plot!](income_age_cubic.jpeg)
A Scatter Plot of Age and Income with a Cubic Relationship

Lastly, what if we saw something that was just plain *weird*?

![A scatter plot!](income_age_weird.jpeg)
A Scatter Plot of Age and Income with a Piecewise Linear, Discontinuous Function

This is my favorite example because it shows a [piece-wise linear function](https://en.wikipedia.org/wiki/Piecewise_linear_function) and, while strange looking, these relationships are very common phenomena in the wild.

Why?

Because often times we are modeling behaviors or decisions by other systems in the world, and those systems, decisions, and behaviors often have weird boundary points/thresholds. In the credit world, you'll often see this because a lender's credit policy systematically rejects applications with a certain set of criteria, which would lead to visualizations identical to this.

## What do we do with all of this information?

This is the fun part! 

If you're doing data science and modeling data with lots of features/attributes, you probably don't want to do this for hundreds or thousands of different variables. Instead, you may benefit from finding a way to estimate the univariate relationship algorithmically. 

But how?

When you have a bivariate relationship like this you don't want to have to tediously [engineer features](https://en.wikipedia.org/wiki/Feature_engineering) to estimate the underlying function, rather you'd prefer to have a machine learn (or estimate) the function using [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning).

But which algorithm and how do I use it? Well, there are a few options people typically use:

- [Polynomial Regression](https://en.wikipedia.org/wiki/Polynomial_regression)
- [Multivariate Adaptive Regression Splines (MARS)](https://en.wikipedia.org/wiki/Multivariate_adaptive_regression_spline)
- [Decision Trees](https://en.wikipedia.org/wiki/Decision_tree_learning)
- [Weight of Evidence](https://documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.3&docsetId=casstat&docsetTarget=casstat_binning_details03.htm&locale=en)

And each has its own unique benefits.

But my personal favorite is MARS. If we used R's [earth package](https://cran.r-project.org/web/packages/earth/earth.pdf) we can estimate this function automatically in a few simple lines of code and get the predicted fit below.

![A scatter plot!](income_age_mars.jpeg)
A Scatter Plot of Age and Income using Function Approximation

And here's the R code to generate it.

```R
library(earth)
earth.mod <- earth(income ~ age, data = df)
plotmo(earth.mod)
print(summary(earth.mod, digits = 2, style = "pmax"))
dft$preds <- predict(earth.mod, df)[,1]
```

Short, sweet, and effective—my favorite combination. 

The other algorithms all have pros and cons, and I largely recommend to use each one depending on how smooth or *not*-smooth the underlying behavior of your data is (and how much you care to really account for it). 

Anyways, I hope you liked this post; it was a very enjoyable excuse for me to make some graphs.

*Have some feedback? Feel free to [let me know](https://twitter.com/franciscojarceo)!*

A Data Scientist's guide to Function Approximation

Function Approximation using Data Science Techniques

*TL;DR: A Data Science Tutorial on the benefits of Docker.*

## Some History
I began my foray into what's now called [Data Science](https://en.wikipedia.org/wiki/Data_science) back in 2011. I was doing my first master's in economics and statistics and I was doing econometrics research on consumer demand based on survey data using [SAS](https://www.sas.com/en_us/company-information.htmlhttps://www.sas.com/en_us/company-information/profile.html).

![Look at that rise!](data-science-google-trends.png)
Looks like I graduated at an interesting time.

Technology was *much* different then, distributed computing and Open Source Software (OSS) was only starting to get the popularity and attention it has now. More practically, most businesses weren't using cloud services, they were using their own servers for storing and managing their data (i.e., real physical machines) with fixed RAM and a lot of overhead (read: chaos when the power goes out).

The most sophisticated analytical shops used SAS* to process their data since it was a very efficient way to analyze large data out of memory.

But it wasn't fault tolerant or stable. Software libraries for different mathematical frameworks have evolved so much over time and they just kept changing, so the infrastructure kept changing, too.

In short, the way data scientists did analytics was pretty brittle: most places didn't use version control or servers; code was sent via emails; and deploying models was usually done in an Oracle/MySQL table that ran a query, joins, and a sum-product. It was the Wild West.

Cloud computing and OSS changed the game. [R](https://cran.r-project.org), [Python](https://www.python.org), [Hadoop](https://hadoop.apache.org), [Spark](https://spark.apache.org), [CUDA](https://developer.nvidia.com/cuda-toolkit), and other frameworks completely influenced how we thought about that infrastructure.

Python, in particular, has been one of the greatest contributions to data science and it has truly helped push the field further.

## Python, the Beautiful

Python wasn't the original data science language, R and SAS were much more popular back in the early and mid-2010s but two popular data mining libraries helped Python skyrocket in the data science community ([sklearn](https://scikit-learn.org/stable/) and [xgboost](https://en.wikipedia.org/wiki/XGBoost)). Then in 2015 people made advances in deep learning frameworks (moving away from [Theano](https://en.wikipedia.org/wiki/Theano_(software))) and creating things like [Caffe](https://caffe.berkeleyvision.org), [Keras](https://en.wikipedia.org/wiki/Keras), [Tensorflow](https://en.wikipedia.org/wiki/TensorFlow), and eventually [PyTorch](https://en.wikipedia.org/wiki/PyTorch) (my personal favorite).

All of the stuff under the hood changed dramatically and it made the infrastructure around deploying these models change dramatically, too.

The ever-evolving software made getting data science infrastructure up and running really annoying, time consuming, and eventually kind of wasteful because it would get stale quickly, but the world has evolved again.

Cue [Docker](https://en.wikipedia.org/wiki/Docker_(software)) and the emergence of [containerization](https://hackernoon.com/what-is-containerization-83ae53a709a6).

## Docker and Jupyter Notebooks

Docker is basically a way to easily configure a mini-computer in your computer. The idea being, that if you configure it with a single file declaring what stuff (i.e., software) you need in it, you can deploy that same container to some production environment. In real, customer-facing applications even a small subversion change of a single library can break your entire service.

Docker can help mitigate that risk.

![It works on my machine!](docker.jpg)
This meme is suprisingly accurate.

What's interesting is that people have made attempts at doing similar things for a long time ([virtual environments](https://virtualenv.pypa.io/en/latest/), [Pyenv](https://github.com/pyenv/pyenv), [virtual box](https://www.virtualbox.org), etc.) and while most of them were helpful, they all still had really annoying issues come up constantly...but Docker is much more comprehensive.

So how is that relevant for Python and data scientists? Well, if you're doing data science work, odds are you're probably using Python or R and you might be doing that work using a [Jupyter Notebook](https://jupyter.org). 

The [Jupyter Project](https://jupyter.org/about) has created a wonderful [Data Science](https://hub.docker.com/r/jupyter/datascience-notebook/) docker image that allows you to trivially get up and running.

All you have to do is [install Docker](https://docs.docker.com/engine/install/) and run the following in your terminal:

```bash
docker run -it -p 8888:8888 -v /your/folder/:/home/jovyan/work --rm --name jupyter jupyter/datascience-notebook
```
You'll see some output and at the end of it you should see something like: 

```
[C 2021-02-14 14:37:06.596 ServerApp] 
 
 To access the server, open this file in a browser:
 file:///home/jovyan/.local/share/jupyter/runtime/jpserver-6-open.html
 Or copy and paste one of these URLs:
 http://7c94e4cf2dc1:8888/lab?token=this-will-be-a-magical-token
 or http://127.0.0.1:8888/lab?token=this-will-be-a-magical-token
```
Click on that link in your terminal and you should be directed to a page that looks like the image below:

![So many options!](docker-for-data-science.png)
Who is using Julia these days?

And there you have it, you can start ripping through all sorts of data in minutes!

A great benefit of this particular docker image is that it has most of the Python/R libraries you want already out of the box but if you want to add another, you can do that right in your notebook by using [Jupyter Magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) in a cell like so:
```bash
%pip install snowflake
```

![Simple install](pip_install.png)
Wow, that was easy.

And now you can use that library. See how nice it is not to have to [dual boot](https://en.wikipedia.org/wiki/Multi-booting) your computer to install something?

## Conclusion

Data science infrastructure is going to continue to evolve very heavily, so I imagine this post will be outdated in 2 years but currently this is an extremely fast and painless way to get up and running.

I can't emphasize enough how miserable managing different Operating System (OS), Python, or other software library versions really is. Debugging these things used to take days or weeks and now it's just trivial, so I'd really recommend this approach. An added benefit is that this will also save you an extraordinary amount of time when you go to deploy your model...but more on that later. 😉

*Have some feedback? Feel free to [let me know](https://twitter.com/franciscojarceo)!*

---
*As a brief side note, the history of SAS is amazing and I really recommend reading the [Wikipedia page](https://en.wikipedia.org/wiki/SAS_Institute) on it. Most of the large banks, pharmaceutical firms, several government agencies, research institutions, and other major organizations in the world still operate on SAS because it's so deeply embedded into their business. Now, that technology can no longer be decoupled from their core infrastructure, which is interesting.*

A Data Scientists Guide to using Docker containers to quickly spin up a Jupyter Notebook

How to use Docker to Launch a Jupyter Notebook

*TL;DR: A Data Science Tutorial on using K-Means and Decision Trees together.*

Customer segmentation (sometimes called [Market Segmentation](https://en.wikipedia.org/wiki/Market_segmentation)) is ubiqutous in the private sector. We think about bucketing people into $k$ mutually exclusive and collectively exhausting (MECE) groups. The premise being that instead of having 1 strategy for delivering a product or experience, providing $k$ experiences or strategies will yield much better engagement or acquisition from our customers.

Generally speaking, this makes sense; it's intuitive. Provide people a more curated experience and they will enjoy it more...and the more personalized the better. 

Netflix, Spotify, YouTube, Twiter, Instagram, and all of the big tech companies have mastered personalization by using robust, computationally expensive, and sophisticated machine learning pipelines. But the world has been doing this for a long time, just a much less sophisticated version.

So I thought I'd give a technical demo of what customer segmentation looks like in a basic way using a trick I've used for years.

Here are the things I'd like to cover during this demo:

1. What options do I have to segment my customers?
2. How do I actually do the segmentation?
3. What can I do with my new customer segments?
4. How do I know that my segments are effective?
5. How do I know when my segments have changed?

## Approaches to Customer Segmentation

The phrase "Customer Segments" tends to mean different things across different industries, organizations, and even across business functions (e.g., marketing, risk, product, etc.). 

As an example, for a consumer products retailer, they may refer to customer segments using both demographic information or their purchase behavior, where a lender may refer to their segments based on credit score bands. While very meaningfully different from a business perspective, the same algorithms can be used for both problems.

Analytically speaking, I've seen Customer Segments defined really in two main ways: (1) Business Segments and (2) Algorithmic Segments. Usually executives refer to their segments in the first category and data scientists focus on the second. The first is really important organizationally because 99% of the people working with your customers don't care about how you bucketed them and customers are the most important thing. Always.

...but how do you *actually* (i.e., in code and data) get to those segments?

### 1. Logical Business Segments
These segments tend to be defined by heuristics and things that make common sense. They are often built on things that are aligned with the goal of the business.

Here are some examples:

- The age of the customer (in years)
- The income of the customer (in dollars or thousands of dollars)
- The amount of money a customer spent in the last year
- The likelihood a customer will spend money at a given store (purchase propensity / propensity to buy)
- The customer's geographic region (e.g., zipcode, state)

In data, some of that customer information would look something like this:

<table>
 <tr>
 <th>User ID</th>
 <th>Age</th>
 <th>Customer Income</th>
 <th>Purchase Propensity</th>
 <th>...</th>
 </tr>
 <tr>
 <td>1</td>
 <td>25</td>
 <td>$45,000</td>
 <td>0.9</td>
 <td>...</td>
 </tr>
 <tr>
 <td>2</td>
 <td>30</td>
 <td>$80,000</td>
 <td>0.4</td>
 <td>...</td>
 </tr>
 <tr>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 </tr>
 <tr>
 <td>n</td>
 <td>56</td>
 <td>$57,000</td>
 <td>0.1</td>
 <td>...</td>
 </tr>
</table>
And so on.

We could apply some logic/rules/code to create segment like:

- Age Buckets
 1. < 25
 2. 25-35
 3. 35-55
 4. 55+
- Income Buckets
 1. < $25K
 2. $25K-50K
 3. $50K-100K
 4. $100-150K
 5. $150K+
- Propensity Buckets
 1. Low: [0, 0.25]
 2. Medium: [0.25, 0.75]
 3. High: [0.75, 1.0]

And map that logic into our data, which would yield
<table>
 <tr>
 <th>User ID</th>
 <th>Age Bucket</th>
 <th>Income Bucket</th>
 <th>Propensity Bucket</th>
 <th>...</th>
 </tr>
 <tr>
 <td>1</td>
 <td>25-35</td>
 <td>$25K-50K</td>
 <td>High</td>
 <td>...</td>
 </tr>
 <tr>
 <td>2</td>
 <td>25-35</td>
 <td>$50K-100K</td>
 <td>Medium</td>
 <td>...</td>
 </tr>
 <tr>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 </tr>
 <tr>
 <td>n</td>
 <td>56</td>
 <td>$50K-100K</td>
 <td>Low</td>
 <td>...</td>
 </tr>
</table>
And so on.

Pretty simple, right? The code for this categorization is simple too (assuming you're using Pandas and Python; though it's also simple in SQL).

```python
# Here's one example
import numpy as np
import pandas as pd

cdf['Income Bucket'] = pd.cut(cdf['Annual Income ($K)'], 
 bins=[0, 25, 35, 55, np.inf], 
 labels=['<25', '25-35', '35-55', '55+']
)
```
This is a really helpful and simple way to understand our customers and it's the way that most businesses do analytics, but we can do more. 😊

### 2. Algorithmic Segments

Segments defined using simple business logic are great because they are so easy to interpret, but that's not free.
By favoring simplicity we have to limit ourselves to (potentially) suboptimal segments. 
This is typically on purpose and entirely fine but, again, we can do better.

So how do we do better?

Cue statistics, data mining, analytics, machine learning, or whatever it's called this week. More specifically, we can use the classic [K-Means Clustering](https://en.wikipedia.org/wiki/K-means_clustering) algorithm to *learn* an optimal set of segments given some set of data.

To skip over many important details ([more here](https://towardsdatascience.com/customer-segmentation-using-k-means-clustering-d33964f238c3)), K-Means is an algorithm that optimally buckets your data into $K$ groups (according to a specific mathematical function called the [euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance)). It's a classic approach and tends to work quiet well in practice (there are a ton of other neat [clustering algorithms](https://en.wikipedia.org/wiki/Cluster_analysis#Algorithms)) but one non-technical challenge is (1) choosing $K$ and (2) explaining what a single cluster actually means to literally anyone else.

Solving (1) is relatively straight-forward. You can run K-means for some number of $K$ from [0, $m$] ($m > 0$ and choose what appears to be a $k$ that sufficiently minimizes the within-cluster sum-of-squares (i.e., $\sum_{i=0}^{n} min_{\mu_j \in C}||x_i - \mu_j||^2$). Here notice that the the majority of the variation of the clusters can be capture by $k=6$.

![The Inertia Function!](inertia.png)
Inertia as a function of k

Now to (2), which is the harder challenge. If I were to plot my data and look at the clusters, I'd have something that looks like:

![K-Means!](kmeans.png)
Look at all 3 of those beautiful dimensions!

How cool, right? This little algorithm learned pretty clear groups that you can see rather obviously in the data. Impressive! And also useless to your boss and colleagues.

More seriously, while you can see these clusters, you can't actually extract a clear description from it, which makes interpreting it really, really hard when you go past 3 dimensions.

So what can you do to make this slightly more meaningful?

Enter [decision trees](https://en.wikipedia.org/wiki/Decision_tree). Another elegant, classic, and amazing algorithm. Decision Trees basically split up your data using simple `if-else` statements. So, a trick that you can use is to take the predicted clusters and run a Decision Tree (Classification) to predict the segment and use the learneed Tree's logic as your new business logic.

I find this little trick pretty fun and effective since I can more easily describe how a machine learned a segment and I can also inspect it. Let's suppose I ran my tree on this learned K-means, what would the output look like?

![Decision Tree Ouput!](decisiontree.png)
Is this really more interpretable?

There you have it, now you have a segmentation that is closer to optimal and somewhat easier to interpret. It's still not as good as the business definition but you could actually read through this and eventually come up with a heuristic driven approach as well, which is why I like it and why I've used it in the past.

And here's the code to run the K-means and the Decision tree.

```python
import pydotplus
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import export_graphviz
from sklearn.externals.six import StringIO 

optimal_clusters = 6
# 6 clusters 6 colors
xcolors = ['red', 'green', 'blue', 'orange', 'purple', 'gray']
# Chose 6 as the best number of clusters
kmeans_model = (KMeans(n_clusters = optimal_clusters,init='k-means++', n_init = 10 ,max_iter=300, 
 tol=0.0001, random_state= 111 , algorithm='elkan') )
kmeans_model.fit(X1)
cdf['pred_cluster_kmeans'] = kmeans_model.labels_
centroids = kmeans_model.cluster_centers_

display(pd.DataFrame(cdf['pred_cluster_kmeans'].value_counts(normalize=True)))

clf = DecisionTreeClassifier()
# Train Decision Tree Classifer
clf = clf.fit(X1, cdf['pred_cluster_kmeans'])

# Predict the response for test dataset
cdf['pred_class_dtree'] = clf.predict(X1)

display(pd.crosstab(cdf['pred_cluster_kmeans'], cdf['pred_class_dtree']))
dot_data = StringIO()
export_graphviz(
 decision_tree=clf, 
 out_file=dot_data, 
 filled=True, 
 rounded=False,
 impurity=False,
 special_characters=True, 
 feature_names=xcol_labels, 
 class_names=cdf['pred_cluster_kmeans'].unique().astype(str).tolist(),

)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue()) 
graph.write_png("./decisiontree.png")
```
## What can you do with your new segments?

Now that we have our customer segments we can do all sorts of different things.
We can create [A/B tests](https://www.optimizely.com/optimization-glossary/ab-testing/) for website experiences or we can test the impact of [changing our prices](https://medium.com/@judaikawa/price-elasticity-statistical-modeling-in-the-retail-industry-a-quick-overview-fdab5350222) to certain customers.
In general, we can just try a bunch of new stuff.

## How do I know if my segments are accurate?

The metric we used in the example above (i.e., within cluster sum-of-squares / inertia) was a reasonably straightforward way to measure the accuracy of your segments from an analytical perspective, but if you wanted to take a closer look, I'd recommend reviewing individual users in each segment. It sounds a little silly and can, in some cases, lead to the wrong conclusions but I firmly believe that in data science, you just have to really **look** at your data. You learn a lot from it.

## How do I know when my segments need to change?

Lastly, segments can change; your customers are always evolving so it's good to re-evaluate your clusters time and again. The emergence of new segments should feel very obvious, since it may be driven by product or acquisition changes. As a concrete example, if you noticed that important businesss metrics split by your segments are starting to behave a little differently, then you can investigate whether it's driven by a change in the segments; sometimes it is, sometimes it's not.


## Conclusion
This tutorial ended up being a little longer than I anticipated but oh well, I hope you enjoyed it.

I've stored the code to reproduce this example in a [Jupyter Notebook](https://github.com/franciscojavierarceo/Python/blob/master/demos/Customer%20Segmentation%20Example.ipynb) available on my GitHub (note to render the interactive 3D visualization you have to run the notebook). To get it up and running you only need to download the notebook, [download the data](https://www.kaggle.com/vjchoudhary7/customer-segmentation-tutorial-in-python), install [Docker](https://www.docker.com/get-started), and simply run:

```bash
docker run -it -p 8888:8888 -v ~/path/to/your/folder/:/home/jovyan/work --rm --name jupyter jupyter/scipy-notebook:17aba6048f44
```

And you should be good to go. Happy segmenting!

*Have some feedback? Feel free to [let me know](https://twitter.com/franciscojarceo)!*

A Data Scientists Guide to Segmenting your Customers using clustering algorithms and decision trees.

Customer Segmentation using Data Science

*TL;DR: A Data Science Tutorial on building a Credit Risk model.*

I previously [wrote](/post/data-science-and-fintech) about some of the work data scientists do in the fintech space, which briefly discussed [credit risk models](https://en.wikipedia.org/wiki/Credit_risk) but I wanted to write a more technical review and talk about what I think are the most important points.

I spent most of my career at banks as a data scientist and built credit risk, fraud, and marketing models in the consumer and commercial space in the US, Australia, and South Africa. I learned a lot from those experiences, so I thought I'd share a simple example of how one of the core pieces of algorithmic/quantitative underwriting is done.

In this post, I'll try to answer the following questions:

- What is a credit risk model?
- What data does a credit risk model use?
- How do you estimate a credit risk model?
- How do you know if your model is performing well?
- What are some common mistakes to avoid?
- What are useful facts to know about credit risk models?

It's worth caveating up front that this is a very narrow take focused only on the analytical aspect of the work and there is an extraordinary amount of legal, compliance, and business work that I am intentionally omitting.

With that said, let's dive right in.

## What is a Credit Risk Model?

In the consumer/retail space, a credit risk model tries to predict the probability that a consumer won't repay money that they've borrowed.
A simple example of this is an [unsecured personal loan](https://www.investopedia.com/terms/u/unsecuredloan.asp). 

Let's suppose you submitted an application to borrow $5,000 from a lender. That lender would want to know the likelihood of [default](https://www.investopedia.com/terms/d/default2.asp) before deciding on (1) whether to give you the loan and (2) the price they want to charge you for borrowing the money. So that probability is probably quite important...but how do they come up with it?

## What Data does a Credit Risk Model Use?

Lenders typically use data from the major [Credit Bureaus](https://www.investopedia.com/personal-finance/top-three-credit-bureaus/) that is [FCRA](https://www.ftc.gov/enforcement/statutes/fair-credit-reporting-act) compliant, which basically means that the legal and reputational risk of using this data is very low.

Typically, you'd purchase a dataset from one of the bureaus (or use data inside one of their analytical sandboxes) and clean the dataset into something that looks like the following:

<table>
 <tr>
 <th>Default</th>
 <th>Inquiries in Last 6 Months</th>
 <th>Credit Utilization</th>
 <th>Average Age of Credit</th>
 <th>...</th>
 </tr>
 <tr>
 <td>Yes</td>
 <td>2</td>
 <td>0.8</td>
 <td>12</td>
 <td>...</td>
 </tr>
 <tr>
 <td>No</td>
 <td>8</td>
 <td>0.0</td>
 <td>2</td>
 <td>...</td>
 </tr>
 <tr>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 <td>...</td>
 </tr>
</table>

And so on.

In summary, it's just a bunch of data about your borrowing history.
It's a little recursive/cyclical because in order to grow your credit you need to have credit but let's ignore that detail.

One of the most important steps in the model development process is *precisely* defining **default** because this will eventually reflect the performance of your portfolio defining it thoughtfully, accurately, and confidently is extremely consequential.

So how do you define it?

Time. 

If you're planning to launch a term loan, you usually set boundaries on the duration of the loan. For revolving lines of credit (like a credit card), you simply apply a reasonable boundary on time that makes good business sense.

Let's say you want to launch a 12 month loan to an underserved market, then you'd want to get historical data of other lenders to build your model on.
It sounds a little surprising that you can do this but that's basically how the bureaus make their money.

An important thing to keep in mind is that you need to make sure you pull the data at 2 different time periods: (1) when the original application was made so you can use data that is relevant for underwriting (and so you don't have forward-looking data resulting in [data leakage](https://www.kaggle.com/dansbecker/data-leakage)) and (2) 12 months later (or whatever time period is appropriate for you) to check if the consumer defaulted on their loan.

There's a lot more to it and you can expand on things in much more elegant ways to handle different phenomena but for the sake of simplicity, this is essentially how it's done.

So, after painfully cleaning up all that data, what do you do with it?

## Building a Probability of Default Model

Now that you have that glorious dataset you can start to run different [Logistic Regressions](https://en.wikipedia.org/wiki/Logistic_regression) or use other classification based [machine learning](https://www.kdnuggets.com/2016/08/10-algorithms-machine-learning-engineers.html) algorithms to find hidden patterns and relationships (i.e., [non-linear functions](https://blog.minitab.com/blog/adventures-in-statistics-2/what-is-the-difference-between-linear-and-nonlinear-equations-in-regression-analysis#:~:text=If%20the%20equation%20doesn't,a%20linear%20equation%2C%20it's%20nonlinear.&text=Thetas%20represent%20the%20parameters%20and,one%20parameter%20per%20predictor%20variable.) and [interaction terms](https://en.wikipedia.org/wiki/Interaction_(statistics))).

Personally, this is the most intellectually engaging part of the work. The other work involved in credit risk modeling is usually very stressful and filled with less exciting graphs but here you get to pause, look at data, and, for a moment, try to make a crude approximation of the world. I find this part *fascinating*.

If we stick with our simple dataset above for our model, we could use our good old friend Python to run that Logistic Regression.

```python
import numpy as np
import statsmodels.api as sm

# Generating the data 
n = 10000
np.random.seed(0)
x_1 = np.random.poisson(lam=5, size=n)
x_2 = np.random.poisson(lam=2, size=n)
x_3 = np.random.poisson(lam=12, size=n)
e = np.random.normal(size=n, loc=0, scale=1.)

# Setting the coefficient values to give us a ~5% default rate
b_1, b_2, b_3 = -0.005, -0.03, -0.15
ylogpred = x_1 * b_1 + x_2 * b_2 + x_3 * b_3 + e
yprob = 1./ (1.+ np.exp(-ylogpred))
yclass = np.where(yprob >= 0.5, 1, 0)
xs = np.hstack([
 x_1.reshape(n, 1), 
 x_2.reshape(n, 1), 
 x_3.reshape(n, 1)
])
# Adding an intercept to the matrix
xs = sm.add_constant(xs)
model = sm.Logit(yclass, xs)
# All that work just to run .fit(), how terribly uninteresting
res = model.fit()
print(res.summary())

 Current function value: 0.163863
 Iterations 8
 Logit Regression Results 
==============================================================================
Dep. Variable: y No. Observations: 10000
Model: Logit Df Residuals: 9996
Method: MLE Df Model: 3
Date: Fri, 29 Jan 2021 Pseudo R-squ.: 0.1056
Time: 00:08:17 Log-Likelihood: -1638.6
converged: True LL-Null: -1832.2
Covariance Type: nonrobust LLR p-value: 1.419e-83
==============================================================================
 coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
const 0.4535 0.209 2.166 0.030 0.043 0.864
x1 -0.0439 0.023 -1.949 0.051 -0.088 0.000
x2 -0.0390 0.035 -1.109 0.267 -0.108 0.030
x3 -0.3065 0.017 -18.045 0.000 -0.340 -0.273
==============================================================================
```

Wow, look at all of that beautiful, useless statistical output! 

It's not *really* useless but 99% of the people involved will not find it useful. 
So we probably need an alternative way to show and inform these results to non-technical stakeholders (but you as a data scientist can look at this as much as you'd like).

![The Glorious Lift Chart!](liftchart.png)
The Beloved Lift Chart

Cue the [Lift Chart](https://www.casact.org/education/rpm/2016/presentations/PM-LM-4.pdf). This chart may look fancy but it's actually pretty simple, it's just [deciling](https://www.investopedia.com/terms/d/decile.asp) your data (i.e., sorting and bucketing into 10 equal groups) according to the *predicted* default rate from your model. It's worth noting that this is a quantized and reversed version of the [ROC chart](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) and they represent the same information.

There are 3 important pieces of information from this graph: 

1. The AUC tells you the performance
2. The closer the two lines are together the more accurate the probability estimate
3. The steeper the slope at the higher deciles, the better the rank order separation

If you had perfect information, you'd get a graph that looks like this:

![The Perfect Lift Chart!](liftchart_optimal.png)
If your model looks like this, you've done something terribly wrong

And here's the code to generate those graphs.

```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.metrics import roc_auc_score

def liftchart(df: pd.DataFrame, actual: str, predicted: str, buckets: int=10) -> None:
 # Bucketing the predictions (Deciling is the default)
 df['predbucket'] = pd.qcut(x=df[predicted], q=buckets)
 sdf = df[[actual, predicted, 'predbucket']].groupby(
 by=['predbucket']).agg({
 actual: [np.mean, sum, len], 
 predicted: np.mean
 }
 )
 aucperf = roc_auc_score(df[actual], df[predicted])
 sdf.columns = sdf.columns.map(''.join) # I hate pandas multi-indexing
 sdf = sdf.rename({
 actual + 'mean': 'Actual Default Rate', 
 predicted + 'mean': 'Predicted Default Rate'
 }, axis=1)
 sdf[['Actual Default Rate', 'Predicted Default Rate']].plot(
 kind='line', style='.-', grid=True, figsize=(12, 8), 
 color=['red', 'blue']
 )
 plt.ylabel('Default Rate')
 plt.xlabel('Decile Value of Predicted Default')
 plt.title('Actual vs Predicted Default Rate sorted by Predicted Decile \nAUC = %.3f' % aucperf)
 plt.xticks(
 np.arange(sdf.shape[0]), 
 sdf['Predicted Default Rate'].round(3)
 )
 plt.show()

# Using the data and model from before!
pdf = pd.DataFrame(xs, columns=['intercept', 'x1', 'x2', 'x3'])
pdf['preds'] = res.predict(xs)
pdf['actual'] = yclass

# Finally, what we all came here to see
liftchart(pdf, 'actual', 'preds')

# This is what it looks like when we have perfect information
pdf['truth'] = pdf['actual'] + np.random.uniform(low=0, high=0.001, size=pdf.shape[0])
liftchart(pdf, 'actual', 'truth', 10)
```
<center>Judge me not by the elegance of my code but by the fact that it runs.</center>

## Evaluating your Default Model

So this visualization is helpful, but how do you quantify the performance into a single number? Well, how about [Accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision)?

It turns out that Accuracy isn't really a great metric when you have a low default rate (or more generally when you have severe [class imbalance](https://en.wikipedia.org/wiki/Accuracy_paradox)). As an example suppose you have a 5% default rate, that means 95% of your data did *not default*, so if your model predicted that no one defaulted, you'd have 95% accuracy.

Accurate, obvious, and entirely useless. Without proper adjustment, this behavior is actually very likely to occur in your model, so we tend to ignore the accuracy metric and instead we focus on the rank order seperation/predictive power of the model.

To measure that predictive power, there are 4 metrics industry professionals typically look at to summarize their model: [Precision, Recall](https://en.wikipedia.org/wiki/Precision_and_recall), the [Kolomogorov-Smirnov (KS) Test](https://en.wikipedia.org/wiki/Kolmogorov–Smirnov_test), and [Gini/AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve).

One point of humor that has been a surprisingly common topic of discussion in my career is the equivalence of Gini and AUC. Bankers like Gini, for whatever inertia related reason, but it's equivalent to AUC via:

$$Gini = 2 * AUC - 1$$ 

and obviously

$$AUC = \frac{Gini+1}{2}$$.

Gini is bound between [-1, 1] and AUC between [0, 1] but technically if your AUC is less than 0.5 (and < 0 for Gini) that means you're doing worse than random ([and you could do better by literally doing the opposite of what your model says](https://stats.stackexchange.com/questions/266387/can-auc-roc-be-between-0-0-5)) so most people will say AUC is between [0.5, 1] and that Gini is between [0, 1].

A good model is usually around 70% AUC / 40% Gini. The higher the better.

It's always useful to look at your lift chart as it can tell you a lot about how predictive the model is at different deciles. In some cases, your model may not have enough variation in the attributes/features/predictors to differentiate your data meaningfully. The metric won't show you that, only a glance at the lift chart will, so it's always a good to review it.

One important point here is that any single metric is very crude and a data set can be pathologically constructed to break it, so while these metrics and charts are very helpful, there are cases where things can still misbehave even though they seem normal.

Now that you have your exciting new default model, you can use it in your underwriting strategy to differentiate amongst your competitors on pricing, approve/decline, and targeting customers, right?

Not quite.

## Common Mistakes and Pitfalls

Before you get too excited about your model, you want to verify that it's behaving logically, so I thought I'd list some items that you will want to check against to make sure nothing disastrous will happen. 

- Double check for [Data Leakage](https://www.kaggle.com/dansbecker/data-leakage)
- Verify that you don't have any [data errors](https://www.datasciencecentral.com/profiles/blogs/common-errors-in-machine-learning-due-to-poor-statistics-knowledg)
- Make sure you've done your best not to fall for [Simpson's Paradox](https://en.wikipedia.org/wiki/Simpson%27s_paradox)
- Validate your model with an [Out of Time](https://statmodeling.stat.columbia.edu/2016/11/22/30560/) Hold Out Sample
- Confirm your model actually has a [representative sample](https://www.investopedia.com/terms/r/representative-sample.asp#:~:text=A%20representative%20sample%20is%20a,three%20males%20and%20three%20females.) of your future portfolio
- Make sure you're not time traveling 
 - This is a form of Data Leakage (Leaky Predictors) and it's basically making sure you don't use data that's in the future of what you're representing (except the default flags that you're trying to predict/model)
- Understand the correlation your *predicted default* may have to your other products in your portfolio
 - This is not important from a statistical perspective but it is very important for your business
- Always be skeptical if your model performs too well
 - This is very likely data leakage and can be very embarrassing if you over-excite management

I could have written a book ([recommendation here](https://www.amazon.com/Credit-Risk-Modelling-Theoretical-Foundations-Diagnostic-ebook/dp/B07FZGG63V)) with a much longer set of information both from a statistical and business lense but, for the sake of brevity, I wrote the most pressing ones and I invite you to search for more information about other important details.

## Some other Interesting Extensions


### Model Extensions
The model I showed above is quite literally the most basic example and the models that most lenders and vendors use are much, much more sophisticated. Many of them have enhanced their models to handle [panel data](https://en.wikipedia.org/wiki/Panel_data) (i.e., cohorts of people over time) and can construct models that are extremely robust.

The best class of models in this elite category (in my personal opinion) are [Discrete Time Survival Models](https://bookdown.org/content/4253/fitting-basic-discrete-time-hazard-models.html). Actually, this isn't *my* opinon, this wisdom was shared with me from two of the former heads of statistical modeling at Capital One, so don't trust my judgement, trust theirs.

### Quantization and the Weight of Evidence
Another interesting implementation detail is that statisticians/data scientists in the credit scoring space often transform the inputs/attributes/features of their model via [quantization](https://en.wikipedia.org/wiki/Quantization). More specifically through a transformation called the [Weight of Evidence](https://multithreaded.stitchfix.com/blog/2015/08/13/weight-of-evidence/). It's neat, troglodytic, and surprisingly effective.


### Adverse Action Reasons
Credit risk modeling requires the underwriter to grant adverse action codes in the case of rejecting a customer who applies for a credit product (this is a driven by the FCRA). Doing this algorithmitically is very doable and different organizations do it slightly differently but the short version is you take the absolute largest partial predictions (i.e., $argmax_{x}f(x_j) = |x_j * \beta_j|$, $\forall j=1,..., k$).

### Model Complexity and Usefulness
Lastly, when building these statistical models the most important thing to think about is the trade-off between complexity and performance on your target population. Typically in the lending space you will not approve all of your customers, so obsessing over model fit on the proportion of your population that won't be approved is not always useful. It's a small point, but extremely consequential and will hopefully save you some time.

## Conclusion

This tutorial was not nearly as code heavy as I would have liked (only showing a very silly regression) but I felt like the content was still rather dense ([the full code is available here](https://github.com/franciscojavierarceo/Python/blob/master/demos/credit-model-demo.py)).

People spend their careers studying these statistical models and the important consequences of them, so I really want to emphasize how much I have trivialized the work here only for the sake of simplicity. I have not talked about how you can improve the model when it's underperforming or how one can try to measure biases that may occur, but these are important pieces for society.

I will conclude by saying that these models are heavily governed by many layers of regulation. As a statistician and a minority, I think this is a good thing.
That's not to say that harm isn't caused by these models because I think there is but I do think machines are more easily governed, evaluated, and modified...unlike human decision making which can be subjective and prone to unknown biases that are difficult to quantify ([here's some additional reading material](https://faculty.haas.berkeley.edu/morse/research/papers/discrim.pdf)).

I hope you found this post useful and interesting.

*Have some feedback? Feel free to [let me know](https://twitter.com/franciscojarceo)!*

A Data Scientists Guide to Building a Basic Credit Risk Model

How to Build a Credit Risk Model

I've spent the last 10 years working in data science, mostly in finance and technology (fintech) and it's been really exciting to see how data science, engineering, and the internet has reshaped all aspects of finance.

[Others have written before](https://fintechtoday.substack.com/p/part-1-what-is-fintech-30-anyway) about how fintech has evolved over the last decade and one area that I think is interesting is how data science and analytics has helped fuel that growth.

>So, how ***exactly*** has data science helped scale fintech?

I think data science has scaled fintech in five key areas.

## 1. Operations<a name="Operations"></a>

The market tends to have a preference for technology companies because of the operational efficiencies that come from technology's scale. Simply put, data scientists help identify data and processes that can be automated to help achieve better operational efficiency.

A concrete fintech example of this is fraud operations. If you're a bank with a credit card product, manually reviewing even 10% of your credit card transactions would be an impossible, and costly, task (it's been [cited](https://www.marketwatch.com/story/why-bitcoin-wont-displace-visa-or-mastercard-soon-2017-12-15) that Visa and Mastercard process 5,000 transactions per second, which would mean that there are 5000 * 60 * 60 * 24=432,000,000 transactions per day to go through).

So, in this circumstance data scientists build technology and predictive models to reduce the amount of manual review.

## 2. Marketing<a name="Marketing"></a>

In fintech, Customer Acquisition Cost (CAC) is everything. Others have written about [CAC and fintech](https://medium.com/unifimoney/the-no-cac-bank-5e0e577d5473) in greater depth, but suffice it to say it is a challenging and competitive problem.

Data scientists focused on marketing try to reduce CAC through a wide variety of strategies.
Some of them are by tightly monitoring product metrics to see which features yield the best ROI for growth, while other approaches take a broader lense by taking a comprehensive view of your marketing investments and, again, optimize the expected return (e.g., through [Marketing Mix Models](https://blog.hurree.co/blog/marketing-mix-modeling)).

Other marketing focused approaches use [propensity models](https://medium.com/the-official-integrate-ai-blog/heres-what-you-need-to-know-about-propensity-modeling-521ab660cb43) to try to maximize customer engagement or acquisition. More concretely, this can involve building a propensity model to convert a customer from an email subscriber to a fully-converted user (e.g., for a lending product or a mobile application), while other propensities may focus on simply getting a customer to re-engage with your product.

## 3. Risk Management<a name="Risk-Management"></a>

This is where I've spent most of my career and I think it's a really hard problem that most fintechs struggle with in the lending space.
Generally speaking, data scientists will build risk models (e.g., for credit risk or fraud risk) to predict the probability of default or some likelihood of delinquency ([more on the difference between them](https://www.investopedia.com/ask/answers/062315/what-are-differences-between-delinquency-and-default.asp)).

Building good predictive models is hard. Building good *risk* models is **extremely** hard.

This is less because of a technology or data problem and more because of regulatory checks and balances in place. Making sure that you adhere to [FCRA](https://www.ftc.gov/enforcement/statutes/fair-credit-reporting-act), [ECOA](https://uscode.house.gov/view.xhtml?req=granuleid%3AUSC-prelim-title15-chapter41-subchapter4&edition=prelim), and other regulatory oversight is hard on its own, adding statistical analysis into the mix makes it more challenging.

Implementation (i.e., getting an algorithm into production that impacts your customers) of these models is a whole other area of data science and one of the areas I personally find quite fun (maybe I'll write more about this topic later).

## 4. Technology<a name="Technology"></a>

Data scientists often work with engineering/technology teams in order to improve the technology stack. This may involve changing an architecture to reduce the latency of certain [microservices](https://microservices.io/) or enhancing the curent stack for a unique problem (cue machine learning and [Airflow DAGs](https://airflow.apache.org)).
While some of this is behind the scenes, it can be some of the most impactful work done by a data scientist in the fintech space because of the broader impact to the core business.

## 5. Product<a name="Product"></a>

Data scientists working with product teams are often tasked with the measurement of different experiences within the product and finding out ways to enhance it. That can vary from creating dashboards to monitor the right metrics, to building a recommendation system to curate something specific for a user. 

Data scientists can fuel product growth, which is why [Facebook, Google, Amazon, Microsoft](https://www.datasciencedegreeprograms.net/lists/five-of-the-largest-companies-that-employ-data-scientists/) and other tech companies hire so many data scientists.

It's worth noting that I've ignored many of the technology and organizational complexities involved when hiring or building data science teams but only because I wanted to keep this post high-level to introduce some of the applications of data science to fintech. In the future, I'll probably write more technical posts of each one of these to give more concrete examples with code and diagrams (which is what I find fun 😊). 

To summarize, data scientists are useful (particularly in fintech) when there are hard problems and data available to solve them.

*Have some feedback? Feel free to [let me know](https://twitter.com/franciscojarceo)!*

How Data Science Scaled the Fintech Revolution

Data Science and Fintech

One of my favorite authors and historical statisticians [Dr. Stephen Stigler](https://stat.uchicago.edu/people/profile/stephen-m.-stigler/) published a wonderful historical review in 1981 titled [*Gauss and the Invention of Least Squares*](https://projecteuclid.org/download/pdf_1/euclid.aos/1176345451). He argued that the prolific [Carl Freidrich Gauss](https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss) discovered [Ordinary Least Squares](https://en.wikipedia.org/wiki/Least_squares) (OLS) in 1809 and fundamentally shaped the future of science, business, and society as we know it.

So, what is OLS and why is it so important?

OLS is often referred to by many things across several different discipilines, some of them are:

- Linear Regression
- Multivariate Regression
- The Normal Equations
- Maximum Likelihood
- Method of Moments
- Singular Value Decomposition of $\bf{X}\bf{w}-\bf{y}=U(\Sigma'\bf{w}-U'-\bf{y})$

But all of them ultimately reflect the same mathematical expression (in scalar notation):

$$y_i = \beta_0 + \sum_{j=1}^{k} \beta_i x_i + \epsilon_i$$

Which yields the famous estimator (i.e., equation) for $\hat{\beta_j}$ as

$$\hat{\beta_j} = \sum_{i=1}^{n} (x_i - y_i)^2 / \sum_{i=1}^n (x_i - \bar{x})^2$$

Or in matrix notation:

$$\bf \hat{\beta} = \bf (X'X)^{-1} X'Y$$.

I find this simple equation to be so extraordinary.

Why? Because of what can be learned from it: the equation basically says "Look at data about $\bf{x}$ and estimate a linear relationship to $\bf{y}$". 

As a concrete example, imagine you wanted to know the relationship between age and income (a simplification of the well-studied [Mincer Equation](https://en.wikipedia.org/wiki/Mincer_earnings_function)), how would you figure this out? A simple linear regression could estimate that relationship and the $\hat{\beta}$ would represent the partial-correlation (sometimes called the marginal effect or coefficient estimate) and it exactly represents the slope of the line below.

![A scatter plot!](scatterplot.png)
A Scatter Plot of Age and Income

Isn't that just amazing??

This single expression is used to estimate models for movie recommendations, businesses, pharmaceuticals, and even decisions about public health. I am constantly amazed at how one little equation could accomplish so much.

To think Gauss had discovered OLS as a method of calculating the orbits of celestial bodies and that today, over 200 years later, humans would use it to for so much of what we do is astounding.

Over the years statisticians, economists, computer scientists, engineers, and psychometricians have advanced OLS in such profound and unique ways. Some of them have been used to reflect data generated from more non-standard distributions (e.g., a [Weibull distribution](https://en.wikipedia.org/wiki/Weibull_distribution)), or to frame the problem to use prior information in a structured way (e.g., through [Bayesian Inference](https://en.wikipedia.org/wiki/Bayesian_inference)), while others have enhanced these equations to learn high-dimensional non-linear relationships (e.g., via [Artificial Neural Networks](https://en.wikipedia.org/wiki/Artificial_neural_network)). Again, all of these are extended from the extraordinary work of Gauss.

There's so much that can be written about all of the advancements that have been made in all of these fields and a short blog post simply won't do it justice, but I thought I'd at least share some thoughts about it.

Somewhere along the way today I came across something related to important equations and it led me to write this, so I hope you enjoyed it. 

I'm such a fan of the history of statistics and mathematics that this piece, while not as structured as I'd like, was very enjoyable to write.

Happy computing!

-Francisco

A brief note about the most important equation in all of statistics.

Ordinary Least Squares


[I previously wrote about 2020](https://franciscojavierarceo.github.io/post/learning-new-things) and I found that reflecting about the chaos of this past year was rather cathartic for me.

While I feel lucky to have endured the pandemic without contracting COVID or suffering job losses, it has been quite isolating. I know that I'm not alone in feeling that and we're all getting through this in our own ways. Regardless, I found that writing my thoughts out loud was helpful, so I suppose I'll continue writing for now.

Since I'm going to try to write more I figured I'd share some of my goals for the new year; so, here are my foolish high-level goals in some unknown order:

1. Be a better husband
2. Talk less, listen more
3. Cook more intentionally
4. Speak more intentionally
5. Help others more effectively
6. Write at least 2x per month
7. Exercise ~5x per week

I hope I'm able to accomplish these. With the exclusion of the last two, these goals aren't particularly measurable (I'll probably find a way to measure them) but I hope to make a good effort. I hope to look back here and attempt to hold myself accountable. Maybe I'll provide an update in June, who knows.

More specific goals that I have that are much more technical in nature are:
- Connecting Django Rest Framework (DRF) and React while handling data and localization complexities;
- Deploying a React Native application to both app stores;
- Putting in some more hours on Airflow / Cloud Composer;
- Learning how to train my recommendation models on Kubernetes.

These will be interesting and I'm making progress on DRF and React but I still have quite a bit of work to do.

Hopefully this year will be better, I'm optimistic about it.

Goals for 2021

I love code. Plain and simple.

I didn't always though. I started programming at 21 years old during my first masters—I was studying statistics and used code only as a means for running regression models and doing microeconometrics research. I started statistical programming back in 2011 using [STATA](https://www.stata.com/) and [SAS](https://www.sas.com/). They weren't *real* programming languages by a computer scientist's standards but that is technically how I started.

I eventually moved onto to learning [SQL](https://en.wikipedia.org/wiki/SQL) and [R](https://www.r-project.org/about.html) when I started working and I found myself often writing Monte Carlo simulations of harmonic regressions, two stage least-squares, and other machine learning/statistical phenomenon.

And that's how it started. I began learning [Python](https://www.python.org/) more actively at work and it mostly increased from there. During my second masters I started learning [Lua](http://www.lua.org/) in order to use [Torch](http://torch.ch/), which was used by [facebook AI](https://ai.facebook.com/) before [PyTorch](https://pytorch.org/) was developed. Then at Goldman I had to learn proprietary tools like Slang. Now I've spent the last year learning much more about [JavaScript](https://www.javascript.com/) and web development in general, and I just couldn't help but reflect and think about how much I actually *enjoy* it.

It's 2021 now and it's officially beeen a 10 year journey learning how to code (although what can only be argued as chaotically and poorly across different disciplines). Back when I first started I had no idea how anything in a computer worked and I was really, really bad at it. 

It was miserable and I always felt deeply insecure about my code. I feel lucky now that I'm no longer code-shy and I don't struggle as much as I used to. That's not to say I don't have typos or bugs (I do!) but it's more to say that it's much easier these days for me to read through and understand how things are working and how to debug.

But that took me 10 years and it was a literal headache for most of the time. I'm curious to see what I'll feel in 10 years and what else I'll have learned.

All that just to show this beautiful piece of code, which is *a* way to write a function which takes as input *n* and yields its [Fibonacci Number](https://en.wikipedia.org/wiki/Fibonacci_number).

```python
def f(n: int) -> int:
 assert n>0, 'n > 0'
 if n < 3:
 return {1:0, 2:1}[n]
 return f(n-1) + f(n-2)
```

Which simply encodes $F_n = F_{n-1} + F_{n-2} \forall n > 1$, where $F_1 = F_2 = 1$.

This code is both brief *and* elegant. That's what I love about it. In 5 little lines so much of human collaboration and information is represented. To think that 5 little lines could do all of that is astounding to me, I genuinely find it beauitful. I guess that's because I just love code.

I Love Code


I'm a huge fan of [GitHub Actions](https://github.com/features/actions). They're so simple but so effective at doing such a broad range of things. In short, you can tell GitHub do something on a computer everytime you push to your repository (or to a branch on a specific repository like the `main` branch).

For today's post, I'll focus on how I used an action to automate deploying my static site built with Next.js to [GitHub Pages](https://pages.github.com).

I made this blog on a repository hosted on GitHub and added a GitHub Action to compile the JavaScript/React code into static HTML files. This really didn't require all that much effort.


Here's what I had to do:

1. Build a Next.js blog following the [detailed tutorial](https://nextjs.org/learn/basics/create-nextjs-app) and store the code on a GitHub repository
2. Create a workflow file in the repository with the following path: [**./github/workflows/integrate.yml**](https://github.com/franciscojavierarceo/franciscojavierarceo.github.io/blob/main/.github/workflows/integrate.yml)
3. Specify that workflow file to export the static files whenever I push to **main** and *commit* the exported files to a separate branch called **gh-pages** (you can just follow the workflow file I used)
4. Manually add a **.nojekyll** file to the **gh-pages** branch (this is to resolve [this bug](https://github.com/vercel/next.js/issues/2029))
5. Configure my repository Settings so that it sources the GitHub Pages build from the **gh-pages** branch

And that's it, adding new blog posts is as simple as creating a new [markdown](https://www.markdownguide.org/) file and pushing to the **main** branch. The GitHub Action will handle all of the rest!

This is much nicer in behavior than my old site, which was built using Jekyll (a Ruby framework) and it's much less work than building a full application with Django to get high quality page loads. I’d add that I’m a huge fan of Django but I think for a static, fast, and lightweight site, Next.js my new go to!

GitHub Actions to Deploy a Static Site built with Next.js

Deploying a Next.js Site using Github Actions

In the spirit of my last blog post talking about learning new things, I decided to learn <a href="https://nextjs.org/">Next.JS</a>, which is a JavaScript framework built on top of <a href="https://reactjs.org/">React</a> for web development. One of the main reasons I was particularly interested in Next.JS was because of it's opinionated optimizations.

In my last post I mentioned that I spent a lot of time learning about the technical details of Search Engine Optimization (SEO) and trying to optimize my Django application for that. I was ultimately reasonably successful in it but much of the logic to do so was tedious to author and rather unpleasant, which is why I'm excited about Next.JS, since it automates much of this for you right out of the box.

To learn it, I built this blog using their <a href="https://nextjs.org/learn/basics/create-nextjs-app">tutorial</a>, which I highly recommend!

My goal is to move <a href="https://www.unidosfin.com/en">Unidos</a> to Next.JS for the front-end and keep the Django back-end, much of which I've already converted to APIs.

Learning Next.js

2020 was a **difficult year for most people**, this was true for me as well. Fortunately, along the way I was able to learn some new things. More specifically, I spent a lot of my time learning how to develop web applications (particularly in <a href="https://www.djangoproject.com/">Django</a>), which included:

- Building a Content Management System (<a href="https://en.wikipedia.org/wiki/Content_management_system">CMS</a>)
- Image optimization and various forms of cacheing
- User authentication and security best practices (e.g., two factor authentication)
- Developing APIs using Django Rest Framework
- The benefits of JavaScript code compression/minification
- How to adhere to SEO best practices for web content
- Building Webhooks for 3rd party APIs
- Building an application that handles multiple languages in a database, as an API, and that optimizes for SEO

It has been an interesting exercise to be much more hands on with JavaScript this past year, especially since my expertise is mainly in Python. As a data scientist, it's interesting to learn how the web is developed and how data is created from the web.

In statistics, the Data Generating Process (<a href="https://en.wikipedia.org/wiki/Data_generating_process">DGP</a>) is an extremely critical part of approximating the world with mathematical functions. So, I wanted to learn very rigorously how the web is developed so that I could better understand the DGP and ultimately better approximate the things I'm particularly interested in.

As an entrepreneur this is a *terrible* way to allocate one's attention in order to accomplish one's business goals, but the knowledge I've gained over the year has been *extremely* valuable and it compounds significantly. I think the knowledge investment I've made this past year has better helped me prepare for much faster growth going forward.

What I've learned over the past 10 years is that investing in learning has exponential rewards, especially in time. Things that used to take me days or hours to accomplish can now be solved in minutes.

So, while my business didn't thrive this year, I do think I have learned a significant amount of information and that's invaluable.

Here's to hoping 2021 is more balanced on learning and execution. I'll try to write a little more frequently about some of the things I'm learning, so I hope you find it interesting.

Francisco Javier Arceo

Hello there! 👋