The past few months have been a slap in the face for me. Before moving to Berlin, I had everything figured out. I planned the semester down to what time I’d wake up each day, which dance school I’d attend and the books I’d read. Yes, there was an Excel sheet involved. Everything had a budget and a time slot; everything made sense. Until it didn’t.
Two months into my stay, I found myself lost, lacking direction and wholly unsatisfied with day-to-day life. Wedged between statistics classes, endless assignments and the daily struggle to find a sense of meaning and motivation, a hole seemed to open up inside, reminding me that everything I thought I knew about “living my best life” just was not working.
Then, as so often happens when you pay attention, insight began to emerge from the very sources that seemed to be grinding me down. While navigating my way through my computational science courses – a last-minute change of major that proved to be no joke – it dawned on me just how applicable the principles of data science are to our everyday lives. I find three things true of both statistics and personal growth:
You are your richest dataset.
At the end of the day, it’s about the system, not the details.
Personal growth, like data science, is best driven by curiosity.
You are your richest dataset. Understand it and learn from it.
The real world is a giant feedback mechanism. Is it a coincidence that the most talented innovators and erudite futurists are those who see the whole and understand the sweep of human history? “Know thyself” lies at the heart of good data science and real personal growth. Our past experiences, emotional patterns, desires and dislikes are a vast store of data constantly being mined and updated. As we grow up, we develop a complex model that takes this data as input and churns out beliefs, narratives and expectations as output. But this model can break down, and often does. Predictions don’t match observations and so we experience a sense of frustration with the world. We find ourselves stuck in a rut because our model lacks the ability to learn from its own mistakes. We keep feeding it the same data, expecting fresh outcomes. Didn’t someone say the definition of insanity is doing the same thing over and over, expecting a different result?
In data science, you learn that your ability to predict the future rests on how well you can explain the past. A good model always incorporates historical data with acute precision. The catch is that not all data is relevant. It’s a bit like watching the news these days; it’s important to know what’s going on, but nine times out of ten, what’s “going on” is a noisy distraction that has little bearing on the true state of affairs.
This principle can be difficult to accept. People may go through life hurting themselves because they reinforce a narrative developed a long time ago based on some negative experience. Overuse of a model creates a reinforcing feedback loop, making it harder to change over time. I grew up being told (implicitly) that intelligent people get science degrees, that anything else is shallow and a waste of time. I spent a year taking harrowing physics courses just to please the voice inside me that said if I didn’t, I’d be a failure. That mindset continues to limit me in subtle ways and I know I still have a lot of unlearning to do.
If your life were a statistical model, what would it look like? What variables would be weighted more heavily than others, and why? How well would the model fare when faced with new information? Would it be able to adapt if it no longer served you well?
At the end of the day, it’s about the system, not the details.
There’s such a thing as overfitting. Distil every lesson too perfectly, scrutinize every experience with equal attention, and your model quickly breaks down. We are more than the sum of our parts, and hyper-analyzing every aspect of one’s own existence – “carving nature at its joints”, as Plato put it – does not make us any better at living out and actually enjoying that existence. A good data scientist knows that the best model always leaves room for randomness, chance, synchronicity. In life, we call that mystery, beauty, wonder.
The other day, a friend revealed that he lives by the maxim, “Don’t set goals. Create systems.” So many of us are brought up to pursue massive goals, spending our lives chasing milestones, proving that we’re worthy, checking items off a list. The prize for this effort is recognition, fame and success. But what are goals without a sense of sweet, quiet content? This is where models of self-improvement lose their effectiveness, and where my friend’s approach seems so attractive.
What if we focused less on the details and more on the bigger picture? What if our day-to-day activities were based not on what needs to be done, but how we want to do life? Tasks would become less time and achievement bound, and more geared toward how they make us and others feel; the parts of us they challenge and the values they allow us to live out. My friend, for example, quit his 9 to 5 job in order to spend more time with his son. He’s a freelance legal consultant who now spends much of his day being wholly unproductive by societal standards. While others are writing emails and signing deals, he’s probably wiping the chunk of food that just slid down his son’s cheek. When he does work, he works creatively and energetically, brainstorming over coffee, giving legal workshops, helping entrepreneurs navigate the intricacies of their business. His routine is flexible. He’s an ideator and people’s person, and has found ways to weld his professional life into a greater system that constitutes the life he actually wants to live. Not every day is perfect. Sometimes deadlines don’t get met. But every day, he’s tweaking the system that allows him to do what he does best and to become better at it. In his model, being an innovative legal consultant is as important as being a father.
Both data science and life are inherently empirical – we do not master either by perfecting the lines of code or minute-by-minute allocations of our time. We master them by observing, engaging and learning – and then creating a system that reflects what we value most.
Personal growth, like data science, is best driven by curiosity.
You may have heard the term “p-hacking”. It’s what happens when researchers manipulate their methods either consciously or subconsciously to produce a desired p-value – that make-or-break statistical number that tells you the probability of the same results being achieved in real life as in a controlled test. P-values show statistical significance, which is the mathematician’s way of asking “How seriously should I take these results?”
When data scientists p-hack, they’ve stopped trying to prove a hypothesis. Instead, they’re gathering evidence to support what they already believe or want others to believe. We’re victim to this bias in our own lives all the time. When our model breaks down and we continue expecting different results from the same inputs, we tell ourselves it’s okay. We find ways to spin the evidence in a positive light. We’re not the problem – everyone else is. Whatever discrepancies we observe are minor and insignificant. We fool ourselves by “proving” the same old hypothesis, constantly gathering evidence in favour of our current mode of thinking and rejecting anything that suggests there is room for improvement.
It’s scary to think about, but given the right tweaks to experimental design, even the most rigorous data analyst could claim to find patterns that don’t exist, or disprove ones that clearly do. Exploratory data analysis is driven by the goal of finding and understanding these interesting patterns. But unlike p-hacking, it necessitates that during the process of data discovery, we don’t manipulate what we see with an agenda in mind. The approach is pure curiosity.
I’ve fallen prey to setting milestones that don’t challenge my way of thinking. There’s a big difference between going for personal growth and going for personal satisfaction. We don’t like to admit that the one sometimes contradicts the other.
So what of it?
Data science has taught me to spend less time making plans and more time reflecting. The systemic approach has made it easier for me to update my beliefs, seek and respond to feedback, and identify the parts of my own model that need redesigning.
My conclusion: There is no “one size fits all”. Experiment before you try to perfect. Treat yourself like an elegant statistical model that is adept at incorporating nuanced feedback, cancelling the noise and adapting to complex environments. Build elasticity into your foundations by learning from your mistakes and being open to the inputs of others. Shift your focus from “When?” and “How?” to “Why?”, and watch as the pieces start to fit together. Your life is not so much a linear outcome of goals as an emergent property of integrated systems.
One of the most satisfying things about being a data scientist is being able to use newly developed skills to shed light on real life. I’ve spent far too long looking for “new and interesting applications” when the most obvious one of all has been staring right back at me.