The Ambiguity of the Data Science Profession

Ambiguity in a profession makes sketching out your career quite a bit more difficult than it already is.

The data science profession has seen many changes and developments over the last few years. It started out as a field where anything related to data was relevant to your work. Sure, there were already quite a few specializations to be had but they were not recognized as such.

As the field matured, so did the need for specializations and we saw the rise of machine learning engineers, NLP engineers, data engineers, data analysts, BI developers, etc.

On the surface, data scientists became generalists compared to their specialist counterparts even though many data scientists were secretly specialists.

Data scientist is often refered to as a “catch-all title”

Quite unexpectedly, the increasing number of specializations brought about some ambiguity in the data science profession.

Because, if there are so many specializations to be had in the data science profession should a starting data scientist be familiar with all of them?

How much general knowledge about the field can one data scientist have?

I can’t say that I have not experienced this ambiguity myself! When I started in the field, it was quite overwhelming to not have a clear career path as a data scientist.

And I can promise you that this feeling has not gone away and will not anytime soon… but you can learn to embrace it and deal with the ambiguity as you go.

Here, I want to focus on two things. First, the difficulties and dangers of ambiguity in our field. Having a good understanding of the problem is half of the work in solving it. Second, which steps can you take to alleviate some of the difficulties and ambiguity surrounding our field?

1. The Dangers of Ambiguity

I doubt I am the only one that has struggled with ambiguity. Especially in the data science profession, this can be quite troublesome as there are so many perspectives on what a data scientist should know. Trying to follow all those perspectives can and most likely will be a full-time job!

Recognizing some of the dangers is half the battle. Knowing which pitfalls to prevent can be a tremendous help in dealing with this ambiguity.

Impostor syndrome

The impostor syndrome hits hard with all the ambiguity involved in the profession. I have had and often still have much trouble navigating through the wild west of data science.

Which technologies do I want to learn? Which skills are necessary for a data scientist? Do I start with python or should I learn R? I need at least a STEM Ph.D. to start in the field, right? Should I spend my time doing LeetCode?

The impostor syndrome is all too real and difficult to shake.

It is easy to become overwhelmed at the sight of so many incredible people doing wonderful things and the amount of knowledge there is to be learned. Not only can it be quite overwhelming, you might end up trying to learn it all only to end up having broad but shallow knowledge of the field.

Do not try to learn it all. Focus on a select few subfields that interest you.

My advice would be to focus on a select few subfields that interest you in order to maximize your learning potential. Learning should be fun! There is little to gain trying to do it all other than the risk of burn-out.

Gatekeeping

As these titles become more all-encompassing and vague, the need for gatekeeping increases. Because after all, if you are a data scientist working hard to keep up with all of the skills necessary, it would be a shame if somebody without those skills would use the same title, right? — No!

Although some truth can be found in gatekeeping, try to follow a path of your own creation.

Gatekeeping is not always a bad thing but it may prevent actual entry into the field. We really cannot expect junior data scientists, or sometimes even senior data scientists, to be familiar with every single subfield.

The title might actually be a senior position

If we assume that data scientists should be proficient in one or more programming languages, be familiar with complex statistical constructs and math, and be strong communicators, does that sound like a junior position to you?

I have heard colleagues suggest it before but it might be worth considering that data science in many cases might actually be a senior position. Perhaps I am just a slow learner but I did not have those skills fresh out of college.

A job title does not define your skills, you do.

And if that is the case, then we could all agree that a junior data scientist should not be expected to know it all!

Assuming that every data scientist should have a certain set of skills, and those skills are typically quite difficult to attain, how do we then differentiate between junior and senior data scientists?

In other words, do not focus too much on the job title that you have but merely on the skills that you would like to develop. Comparing data scientists is actually quite a difficult thing to do.

2. Dealing with Ambiguity

Dealing with this ambiguity, for both juniors coming into the field as well as seniors not knowing what to pursue next, can be quite difficult.

As with most things in life, having a plan and knowing what to pursue is half of the work. Below, I share some tricks that might help you choose how to deal with this ambiguity and define the path that suits you best.

Specialize

To prevent ambiguity from taking over your career it might be worthwhile to specialize in a field that interests you. It allows you to take some control over what you want to pursue and the way in which you would approach it.

T-profile; know a little about several fields and specialize in something that interests you.

As data scientists, there are many fields in which we can specialize, including but not limited to natural language processing, computer vision, deep learning, geospatial analytics, etc. By choosing one specialization, you can keep a focus on a field of interest without being overwhelmed by every single field you are “supposed” to be familiar with.

There really is no need to know it all finding a way to differentiate yourself is key

Whilst specializing, it should be noted that data scientists often are expected to have a T-profile; general knowledge about a few subfields but also specialized knowledge in a single field. That way, you can focus on your specialism whilst keeping an eye out for other technologies without going all-in.

Embrace it

This might sound a bit counter-intuitive as I have been focusing on the dangers of ambiguity but it actually gives way to innovation. There are those that thrive in ambiguous environments, and the profession of a data science consultant comes to mind.

As a consultant, the focus lies more in having seen quite a number of domains and worked with different skillsets without necessarily being specialized in one or another. As a result, your specialism might be dealing with ambiguity, knowing how to structure a project that has not been clearly defined.

Creating structure and clarity out of ambiguity is an amazing skill to have regardless of the profession you are in!

I touched briefly upon it but the presence of ambiguity gives some room for innovation. For the entrepreneurs among us, embracing this ambiguity and knowing the impact you could have traveling this path is a good opportunity to make your mark! Giving structure to ambiguity is a truly underestimated skill.

Give yourself a break!

It is hard to catch up with the developments in the data science field, doing so would require several full-time jobs. This feeling, as I have experienced quite often, is quite overwhelming! As a result, I have often felt rather demotivated trying to follow along with the latest developments. As you can imagine, not the greatest feeling to be had…

Don't get me wrong, pursuing your career can be very rewarding! There is a balance to everything and I think it is dangerous to underestimate the necessity of stepping back once in a while to recharge.

Giving yourself a break once in a while and taking that step back can make all the difference in the world. Being trapped in ambiguity and the chaos that follows it makes it difficult to structure your thoughts and processes. Whatever you decide to do, whether it is specializing in natural language processing or improving your skills as a consultant, make sure to do it from a calm state of mind.

Remember that being a data scientist is unlikely to be your entire identity

I really cannot overstate this but pursuing some goal, whether it is your career or some side job, is so much easier if you do it from a solid personal foundation. Personally, I believe that happiness and health should come first. That way, you have a lot more mental freedom to work towards your goals but also have a safety net for when things go wrong.

As an example, having a good work-life balance has helped me tremendously make the most out of my time. My “big” ideas come during off-moments, moments when I am not a data scientist and doing something entirely different.

Limiting social media

Social media has taken away many barriers to communication and allows us to share meaningful knowledge but it also has its disadvantages.

Browing Twitter, LinkedIn, and other social media platforms, I still find it rather intimidating seeing all of these experts in the field pump out a new model every few weeks. I felt like I had to follow, understand, and learn it all.

Whenever I saw these interesting developments in a field I was not pursuing, I started questioning the path I was currently taking.

Limiting your social media intake might be difficult but the returns are definitely worth it! Image by the author.

As you can imagine, this only increased the ambiguity I was feeling and did not help structure my goals. The solution? Limiting social media!

When I say, limit social media, that does not mean never using it again but merely being intentional with how you are using it. For example, I take half an hour each morning to answer issues on my open-source packages and spend whatever is left checking out relevant social media.

You decice what you want to pursue, not everyone else.

By limiting how much I spent on social media and being intentional with how I am using it, I take back some of the control and narrative of how my day will go. Instead of being reactive toward what I see online, I started being active with what I wanted to accomplish.

Structure your workflow

When limiting social media, you reduce the reactive nature of your process which creates an opportunity for developing a structured workflow to reach your goals.

Unfortunately, I do not know your perfect workflow as it is different for each individual. However, there are many interesting resources that help you develop such a workflow, such as Make Time, Atomic Habits, The Sprint Book, etc.

Having a feasible plan in place can be a major help in achieving your goals.

Having read quite a few of them myself, for what it’s worth, I believe that many of them boil down to being intentional with your time. Make your goal as explicit as possible and the process of achieving it. “Learning more about natural language processing” is rather vague and only contributes to the ambiguity.

One thing that has helped me greatly is finding out a workflow that works for you. By actively choosing what you want to work on, instead of reactively, you can accomplish so much more!

The same applies to what you want to work on each day. Sitting down before work and writing down what you want to achieve is a great help in structuring your workflow. Do note that writing down big lofty and unachievable goals will only demotivate you. Breaking it down into small and feasible goals each day will help you feel motivated throughout this process.