The Paradox in Data Science

Even though data science is considered to be a “sexy” profession as it has a lot of demand, there seems to be a shortage of “good” data scientists, those few individuals that deliver what the field promises, even if those promises are quite unrealistic at times. One would expect that the more data scientists there are out there, the more adept ones will be available. However, this is not what’s observed! For that we can either blame the media (maybe there is a fake news article out there!) or we can delve deeper into the problem and find out why this phenomenon takes place.
 
The main issue of this whole “paradox” of sorts is that we assume some kind of normal-like distribution of data science competence, for some reason. However, this kind of distributions are rarely encountered in cases like that, when you have a more Zipfian phenomenon, where a few cases make up the majority of the area under the distribution. Just like a small number of websites account for the majority of the traffic on the web, a small number of data scientists account for all the glory and all the contribution to the field. Before you start thinking that data science is an elitist society of sorts, let’s consider how other scientific fields are. You have a few talented and/or hard-working individuals who make the headlines, and a lot of other, more average ones, who you’ll never hear about, unless you frequent the conferences they go to. The difference is that most scientists have some academic position so they make ends meet somehow, even if in the majority of cases they are not paid nearly enough for the work they do, or the research they contribute to their field. However, the data scientists who fail to stand out end up being unemployed, getting absorbed in some odd jobs vaguely related to data science, or they end up spending all their time doing Kaggle competitions. That’s not because they are inferior in any way to their academic counterparts, but simply because the standards in the industry are (much) higher, so if you don’t bring enough value to improve the bottom line, you are unlikely to linger in an organization.
 
However, being average in a field that is rapidly evolving is not only acceptable but quite natural. Unless you are super motivated, you’ll have to make a choice about what you will focus on. No-one can be a great programmer, an excellent analyst, and an adept in big data tech, while at the same time have a silver tongue that will charm everyone in a meeting room. These imaginary data scientists, who are often referred to as unicorns, are not around, though it is possible that they will eventually come about, once there is enough infrastructure in place to allow for someone to evolve into such a multi-faceted professional. Until then, the closest you can get are some super talented data scientists who would probably be successful no matter what tech role they would undertake. Since this kind of people are exceptions, it is natural that they wouldn’t come about as often as we would want them to. Also, after a certain level of competence, such a professional would not look for a data science position at all. A person like that who is tech savvy and also adept in the ways of the business world, would probably evolve into an entreperneur and start his own company. After all, data scientists tend to be smart, so they wouldn’t settle for a salary, no matter how high, if they could hit the jackpot of a successful startup.
 
After considering the situation from a couple of different angles, it doesn’t seem so paradoxical after all. Data scientists will be many, but the few exceptional ones that many companies have come to want to recruit will always be in short demand. Now, we could start blaming this or the other factor that brings about this phenomenon, or we could start having more realistic expectations about the data scientists out there. It may not be easy since they are an expensive resource, but in the long run, it’s probably the only sustainable option.