Natural Language Processing, or NLP for short, is a very popular Data Science methodology that has gained a lot of traction over the past few years as more and more companies have realized that it’s easy to get access to text data and use it to derive valuable insights. Twitter, for example, offers a rich data stream which when processed properly, it can yield a lot of insights on a particular topic or brand, using just NLP as a paid resource. However, NLP wouldn’t have gone far if it weren’t for A.I., since the latter allows it to go beyond the rudimentary statistical models that NLP has in its toolkit. So, what’s the relationship between these two fields and how is it expected to evolve in the years to come?
Just to clarify NLP is so much more than just running a Bayesian classification system, or a regression model on text data that’s been encoded into binary features. NLP also involves topic discovery, text similarity, and even summarization of a document, among other things. All these tasks would be extremely difficult, if not impossible, if it weren’t for A.I. So, NLP is at least partly dependent on A.I., at least for applications that a really worth a data scientist’s time. Of course A.I. is not there is displace Stats, but rather complement this more formal approach. Think of it as the fox that works side-by-side with the hedgehog against a common enemy, rather than two animals fighting each other for dominance.
What about the other side of the relationship? Does A.I. need NLP in any way? Well, the short answer is “it depends” since A.I. is very application-specific. So, for any A.I. system that involves communicating with a human as a main part of its agenda it is important for it to be able to use natural language as much as possible. So, NLP is not only useful but also necessary. That’s something we observe a lot nowadays with chatbots, for example, A.I. systems geared at emulating human communication through a web API, in order to convey useful information or facilitate a certain action. Also, personal assistants like Cortana greatly depend on NLP to connect with their users. However, A.I. systems like the one in many strictly operational scenarios, such as autonomous vehicles, don’t really need NLP since they don’t communicate with the users, at least not as a primary function. This is bound to change in the future though, as it would be easier to market a vehicle that you can talk with, particularly in case of an unexcepted situation, such as an engine problem.
Naturally, the relationship of NLP and A.I. is as much essential as it is conditional. Still, as general A.I. is getting closer and closer, we should expect NLP being an inherent part of A.I. since such a system should be able to excel in pretty much every task that a human can undertake (as well as some tasks beyond our abilities, such as handling big data). So, instead of seeing NLP and A.I. as static entities (hedgehog-like approach), we ought to view them as co-evolving ones (fox-like approach) that at the present moment they have a co-dependent relationship. Still, a A.I. becomes more and more advanced, it is not far-fetched to expect NLP being just another module of an A.I. system, much like the linguistic center is part of the human brain, which is the primary center of intelligence.
What does all this mean for us, data scientists? Clearly, there is no point ignoring either one of these fields, even if our specialty lies in some other part of data science. So, at the very least we ought to be informed about what’s happening in NLP and how A.I. influences data science. We don’t need to write our own algorithms on these fields, but at one point we should be able to tackle an NLP problem, preferably using some A.I. method, or develop an A.I. system that makes use of NLP in the back-end. It may not be easy but as the relationship between NLP and A.I. becomes stronger, it’s bound to become something of a requirement in the near future.
Zacharias Voulgaris, PhD
Passionate data scientist with a foxy flair when it comes to technology, technique, and tests.