The Challenges of NLP

As we saw in Chapter 1, language is the key to the Turing Test, which is meant to validate AI. Language is also something that sets us apart from animals.

But this area of study is exceedingly complex. Here are just some of the challenges with NLP:

• Language can often be ambiguous. We learn to speak in a quick fashion and accentuate our meaning with nonverbal cues, our tone, or reactions to the environment. For example, if a golf ball is heading toward someone, you’ll yell “Fore!” But an NLP system would likely not understand this because it cannot process the context of the situation.
• Language changes frequently as the world changes. According to the Oxford English Dictionary, there were more than 1,100 words, senses, and subentries in 2018 (in all, there are over 829,000)³. Some of the new entries included mansplain and hangry.
• When we talk, we make grammar mistakes. But this is usually not a problem as people have a great ability for inference. But this is a major challenge for NLP as words and phrases may have multiple meanings (this is called polysemy). For example, noted AI researcher Geoffrey Hinton likes to compare “recognize speech” and “wreck a nice beach.”⁴
• Language has accents and dialects.
• The meaning of words can change based on, say, the use of sarcasm or other emotional responses.
• Words can be vague. After all, what does it really mean to be “late”?
• Many words have essentially the same meaning but involve degrees of nuances.
• Conversations can be non-linear and have interruptions.

Despite all this, there have been great strides with NLP, as seen with apps like Siri, Alexa, and Cortana. Much of the progress has also happened within the last decade, driven by the power of deep learning.

Now there can be confusion about human languages and computer languages. Haven’t computers been able to understand languages like BASIC, C, and C++ for years? This is definitely true. It’s also true that computer languages have English words like if, then, let, and print.

But this type of language is very different from human language. Consider that a computer language has a limited set of commands and strict logic. If you use something incorrectly, this will result in a bug in the code—leading to a crash. Yes, computer languages are very literal!