In the next few sections, we\u2019ll look at the different steps in more detail.<\/p>\n\n\n\n
Step #1\u2014Cleaning and Preprocessing<\/h3>\n\n\n\n
Three things need to be done during the cleaning and preprocessing step: tokenization, stemming, and lemmatization.<\/p>\n\n\n\n
Tokenization<\/h4>\n\n\n\n
Before there can be NLP, the text must be parsed and segmented into various parts\u2014a process known as\u00a0tokenization. For example, let\u2019s say we have the following sentence: \u201cJohn ate four cupcakes.\u201d You would then separate and categorize each element. Figure\u00a06-1\u00a0illustrates this tokenization.<\/p>\n\n\n\n All in all, kind of easy? Kind of.<\/p>\n\n\n\n After tokenization, there will be normalization of the text. This will entail converting some of the text so as to make it easier for analysis, such as by changing the case to upper or lower, removing punctuation, and eliminating contractions.<\/p>\n\n\n\n But this can easily lead to some problems. Suppose we have a sentence that has \u201cA.I.\u201d Should we get rid of the periods? And if so, will the computer know what \u201cA I\u201d means?<\/p>\n\n\n\n Probably not.<\/p>\n\n\n\n Interestingly enough, even the case of words can have a major impact on the meaning. Just look at the difference between \u201cfed\u201d and the \u201cFed.\u201d The Fed is often another name for the Federal Reserve. Or, in another case, let\u2019s suppose we have \u201cus\u201d and \u201cUS.\u201d Are we talking about the United States here?<\/p>\n\n\n\n Here are some of the other issues:<\/p>\n\n\n\n As you can see, it can be easy to mis-parse sentences (and in some languages, like Chinese and Japanese, things can get even more difficult with the syntax). But this can have far-ranging consequences. Since tokenization is generally the first step, a couple errors can cascade through the whole NLP process.<\/p>\n\n\n\n Stemming describes the process of reducing a word to its root (or lemma), such as by removing affixes and suffixes. This has actually been effective for search engines, which involve the use of clustering to come up with more relevant results. With stemming, it\u2019s possible to find more matches as the word has a broader meaning and even to handle such things as spelling errors. And when using an AI application, it can help improve the overall understanding.<\/p>\n\n\n\n Figure\u00a06-2\u00a0shows an example of stemming.<\/p>\n\n\n\n There are a variety of algorithms to stem words, many of which are fairly simple. But they have mixed results. According to IBM:<\/p>\n\n\n\n The Porter algorithm, for example, will state that \u2018universal\u2019 has the same stem as \u2018university\u2019 and \u2018universities,\u2019 an observation that may have historical basis but is no longer semantically relevant. The Porter stemmer also does not recognize that \u2018theater\u2019 and \u2018theatre\u2019 should belong to the same stem class. For reasons such as these, Watson Explorer Engine does not use the Porter stemmer as its English stemmer.5<\/sup><\/p>\n<\/blockquote>\n\n\n\n In fact, IBM has created its own proprietary stemmer, and it allows for significant customization.<\/p>\n\n\n\n Lemmatization is similar to stemming. But instead of removing affixes or prefixes, there is a focus on finding similar root words. An example is \u201cbetter,\u201d which we could lemmatize to \u201cgood.\u201d This works so long as the meaning remains mostly the same. In our example, both are roughly similar, but \u201cgood\u201d has a clearer meaning. Lemmatization also may work with providing better searches or language understanding, especially with translations.<\/p>\n\n\n\n Figure\u00a06-3\u00a0shows an example of lemmatization.<\/p>\n\n\n\n To effectively use lemmatization, the NLP system must understand the meanings of the words and the context. In other words, this process usually has better performance than stemming. On the other hand, it also means that the algorithms are more complicated and there are higher levels of computing power required.<\/p>\n\n\n\n Once the text has been put into a format that computers can process, then the NLP system must understand the overall meaning. For the most part, this is the hardest part.<\/p>\n\n\n\n But over the years, researchers have developed a myriad of techniques to help out, such as the following:<\/p>\n\n\n\n For many of these processes, we can use deep learning models. They can be extended to more areas of analysis\u2014to allow for seamless language understanding and generation. This is a process known as distributional semantics.<\/p>\n\n\n\n With a convolutional neural network (CNN), which we learned about in Chapter\u00a04, you can find clusters of words that are translated into a feature map. This has allowed for applications like language translation, speech recognition, sentiment analysis, and Q&A. In fact, the model can even do things like detect sarcasm!<\/p>\n\n\n\n Yet there are some problems with CNNs. For example, the model has difficulties with text that has dependencies across large distances. But there are some ways to handle this, such as with time-delayed neural networks (TDNN) and dynamic convolutional neural networks (DCNN). These methods have shown high performance in handling sequenced data. Although, the model that has shown more success with this is the recurrent neural network (RNN), as it memorizes data.<\/p>\n\n\n\n So far, we have been focused mostly on text analysis. But for there to be sophisticated NLP, we also must build voice recognition systems. We\u2019ll take a look at this in the next section.<\/p>\n\n\n\n In 1952, Bell Labs created the first voice recognition system, called Audrey (for Automatic Digit Recognition). It was able to recognize phonemes, which are the most basic units of sounds in a language. English, for example, has 44.<\/p>\n\n\n\n![\"..\/images\/480660_1_En_6_Chapter\/480660_1_En_6_Fig1_HTML.jpg\"\/](\"https:\/\/learning.oreilly.com\/api\/v2\/epubs\/urn:orm:book:9781484250280\/files\/images\/480660_1_En_6_Chapter\/480660_1_En_6_Fig1_HTML.jpg\")
\n
Stemming<\/h4>\n\n\n\n
![\"..\/images\/480660_1_En_6_Chapter\/480660_1_En_6_Fig2_HTML.jpg\"\/](\"https:\/\/learning.oreilly.com\/api\/v2\/epubs\/urn:orm:book:9781484250280\/files\/images\/480660_1_En_6_Chapter\/480660_1_En_6_Fig2_HTML.jpg\")
\n
Lemmatization<\/h4>\n\n\n\n
![\"..\/images\/480660_1_En_6_Chapter\/480660_1_En_6_Fig3_HTML.jpg\"\/](\"https:\/\/learning.oreilly.com\/api\/v2\/epubs\/urn:orm:book:9781484250280\/files\/images\/480660_1_En_6_Chapter\/480660_1_En_6_Fig3_HTML.jpg\")
Step #2\u2014Understanding and Generating Language<\/h3>\n\n\n\n
\n
Voice Recognition<\/h2>\n\n\n\n