What is tokenization in NLP?

devquora
devquora

Posted On: Feb 22, 2018

 

Natural Language Processing aims to program computers to process large amounts of natural language data. Tokenization in NLP means the method of dividing the text into various tokens. You can think of a token in the form of the word. Just like a word forms into a sentence. It is an important step in NLP to slit the text into minimal units.

    Related Questions

    Please Login or Register to leave a response.

    Related Questions

    NLP interview questions

    What is NLP?

    Natural Language Processing or NLP is an automated way to understand or analyz..

    NLP interview questions

    List some Components of NLP?

    Below are the few major components of NLP.Entity extraction: It involves segmenting a sentence to identify and extract entities, such as a person (real or fictional), organization, geographies, ev...

    NLP interview questions

    List some areas of NLP?

    Natural Language Processing can be used forSemantic Analysis Automatic summarization Text classification Question AnsweringSome real-life example of NLP is IOS Siri, the Google assistant, A...