Tokenization

there's no need for a tokenization phase, and lookahead assertions can be used for a limited degree of context-sensitivity.

没有必要再进行符号化解析，或者用于有限度上下文敏感的前向断言。

user input may be used to adjust filtering and tokenization of the messages.

用户输入的内容可能会用于校正过滤和用于标记消息。

secondly, the processing of the data required by search and classification,( in other words, document fetching, tokenization, lemmatization, and so on) is the same to a large extent.

其次，搜索和分类所需的数据处理（换句话说，就是获取文档、识别符号、对变体进行规范化等等）在很大程度上是相同的。

tokenization matters a lot for random text collections; in fairness to nltk, its bundled corpora have been packaged for easy and accurate tokenization with wstokenizer().

断词方法对随机文本集合来说至关重要；公平地讲，NLTK捆绑的全集已经通过WSTokenizer（）打包为易用且准确的断词工具。

it supports natural language processing like word stem reduction and tokenization in multiple languages.

它支持自然语言处理，比如多种语言中的词根缩减和标记化。

the result is a sequence of line terminators and input characters, which are the terminal symbols for the third step in the tokenization process.

这样的结果就是一些老的行终端结束符和输入字符，这些终端字符对应着第三步标识符符号化的处理。

in this section, you receive characters as they're available, then tokenize the current result to search for the title token ( actually, but you've stripped the brackets as part of tokenization).

在这一部分，您会即时接收到字符，然后将分析当前结果以搜索到Title（实际上是，但是您已经将括号作为标记的一部分）。

apple is working with major credit card companies like visa, american express and mastercard to integrate a so-called tokenization system into apple pay.

苹果正在与Visa、美国运通公司（American Express）和万事达等主要信用卡公司合作，将所谓的标记化系统整合进Apple Pay。