Click or drag to resize
TokenType Enumeration
The type of a token, e.g. whether the token represents a word, punctuation, etc.

Namespace: Sdl.LanguagePlatform.Core.Tokenization
Assembly: Sdl.LanguagePlatform.Core (in Sdl.LanguagePlatform.Core.dll) Version: 1.6.0.0 (1.7.3164.0)
Syntax
Copy
public enum TokenType
Members
  Member nameValueDescription
Unknown0 The token type is unknown or has not yet been determined.
Word1 The token represents a word.
Abbreviation2 The token represents an abbreviation.
CharSequence3 The token represents a character sequence. This is used in contexts (such as tokenization of some far-east languages) where word boundaries cannot easily be determined.
GeneralPunctuation4 The token represents general punctuation.
OpeningPunctuation5 The token represents opening punctuation, such as a left parenthesis, which is typically followed by the closing counterpart.
ClosingPunctuation6 The token represents closing punctuation, such as a right parenthesis, which is typically preceded by the opening counterpart.
Date7 The token represents a complex date expression.
Time8 The token represents a complex time expression.
Variable9 The token represents a substitutable variable.
Number10 The token represents a number.
Measurement11 The token represents a measurement, which is a combination of a number followed by a unit.
Whitespace12 The token represents a non-empty sequence of whitespace.
Acronym13 The token represents an acronym.
Uri14 The token represents a URI/URL.
OtherTextPlaceable15 The token represents an unspecified text placeable (not a tag placeables), which can be placed, but not auto-substituted.
UserDefined16 The token represents a user-defined entity.
Tag17 The token represents a tag.
AlphaNumeric18
See Also
Community
Edit

Be the first to Edit the community content of this topic.
Comments