| Token Class |
Represents a generic, abstract token, which is a sequence of characters in the input. A
token is identified using a tokenizer, which breaks up the sequence of characters in the
input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily
contiguous.
Inheritance HierarchySystemObject
Sdl.LanguagePlatform.CoreSegmentElement
Sdl.LanguagePlatform.Core.TokenizationToken
Sdl.LanguagePlatform.Core.TokenizationDateTimeToken
Sdl.LanguagePlatform.Core.TokenizationNumberToken
Sdl.LanguagePlatform.Core.TokenizationSimpleToken
Sdl.LanguagePlatform.Core.TokenizationTagToken
Sdl.LanguagePlatform.Core.TokenizationTokenBundle
Sdl.LanguagePlatform.CoreSegmentElement
Sdl.LanguagePlatform.Core.TokenizationToken
Sdl.LanguagePlatform.Core.TokenizationDateTimeToken
Sdl.LanguagePlatform.Core.TokenizationNumberToken
Sdl.LanguagePlatform.Core.TokenizationSimpleToken
Sdl.LanguagePlatform.Core.TokenizationTagToken
Sdl.LanguagePlatform.Core.TokenizationTokenBundle
Namespace: Sdl.LanguagePlatform.Core.Tokenization
Assembly: Sdl.LanguagePlatform.Core (in Sdl.LanguagePlatform.Core.dll) Version: 1.6.0.0 (1.7.3164.0)
SyntaxThe Token type exposes the following members.
Constructors| Name | Description | |
|---|---|---|
 | Token |
Initializes a new instance with default values.
|
| Token(String) |
Initializes a new instance with the specified token text.
|
| Token(Token) |
Initializes a new instance with the values from another instance, creating a deep copy.
|
Properties| Name | Description | |
|---|---|---|
 | IsPlaceable |
Gets a boolean value indicating whether this token represents a placeable.
Always returns false for this class. Derived
classes should override appropriately.
|
| IsPunctuation |
true iff the token is a general punctuation, an opening punctuation, or a closing punctuation
|
| IsSubstitutable |
Gets a boolean value indicating whether this token can be substituted (e.g. auto-localized) across segments.
All substitutables are placeables, but not all placeables can be substituted (e.g. acronyms).
|
| IsWhitespace |
Gets a value which indicates whether this token is a whitespace token or not.
|
| IsWord |
true iff the token is a word, an abbreviation, or an acronym
|
| Span |
Gets or sets the span covered by this token in the original segment.
|
| Text |
Gets or sets the token's text.
|
| Type |
Gets, or sets, the TokenType of this token.
|
Methods| Name | Description | |
|---|---|---|
| AcceptSegmentElementVisitor |
Visitor pattern implementation. Derived classes should call back on the
appropriate method on the visitor.
(Inherited from SegmentElement.) |
| Duplicate |
Creates a new instance that is a deep copy of this instance.
(Inherited from SegmentElement.) |
| Equals | Equals(object) (Overrides ObjectEquals(Object).) |
 | Finalize | Allows an object to try to free resources and perform other cleanup operations before it is reclaimed by garbage collection. (Inherited from Object.) |
| GetHashCode | GetHashCode(object) (Overrides ObjectGetHashCode.) |
| GetSimilarity |
Computes the degree of similarity to another instance.
(Inherited from SegmentElement.) |
| GetTokenType | |
| GetType | Gets the Type of the current instance. (Inherited from Object.) |
| GetWeakHashCode |
See GetWeakHashCode (Overrides SegmentElementGetWeakHashCode.) |
| MemberwiseClone | Creates a shallow copy of the current Object. (Inherited from Object.) |
| SetTokenType | |
| ToString | Returns a string that represents the current object. (Inherited from Object.) |
See AlsoCommunityComments