Click or drag to resize
Sdl.LanguagePlatform.Core.Tokenization Namespace

[Missing <summary> documentation for "N:Sdl.LanguagePlatform.Core.Tokenization"]

Classes
  ClassDescription
Public classAutoLocalizationSettings
Contains specialized settings for auto-localization of tokens.
Public classDateTimeToken
A Token which represents a date or time expression.
Public classGenericPlaceableToken
Represents a generic placeable token which is not one of the predefined placeable token classes. Generic placeable tokens can never be auto-localized, but may be auto-substitutable.
Public classMatch
A match object which is returned by FST, FSA, or regex matches
Public classMeasureToken
A Token which represents a measurement, which consists of a numeric value and a unit.
Public classNumberToken
A Token which represents a numeric value.
Public classPhysicalUnit
Encapsulates information about the available physical units and their interrelationships.
Public classPrioritizedToken
A Token with an assigned priority, usually originating from a recognizer's priority. This class is for internal purposes only and should not be used in third-party applications.
Public classSimpleToken
A Token which represents a simple token, such as a word, whitespace, or punctuation.
Public classTagToken
A Token which encapsulates a tag in the input.
Public classToken
Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.
Public classTokenBundle
A special Token which represents a set of alternatives (i.e. an ambiguous analysis) of other tokens which cover the exactly same input span.
Interfaces
  InterfaceDescription
Public interfaceILocalizableToken
Defines the interface for auto-localizable tokens. Localizable tokens have a value, and their surface representation ("text") can be automatically converted into a target culture representation, given the token's value and the target culture.
Enumerations
  EnumerationDescription
Public enumerationBuiltinRecognizers
Enumerates the known types of special token recognizers.
Public enumerationDateTimePatternType
Enumerates the different types of a date or time pattern.
Public enumerationLocalizationParametersSource
Controls which tokens are used to obtain detailed localization parameters, such as the numeric group separator override, or whitespace handling between a number and the unit in measurements.
Public enumerationNumericSeparator
The numeric separators type which can occur in a number token.
Public enumerationSign
The sign of a number
Public enumerationTokenizerFlags
Flags controlling tokenizer behaviour
Public enumerationTokenType
The type of a token, e.g. whether the token represents a word, punctuation, etc.
Public enumerationUnit
Enumerates the units known by the system. Only those units are listed which may require cross-system conversion (not yet implemented).
Public enumerationUnitSeparationMode
Controls how units are separated from the numeric value in measurements.
Public enumerationUnitSystem
The unit system of a unit (metric, BIS, etc.)
Public enumerationUnitType
Enumerates the known types of a unit, i.e. whether it represents a length, area, etc.
Community
Edit

Be the first to Edit the community content of this topic.
Comments