Tokenizer
class Tokenizer extends Object
internal |
Simple lexical analyser.
Constants
VALUE |
|
OFFSET |
|
TYPE |
|
Methods
__construct(array $patterns, $flags = '')
No description
array
tokenize($input)
Tokenizes string.
static array
getCoordinates($text, $offset)
Returns position of token in input string.
Details
at line 34
__construct(array $patterns, $flags = '')
No description
at line 46
array
tokenize($input)
Tokenizes string.
at line 82
static array
getCoordinates($text, $offset)
Returns position of token in input string.