// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

[[_ugr.tools.ruta.language.seeding]]
= Basic annotations and tokens

The UIMA Ruta system uses a JFlex lexer to initially create a seed of basic token annotations.
These tokens build a hierarchy shown in <<_figure.ugr.tools.ruta.language.seeding.basic_token>>.
The "`ALL`" (green) annotation is the root of the hierarchy.
ALL and the red marked annotation types are abstract.
This means that they are actually not created by the lexer.
An overview of these abstract types can be found in <<_table.ugr.tools.ruta.language.seeding.basic_token.abstract>>.
The leafs of the hierarchy (blue) are created by the lexer.
Each leaf is an own type, but also inherits the types of the abstract annotation types further up in the hierarchy.
The leaf types are described in more detail in <<_table.ugr.tools.ruta.language.seeding.basic_token.created>>.
Each text unit within an input document belongs to exactly one of these annotation types. 

[[_figure.ugr.tools.ruta.language.seeding.basic_token]]
.Basic token hierarchy 
image::images/tools/ruta/language/basic_token/basic_token.png[Basic token hierarchy.]


[[_table.ugr.tools.ruta.language.seeding.basic_token.abstract]]
.Abstract annotations
[cols="1,1,1", frame="all", options="header"]
|===
| Annotation
| Parent
| Description

|ALL
|-
|parent type of all tokens

|ANY
|ALL
|all tokens except for markup

|W
|ANY
|all kinds of words

|PM
|ANY
|all kinds of punctuation marks

|WS
|ANY
|all kinds of white spaces

|SENTENCEEND
|PM
|all kinds of punctuation marks that indicate the end of a sentence 
|===


[[_table.ugr.tools.ruta.language.seeding.basic_token.created]]
.Annotations created by lexer
[cols="1,1,1,1", frame="all", options="header"]
|===
| Annotation
| Parent
| Description
| Example

|MARKUP
|ALL
|HTML and XML elements
| `<p class="Headline">`


| NBSP
| SPACE
| non breaking space
| ` `


|AMP
|ANY
|ampersand expression
| `&`

|BREAK
|WS
|line break
| `\n`


|SPACE
|WS
|spaces
| ` `


|COLON
|PM
|colon
| `:`


|COMMA
|PM
|comma
| `,`


|PERIOD
|SENTENCEEND
|period
| `.`


|EXCLAMATION
|SENTENCEEND
|exclamation mark
| `!`


|SEMICOLON
|PM
|semicolon
| `;`


|QUESTION
|SENTENCEEND
|question mark
| `?`


|SW
|W
|lower case work
| `annotation`


|CW
|W
|work starting with one capitalized letter
| `Annotation`


|CAP
|W
|word only containing capitalized letters
| `ANNOTATION`


|NUM
|ANY
|sequence of digits
| `0123`


|SPECIAL
|ANY
|all other tokens and symbols
| `/`
|===