// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

[[_section.ugr.tools.ruta.workbench.testing]]
= Testing

The UIMA Ruta Workbench comes bundled with its own testing environment that allows you to test and evaluate UIMA Ruta scripts.
It provides full back-end testing capabilities and allows you to examine test results in detail. 

To test the quality of a written UIMA Ruta script, the testing procedure compares a previously annotated gold standard file with the resulting xmiCAS file created by the selected UIMA Ruta script.
As a product of the testing operation a new xmiCAS file will be created, containing detailed information about the test results.
The evaluators compare the offsets of annotations and, depending on the selected evaluator, add true positive, false positive or false negative annotations for each tested annotation to the resulting xmiCAS file.
Afterwards precision, recall and f1-score are calculated for each test file and each type in the test file.
The f1-score is also calculated for the whole test set.
The testing environment consists of four views: Annotation Test, True Positive, False Positive and False Negative.
The Annotation Test view is by default associated with the UIMA Ruta perspective. 

[NOTE]
====
There are two options for choosing the types that should be evaluated, which is specified by the preference "`Use all types`".
If this preference is activated (by default), then the user has to selected the types using the toolbar in the view.
There are button for selecting the included and excluded types.
If this preference is deactivated,  then only the types present in the current test document are evaluated.
This can result in missing false positive, if  the an annotation of a specific type was created by the rules and no annotation of this type is present in the test document. 
====

<<_figure.ugr.tools.ruta.workbench.testing.script_explorer>> shows the script explorer.
Every UIMA Ruta project contains a folder called "`test`".
This folder is the default location for the test-files.
In the folder each script file has its own subfolder with a relative path equal to the scripts package path in the "`script`" folder.
This folder contains the test files.
In every scripts test folder, you will also find a result folder where the results of the tests are saved.
If you like to use test files from another location in the file system, the results will be saved in the "`temp`" subfolder of the project's test folder.
All files in the temp folder will be deleted once Eclipse is closed. 


[[_figure.ugr.tools.ruta.workbench.testing.script_explorer]]
.Test folder structure. 
image::images/tools/ruta/workbench/testing/script_explorer.png[The test folder structure.]


[[_section.ugr.tools.ruta.workbench.testing.usage]]
== Usage

This section describes the general proceeding when using the testing environment. 

Currently, the testing environment has no own perspective associated to it.
It is recommended to start within the UIMA Ruta perspective.
There, the Annotation Test view is open by default.
The True Positive, False Positive and False Negative views have to be opened manually: "`Window -> Show View -> True Positive/False Positive/False Negative `". 

To explain the usage of the UIMA Ruta testing environment, the UIMA Ruta example project is used again.
Open this project.
Firstly, one has to select a script for testing: UIMA Ruta will always test the script, that is currently open and active in the script editor.
So, open the "`Main.ruta`" script file of the UIMA Ruta example project.
The next <<_figure.ugr.tools.ruta.workbench.testing.annotation_test_initial_view,figure>>.
shows the Annotation Test view after doing this. 


[[_figure.ugr.tools.ruta.workbench.testing.annotation_test_initial_view]]
.The Annotation Test view. Button from left to right:Start Test; Select excluded type; Select included type; Select evaluator/preferences; Export to CSV; Extend Classpath 
image::images/tools/ruta/workbench/testing/annotation_test_initial_view_2_2_0.png[The Annotation Test view.]

All control elements that are needed for the interaction with the testing environment are located here.
At the top right, there is the buttons bar.
At the top left of the view the name of the script that is going to be tested is shown.
It is always equal to the script active in the editor.
Below this, the test list is located.
This list contains the different files for testing.
Right next to the name of the script file you can select the desired view.
Right to this you get statistics over all ran tests: the number of all true positives (TP), false positives (FP) and false negatives (FN). In the field below, you will find a table with statistic information for a single selected test file.
To change this view, select a file in the test list field.
The table shows a total TP, FP and FN information, as well as precision, recall and f1-score for every type as well as for the whole file. 

There is also an experimental feature to extend the classpath during testing, which allows to  evaluate scripts that call analysis engines in the same workspace.
Therefore, you have to toggle the button in the toolbar of the view. 

Next, you have to add test files to your project.
A test file is a previously annotated xmiCAS file that can be used as a golden standard for the test.
You can use any xmiCAS file.
The UIMA Ruta example project already contains such test files.
These files are listed in the Annotation Test view.
Try do delete these files by selecting them and clicking on ``Del``.
Add these files again by simply dragging them from the Script Explorer into the test file list.
A different way to add test-files is to use the "`Load all test files from selected folder`" button (green plus). It can be used to add all xmiCAS files from a selected folder. 

Sometimes it is necessary to create some annotations manually: To create annotations manually, use the "`Cas Editor`" perspective delivered with the UIMA workbench. 

The testing environment supports different evaluators that allow a sophisticated analysis of the behavior of a UIMA Ruta script.
The evaluator can be chosen in the testing environment's preference page.
The preference page can be opened either through the menu or by clicking on the "`Select evaluator`" button (blue gear wheels) in the testing view's toolbar.
Clicking the button will open a filtered version of the UIMA Ruta preference page.
The default evaluator is the "Exact CAS Evaluator", which compares the offsets of the annotations between the test file and the file annotated by the tested script.
To get an overview of all available evaluators, see <<_section.ugr.tools.ruta.workbench.testing.evaluators>>

This preference page (see <<_figure.ugr.tools.ruta.workbench.testing.preference>>)  offers a few options that will modify the plug-ins general behavior.
For example, the preloading of previously collected result data can be turned off.
An important option in the preference page is the evaluator you can select.
On default the "exact evaluator" is selected, which compares the offsets of the annotations, that are contained in the file produced by the selected script with the annotations in the test file.
Other evaluators will compare annotations in a different way. 


[[_figure.ugr.tools.ruta.workbench.testing.preference]]
.The testing preference page view 
image::images/tools/ruta/workbench/testing/preference_2_2_0.png[The testing preference page view.]

During a test-run it might be convenient to disable testing for specific types like punctuation or tags.
The "`Select excluded types`" button (white exclamation in a red disk) will open a dialog (see <<_figure.ugr.tools.ruta.workbench.testing.excluded_types>>)  where all types can be selected that should not be considered in the test. 


[[_figure.ugr.tools.ruta.workbench.testing.excluded_types]]
.Excluded types window 
image::images/tools/ruta/workbench/testing/excluded_types.png[Excluded types window.]

A test-run can be started by clicking on the start button.
Do this for the UIMA Ruta example project. <<_figure.ugr.tools.ruta.workbench.testing.annotation_test_test_run>> shows the results. 


[[_figure.ugr.tools.ruta.workbench.testing.annotation_test_test_run]]
.The Annotation Test view. 
image::images/tools/ruta/workbench/testing/annotation_test_test_run_2_2_0.png[The Annotation Test view.]

The testing main view displays some information on how well the script did after every test run.
It will display an overall number of true positive, false positive and false negatives annotations of all result files as well as an overall f1-score.
Furthermore, a table will be displayed that contains the overall statistics of the selected test file as well as statistics for every single type in the test file.
The information displayed are true positives, false positives, false negatives, precision, recall and f1-measure. 

The testing environment also supports the export of the overall data in form of a comma-separated table.
Clicking the "`export data`" button will open a dialog window that contains this table.
The text in this table can be copied and easily imported into other applications. 

When running a test, the evaluator will create a new result xmiCAS file and will add new true positive, false positive and false negative annotations.
By clicking on a file in the test-file list, you can open the corresponding result xmiCAS file in the CAS Editor.
While displaying the result xmiCAS file in the CAS Editor, the True Positive, False Positive and False Negative views allow easy navigation through the new tp, fp and fn annotations.
The corresponding annotations are displayed in a hierarchic tree structure.
This allows an easy tracing of the results within the testing document.
Clicking on one of the annotations in those views will highlight the annotation in the CAS Editor.
Opening "`test1.result.xmi`" in the UIMA Ruta example project changes the True Positive view as shown in <<_figure.ugr.tools.ruta.workbench.testing.true_positive>>.
Notice that the type system, which will be used by the CAS Editor to open the evaluated file,  can only be resolved for the tested script, if the test files are located in the associated folder structure that is the folder with the name of the script.
If the files are located  in the temp folder, for example by adding the files to the list of test cases by drag and drop, other strategies to find the correct type system will be applied.
For UIMA Ruta projects,  for example, this will be the type system of the last launched script in this project. 


[[_figure.ugr.tools.ruta.workbench.testing.true_positive]]
.The True Positive view. 
image::images/tools/ruta/workbench/testing/true_positive.png[The True Positive view.]


[[_section.ugr.tools.ruta.workbench.testing.evaluators]]
== Evaluators

When testing a CAS file, the system compared the offsets of the annotations of a previously annotated gold standard file with the offsets of the annotations of the result file the script produced.
Responsible for comparing annotations in the two CAS files are evaluators.
These evaluators have different methods and strategies implemented for comparing the annotations.
Also, an extension point is provided that allows easy implementation of new evaluators. 

Exact Match Evaluator: The Exact Match Evaluator compares the offsets of the annotations in the result and the golden standard file.
Any difference will be marked with either a false positive or false negative annotations. 

Partial Match Evaluator: The Partial Match Evaluator compares the offsets of the annotations in the result and golden standard file.
It will allow differences in the beginning or the end of an annotation.
For example, "corresponding" and "corresponding " will not be annotated as an error. 

Core Match Evaluator: The Core Match Evaluator accepts annotations that share a core expression.
In this context, a core expression is at least four digits long and starts with a capitalized letter.
For example, the two annotations "L404-123-421" and "L404-321-412" would be considered a true positive match, because "L404" is considered a core expression that is contained in both annotations. 

Word Accuracy Evaluator: Compares the labels of all words/numbers in an annotation, whereas the label equals the type of the annotation.
This has the consequence, for example, that each word or number that is not part of the annotation is counted as a single false negative.
For example in the sentence: "Christmas is on the 24.12 every year." The script labels "Christmas is on the 12" as a single sentence, while the test file labels the sentence correctly with a single sentence annotation.
While, for example, the Exact CAS Evaluator is only assigning a single False Negative annotation, Word Accuracy Evaluator will mark every word or number as a single false negative. 

Template Only Evaluator: This Evaluator compares the offsets of the annotations and the features, that have been created by the script.
For example, the text "Alan Mathison Turing" is marked with the author annotation and "author" contains 2 features: "FirstName" and "LastName". If the script now creates an author annotation with only one feature, the annotation will be marked as a false positive. 

Template on Word Level Evaluator: The Template On Word Evaluator compares the offsets of the annotations.
In addition, it also compares the features and feature structures and the values stored in the features.
For example, the annotation "author" might have features like "FirstName" and "LastName". The authors name is "Alan Mathison Turing" and the script correctly assigns the author annotation.
The feature assigned by the script are "Firstname : Alan", "LastName : Mathison", while the correct feature values are "FirstName Alan" and "LastName Turing". In this case, the Template Only Evaluator will mark an annotation as a false positive, since the feature values differ. 