tutorial status: under revision to update to Stanford NER 4.0.0
time stamp: 2020-10-05
builds on an earlier version by Michael Hanl

• Stanford NER visualisation
• Stanford NER from Python

The Stanford Named Entity Recognizer (NER) developed by Stanford University and licensed under the GNU GPL is an application which locates and classifies named entities in a text into predefined categories. Entities can be understood as elements having a distinct, separate existence and name. These phrases contain, for instance, the names of persons, organizations, locations, times and quantities. The detection of named entities can be useful for computational analysis of literary texts, news texts and research scenarios requiring information on named entities. In literary studies, questions concerning the narrative structure can be answered: Which characters are introduced in which part of the text? Which places are mentioned? Moreover, named entity recognition is a sub-task in the field of information extraction. It is an important component if you are dealing with question answering or co-reference resolution.

This manual offers a brief introduction to the Stanford NER explaining its installation and usage.

Running the application requires a Java Development Kit (JDK) version 8 or later. A free download of the Stanford NER tool is available here.

Stanford NER can be run under Windows as well as under Unix/Linux and MacOSX.

After downloading the zip archive you unzip the file to a directory of your choice. The package contains a simple GUI (graphical user interface) providing a simple visualisation of the named entities. For an initial run with the GUI, start the tool from the command line or just by double clicking the file ner-gui.bat (Windows) or ner-gui.sh (Linux/Unix/MacOSX) for using the GUI. The GUI is suitable for testing and demo purposes, but not for classifying larger files or several files. However, for productive use, the tool should be used from the command prompt and without the GUI.

The tool is well documented. Besides the readme file downloaded with the application which offers a brief introduction you can refer to the list of FAQ's and a presentation about named entity recognition and the Stanford NER as resources.

Much like the Stanford PoS Tagger, the NER tagger is likewise best run on a command line. In the simplest case, you run the named entity recognition process with this command from the terminal which you can likewise save this to a batch (.bat) file on Windows systems or use the ner.bat batch file that ships with the Stanford NER zip file.

Open a terminal / shell / command prompt, navigate to the directory of the Stanford NER instance and enter the following command consisting of a call for the default ner.bat followed by the input file to be NER tagged:

ner.bat sample.txt

If you complete this command further with a output file name, the Stanford NER tagger will write its output to a file which is to be found in its directory instead of writing the output to the screen:

ner.bat sample.txt' > sample-ner-output.txt

Open the terminal / shell / command prompt and navigate to the Stanford NER directory and enter the following command:

java -mx1000m -cp stanford-ner.jar;lib/* edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier classifiers\english.all.3class.distsim.crf.ser.gz -textFile sample.txt

This command runs an English 3-class classifier that identifies three classes of named entities: PERSON, LOCATION, ORGANIZATION.

For detection of the named entities different classifiers are offered. Based on statistical and lexical information they can be selected in the GUI by the button classifiers according to your research question or entered as a parameter in your command or batch file. The following table shows the underlying entities:

Table 1: Features of the different classifiers

Using model 3 (file ner-eng-ie.crf-3-all2008.ser.gz) means searching for the entities: PERSON, LOCATION and ORGANIZATION whereas model 4 (file ner-eng-ie.crf-4-conll.ser.gz), in addition to the three entities mentioned before, contains the entity MISC. This abbreviation (for miscellaneous) refers to entities that do not belong to the previous three groups, for instance, nationalities. Furthermore, these two models are offered containing additional statistical features, the distributional similarity features (DSF) (file ner-eng-ie.crf-3-all2008-distsim.ser.gz and ner-eng-ie.crf-4-conll-distsim.ser.gz). Although their selection means having results with a better performance their usage also requires more memory.

For a first attempt and a quick usage you can use the option Load Default-CRF, which means using model 3 with the entities person, location and organization as shown in the example under 3.1 above.

Besides its compatibility with other tools provided by the Stanford NLP group, an advantage of the Stanford NER tagger is the fast generation of results. Furthermore, models exist for different languages and sets of named entities which means that it comes with numerous possibilities for application. The performance ranges between 60-70% applying the tool on German texts. Using the models with the additional distributional similarity features the performance can be up to 77%. Concerning English texts, the value ranges between 90-93% with or without distributional similarity features (Spanish texts 80%, Dutch 70-80%). Because of the fact that the application is trained on British news texts (2002-2003) the results for other text types such as literary texts or older texts may differ.

Furthermore, the Stanford NER can be trained on new data. This complex task requires a lot of memory and further knowledge. Useful pieces of information can be found in the FAQ.

For a deeper analysis combined with other tools for text analysis, there is the Stanford NER UIMA annotator which uses a modified version of Stanford NER. Developed by Florian Law you can download it here.

All in all, the Stanford NER offers a fast solution for automatic detection and visualization of named entities. Note that the Stanford NER tagging process is also implemented as part of the Stanford CoreNLP Tools.

[1] Compatible means that the output of the Stanford NER can be used as input for further analysis as tagging and parsing which is useful for building a processing pipeline.