The Future of Ancient Literacy: Classical Language Toolkit and Google Summer of Code

The Classical Language Toolkit (CLTK) is software that brings natural language processing (NLP) to the languages of ancient, classical, and medieval Eurasia. This paper chronicles CLTK’s participation in Google Summer of Code (GSoC), a program run by Google to encourage the growth of open source software. Google pays a stipend to student programmers, who in turn contribute code to an approved project between the months of May and August. In 2016, GSoC accepted CLTK and allotted it funding slots for two students. [1] CLTK, having received over 100 student applications for the program, chose Patrick J. Burns (then a doctoral student in Classics at Fordham University) and Suhaib Khan (an undergraduate at the Netaji Subhas Institute of Technology). Burns proposed to write a multiple-pass, rules-based lemmatizer for the CLTK Core tools; Khan proposed to rework the codebase for the Classical Language Archive, a front-end JavaScript application for use as a reading environment by non-programmers. [2] Kyle P. Johnson acted as the supervisor for the former project and Luke Hollis for the latter. We offer here first a brief introduction to CLTK and then turn to the two summer projects. We conclude with a statement of how we envision the project contributing to the future of ancient literacy, that is, how the entire CLTK ecosystem can work together to offer readers of classical languages a presentation of texts and supporting materials not available in print editions.

Johnson began work on CLTK in 2012 out of a perceived void for open source software offering state-of-the-art natural language processing for ancient languages. [3] Though classicists were pioneers in what today is known as digital humanities, [4] they had not kept pace with the recent maturing of the field of natural language processing. [5] Arising out of the disciplines of computer science and linguistics, NLP has quickly evolved into a significant part of artificial intelligence research. [6]

The software for the CLTK Core is a package written in the Python programming language which is deployed as an easy-to-install package for end users. [7] It currently offers good basic (and some advanced) NLP functionality for Ancient Greek, Latin, Akkadian, Old French, and the Germanic languages. Examples of this functionality includes tools like sentence segmenters, word tokenizers, part-of-speech taggers, and stopword removal. Several more advanced tools are available too, such as pre-trained word2vec models, prosody scanning, and IPA phonetic transcription.

The toolkit’s features for Ancient Greek and Latin are those most developed, however by design all languages supported by CLTK are “first-class citizens,” in that all language-specific data sets and functions are similarly easy to access by end users. [8] One consequence of such parallelism of languages is that CLTK becomes an extensible framework with which language traditions may be studied in comparison to one another with less friction than otherwise possible. For example, were Greek, Hebrew and Latin toolkits available—but in different programming languages, accessible with various conventions, and for different platforms—normalizing input, processing algorithms, and output would be a significant challenge. [9] With a consistent application programming interface, or API, however, users can (in theory, if not entirely yet in practice) analyze language statistics of source texts and their ancient translations, observe diachronic syntactic trends of related languages (such as the evolution of Latin into the Romance languages), or even correlate the representation of ideas between two different literary traditions.

Beyond the software for the Core tools, CLTK also provides digital texts and linguistic data sets. In fact, the project began in part as an effort simply to consolidate the valuable, yet often hard-to-find, NLP data which were spread across the Internet. There was a tremendous amount of valuable information available to be collated, including open access data sets and public-domain text repositories. [10] As data volume and complexity grew, there arose an increasing need for careful organization, access, and dissemination of this data, so as to encourage use. Accordingly, CLTK developed a corpus management system, akin to software management software (the most well-known example being Apple’s App Store). [11] This system allows users to download corpora hosted by the CLTK organization, as well as to identify other corpora which CLTK cannot host, say, for example, due to licensing restrictions. [12] In contrast to most NLP projects, CLTK takes some responsibility for providing users with good texts and assisting them in data management.

For all of CLTK’s efforts to make NLP more accessible to non-technical users, there still remains a significant learning curve to begin using the software, as users must first gain some familiarity with the Unix-style command line and the Python programming language. In the spirit of further breaking down barriers to bringing NLP to the ancient world, the project began a partnership to provide an NLP-backed, online reading environment, an example of which is Khan’s 2016 GSoC contributions to the CLTK Archive.

Hollis mentored Khan’s development of features for a front-end application for CLTK designed to help users perform sophisticated analysis and annotation of literary text data. [13] Taking advantage of the multimodal potential of the web browser, Khan brought together texts already available in the CLTK corpora with user-supplied annotations, images, scholarly citations, and other types of linked data. While originally included in a website called the CLTK Archive, at present offline, these commentary-writing features have since been deployed at several sites, including the Classical Commentaries by the Center for Hellenic Studies (CHS), the Archive of African American Folklore, the Orpheus Digital Collections Platform, and the forthcoming New Alexandria at CHS (which also uses CLTK corpora). [14]

The larger goal of Hollis and Khan’s work was more than just a presentation layer of a data set, but an aesthetically pleasing digital environment to facilitate reading, note-taking, and learning. [15] Two foundational principles directed the scope and direction of Khan’s project: first, the code ought to be thoroughly open, reusable with a variety of text data, and able to be re-implemented by other projects needing annotation. Second, the project aimed to elegantly combine metadata and open-access media. Assuming the user is not overwhelmed with too much information, such metadata can enrich the reading experience.

Hollis chose a microservice architecture (a text-processing back end and display with a JavaScript front end), whereby fundamentally different types of processing are isolated from one another. The CLTK team began the metadata work by constructing a lightweight HTTP API to access the existing functions of the CLTK Core (for example, stemming, text reuse, and scansion). [16] The greatest initial challenge encountered was converting heterogeneous source texts (in XML) into JSON data format for consumption by the front end. We overcame this obstacle by partnering with the originators of ancient language text corpora and taking their advice in reducing the complex data into a simplified JSON of our specification, a process which was repeated with the standardization and inclusion of supplementary materials such as lexica, translations, and images. [17] Programmatically annotated data, that is data processed by the CLTK Core, also needed to be incorporated in the annotation framework. CLTK’s named entity recognition feature was used to create entity labels, which were then used to mine data from Wikipedia, for display inside the Archive website. The CLTK text reuse module was also used to identify intertextuality between authors.

Finally, and at the heart of Khan’s project, was a feature for users to add annotations to the reading environment. Annotations could be made either private or public; in the latter case, the annotations could be integrated into other users’ personal libraries. One obvious use case for this feature was that teachers could share annotations with students, so as to facilitate the comprehension of ancient languages.

Upon completion of this summer’s project, a number of enhancements were considered, mostly concerning adding more metadata and making the update process seamless. Other improvements include allowing users finer-grained control over what metadata components to display in the front end (for example, vocabulary and Wikipedia entities, but not images or JSTOR citations). As open source software, Hollis’s frontend and Khan’s annotation software is available for anyone to deploy and extend.

The 2016 GSoC project for the CLTK Core focused on lemmatization, that is the automated retrieval of dictionary headwords from a given text. For his project, Latin and Greek Backoff Lemmatizer, Burns adopted a strategy commonly applied to part-of-speech identification, namely sequential backoff tagging. For this implementation, he repurposed part-of-speech taggers available in the Natural Language Toolkit (NLTK). [18] With this backoff process, rather than using a single-pass algorithm to return lemmata, several lemmatizers are chained together and used in succession. With each pass, if a lemma is found for a given word, it is assigned and the corresponding word is ignored on subsequent passes. If a lemma is not found, the word is passed down to the next tagger in the chain. The process finishes when all of the lemmatizers—theoretically, an infinite number can be chained together—have been exhausted. Any words that remain untagged at the end of the backoff chain are then assigned a default lemma, such as “UNKNOWN” or None. [19]

In a volume on literacies, it should be mentioned that although the lemmatizers are hardly “literate,” they are grounded in recognizable classical-language reading patterns. One backoff lemmatizer assigns lemmata to indeclinable (and non-ambiguous) words such as adverbs, prepositions, and interjections. The Latin word in will always resolve to the English headword “in.” As a “reader,” the lemmatizer recognizes this and can effectively retrieve the indeclinable word from a lookup table. Another backoff lemmatizer uses regular expressions, a sort of flexible, text-based pattern search, to parse word endings and assign lemmata accordingly. So, for example, a lemma like latinitas (“Latinity”) can be derived using regular expressions from the form latinitati because in the Latin corpus the ending -itati always resolves to -itas. Yet another backoff lemmatizer, the Principal Parts lemmatizer, combines regular-expression-based pattern matching with with a lookup dictionary of valid verb stems. This lemmatizer first matches words with recognizable verb endings, removes the ending, looks up the stem in a principal parts dictionary, and returns a lemma if the stem is valid. So, for example, -eris can be used to derive the lemma for the perfect subjunctive verb audiveris, because audiv- is a valid verb stem (from audio, “to hear”), but the same pass would ignore the dative/ablative plural adjective asperis (from asper, “rough”) because asp- is not a valid verb stem. These lexical and morphological negotiations can again be seen as a “readerly” process and are exactly the kinds word-ending identifications and manipulations asked of every beginning Latin learner. [20] The computer can simply read, stem, and validate forms at a speed and scale unimaginable to a human reader.

Another way in which the lemmatizers imitate the reading process is through context-based decision making. A lemmatizer introduced in the first iteration of the Backoff module is one which “reads” through a pretagged corpus of Latin texts to determine the most likely lemma for a given word based on word frequency. This process is known as training the tagger. [21] The program attempts to distinguish words based on prior “knowledge,” that is information “learned” from the training data, and applies this to new, unseen text. For example, the form bella is found in the training corpus most frequently under the lemma bellum (“war”), so all instances of bella in newly seen text will be tagged as such. While this can lead to false negatives, that is it may mistag a form of the adjective bellus, -a, -um (“beautiful”), the trade-off works in favor of higher accuracy overall for the lemmatizer because bellum is a far more frequent word in the training data than bellus. [22]

A variation of the Train lemmatizer that also replicates classical-language reading practices can be seen in the TrainPOS lemmatizer. This version looks in the training data not only at the most frequent lemma for a given word, but also at the part of speech (POS) of the following word. Accordingly, this lemmatizer can more accurately disambiguate cum (“with”) from cum (“when”), long noted as a challenge for automated lemmatization: the preposition cum is followed with greater frequency by a noun or adjective than its homograph. [23] More experiments need to be done on using extracting lemmata with adjacent POS data, but it could prove useful in resolving many of the ambiguous forms that keep lemmatizers from achieving even greater accuracies.

The GSoC work on a Core tool, that is the lemmatizer, and the Archive developed along separate paths, largely insulated from each other. As CLTK moves forward, however, it is critical to examine how these two projects can be integrated to make each stronger. More importantly, in the context of this article on the role that CLTK plays in the future of digital literacies, it is critical to examine how the Core tools can help readers negotiate texts presented in websites like the Archive and how such websites can give context to output from the Core tools. This output often resembles the kind of paratextual help that was ubiquitous in pedagogical materials from antiquity, such as the compilation of running word-lists or glosses, the addition of macrons, and so on. [24] With this in mind, it is worth reflecting on how the relationship between the Core tools and reading interfaces fits into these practices of ancient literacy.

Raffaella Cribiore has shown, for example, that the physical expression of student texts found in Egyptian papyri makes ample use of reading aids and tools. [25] Moreover, many of these papyrological aids exhibit the analysis of texts into their constituent elements. [26] Many of these find corollaries in the CLTK Core tools. Separation and demarcation of word divisions, whether by spaces, dots, dashes, or other marks, show the long-seated utility in tokenization. Annotation of vowel quantity shows the utility of the macronizer; running word-lists the utility of the lemmatizer. Marginal notes with intertextual references, translations, and various morphological and lexical information hint at valuable tools that CLTK could develop for integration into front-end services.

Yet there is a fundamental difference between the digital presentations of text with their paratextual materials and their ancient models. The technology of papyrus or the codex fixes not only the text, but the paratexts as well. If a learner scribbles in the margin or an editor decides to print glosses at the bottom of the page, these notes become a permanent artefact of the page. A student who wishes to read the text without supplementary materials must resort to physical obstruction, such a piece of paper placed over the edge of the page or a well-placed hand over the undesired support. It can be instructive to read the debate over the arrangement of the page layout of Clyde Pharr’s Aeneid, which commits according to some teachers the ultimate sin of placing too much help alongside the text. [27] The chief technological benefit of the HTML/CSS/Javascript web-publishing trinity and related web publishing schemes is that it destabilizes the fixed reading environment. Dynamic JavaScript libraries (such as jQuery or myriad packages in npm [28] ) give designers almost unlimited control in the addition, subtraction, transformation, rearrangement, and so on, of all content. Perhaps we should think of our reading interfaces not as the modern equivalent of the papyrus scroll or the page—persistent dominant metaphors from the earliest days of web design —but rather as a kind of wax tablet in which texts and paratextual support can be endlessly inscribed, erased, and reinscribed as well as presented in infinite permutations and configurations. So, as opposed to the idea of the digital critical edition, we can perhaps think of the Archive as a digital analytical edition, that is one which aims not to set a definitive version of the text but rather one which lends itself to multiplicity. [29]

Where CLTK can innovate in this respect is by changing the relationship between interface and tools. [30] The Core tools allow users to gather, generate, and present the linguistic data required for NLP research for historical languages. What if the CLTK code that we use for NLP research on the command line could be called directly in the browser? Imagine a web application like jsFiddle, [31] but instead of JavaScript manipulating HTML on the fly, the user could write a script inside the reading environment to transform a given text into a CLTK-generated list of lemmata, or toggle the display of macrons, or bring up a map drawn from the results of a named-entity-recognition script. By not only embedding off-the-shelf tools in the Archive, but also providing a space for user-manipulable scripting, we would open up what could otherwise be a black box for how the Archive presents its underlying data. It places the ability to configure the interface directly into the hands of those who will use the configuration.

The digital analytical edition begins from the idea of treating texts as data and then presents views of this data based on a “set of instructions.” [32] Here we approach a Model-View-Controller (MVC) version of working with classical texts. [33] To take just the projects from GSoC, in an environment where the Core tools and the Archive were tightly integrated, we could pass code using the Backoff Lemmatizer to a text of Virgil’s Aeneid, such that, instead of seeing “Arma virumque cano etc.” as we do in many currently available online editions of this text, we could see the entire text lemmatized before us as Python list: [ ‘arma’, ‘vir’, ‘-que’, ‘cano’, etc.] In theory this view could be changed on the fly to represent any output derived from the underlying CLTK-enhanced Python. As the CLTK web API matures, we expect that such in situ NLP processing will be integrated into its machine-readable layer.

W. A. Johnson refers to literacies as “text-oriented events in particular sociocultural contexts.” [34] His choice of words is serendipitous with respect to how programmers and software architects design user interfaces like the CLTK Archive. “Event” is the programming term for how the user interacts with the browser and so by extension how an Archive user interacts with the reading environment. [35] While there are many online venues available for reading classical literature, in large part they replicate on the screen traditional modes of negotiating texts in their printed form. CLTK, by leveraging innovative work in both the Core and the Archive, can promote an events-driven literacy that literally transforms the way works from antiquity are presented to their audience, offering this generation of classical-text readers levels of engagement and interactivity not realistically available to previous generations.