BitCurator NLP

From BitCurator
Jump to: navigation, search

The BitCurator NLP project is developing software for collecting institutions to extract, analyze, and produce reports on features of interest in text extracted from born-digital materials contained in collections. We are using open source natural language processing libraries to identify items likely to be relevant to preservation, information organization, and access activities. These may include entities (e.g. persons, places, and organizations), potential relationships among entities (e.g. those entities that appear together within documents or set of documents), and topic models to provide insight into how concepts are naturally clustered within the documents. We are developing software that will allow users to create customized reports from text discovered in disk images, providing both command-line executables and a public Python API to extend the capabilities of external tools.

Rationale and Technical Foundation

Born digital collections often include a wide range of complex file formats (for example, Office documents, PDF files, email, and audiovisual materials) from which text may be extracted directly or by automated transcription. Text extraction from arbitrary collections of files is itself a non-trivial task; solving this problem is not a focus of the project. BitCurator NLP projects use existing software platforms including:

textract textacy spaCy scikit-learn TextBlob

to perform text extraction from heterogeneous collections of file and execute NLP tasks such as entity and entity relationship identification, topic modeling (and topic model visualization), and document summarization. The software will allow users to to perform text extraction from heterogeneous collections of file and execute NLP tasks such as entity and entity relationship identification, topic modeling (and topic model visualization), and document summarization. The software will allow users to pick candidate files from a collection and create human- and machine-readable reports (PDF or text, and JSON, respectively) that meaningfully characterize the contents of those files based on raw (unannotated) text that they contain.

Generating topic models from disk image contents


The BitCurator GitHub IO page links to repositories in development. Some of these projects are listed below.

bitcurator-nlp-gentm (GitHub)

Current and past releases

The bitcurator-nlp-gentm project simplifies the process of automatically analyzing disk image collections by text content, allowing archivists and other collecting institution professionals to identify potential topics of interest without requiring manual inspection of individual files or directories.

The software uses dfVFS to automate parsing and extracting file system contents from a wide range of disk image formats and file types. This toolset provides access to disk images stored as raw, EWF (EWF-E01, EWF-Ex01, EWF-S01), QCOW, VHD, and VMDK, and file systems including FAT, HFS, HFS+, NTFS, and ext2/3/4.

Once available file systems are exposed using dfVFS, text is extracted from candidate files using textract. The textract wrapper supports many different formats; the relevant formats can be targeted using a simple configuration file provided with bitcurator-nlp-gentm (for example, to reduct processing time by disregarding those files for which OCR would be required).

Extracted text is cleaned and processed using pyLDAvis - a Python fork of the LDAVis tool described in (Sievert and Shirley, 2014) - to produce visualizations of topic models created from English language terms.


Get Help or Contribute

BitCurator User Group Get support and engage with the community.
Screencasts and Video Tutorials Screencasts and tutorials on our YouTube channel.

Follow the links to the right to request help from the user group or view tutorials on YouTube.

The BitCurator wiki currently includes 26 pages. View statistics on users, pages, and edits. To contribute to this wiki, click on Request account. An email with temporary credentials and instructions will arrive once an admin has approved the request. Original content on this wiki is Creative Commons-licensed.


Sources in the BitCurator NLP GitHub repositories are LGPL v3 licensed except where otherwise noted. This wiki, documentation, and other materials generated by the BitCurator team are licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). All other software included in the BitCurator environment is distributed in accordance with original licenses.

Development, Funding, and Partners

The BitCurator development team is hosted by the School of Information and Library Science at the University of North Carolina, Chapel Hill. Grants from the Andrew W. Mellon Foundation supported the BitCurator project (a partnership between UNC SILS and the Maryland Institute for Technology in the Humanities) through September 2014, and the BitCurator Access project through September 2016. A grant from the Andrew W. Mellon Foundation currently supports the BitCurator NLP project (2016-2018).

BitCurator Consortium members provide continuing support for past projects. If you or your organization use BitCurator tools, please consider joining the Consortium!