17.10.4. Global Terrorism Database (GTD)

This directory provides GTD GeoMesa ingest commands, converter configuration files, and an R script to prepare data from the spreadsheet in which it is distributed.

This work is based on a GTD pull done 2015 December 17. The GTD data distribution is dated June 26, 2015. GTD data is updated annually.

17.10.4.1. Getting GTD data

Download the GTD from (http://www.start.umd.edu) via the contact link. You must fill out a web form. Select the zip file with all data and documents. Then unzip this in a convenient directory. This will result in a number of Excel (.xlsx) spreadsheets and PDF documents.

17.10.4.2. R script

The R script extracts the GTD data from the Excel spreadsheets for a selection of the about 150 fields available. The R script expects two arguments. The first is the working directory, which is where the CSV files will be output. The second is the path to the GTD spreadsheet, relative to the working directory.

The R script extracts the data from the main spreadsheet into a data.frame. The script will then export a subset of columns to a file gtd-include.csv. For convenience, the column names in that file are printed to gtd-column-names.csv. Note there are ~160 available attributes.

The script then handles some data cleaning that could be dealt with by the GeoMesa ingest: removing entries with invalid or missing dates and coordinates. The R script writes this data.frame to a CSV file gtd-clean.csv. This step results in about 18% of the data being dropped from the dataset.

17.10.4.3. Ingest Commands

Check that the gtd simple feature type is available on the GeoMesa tools classpath. This is the default case.

$ geomesa-accumulo env | grep gtd

If it is not, merge the contents of reference.conf with $GEOMESA_ACCUMULO_HOME/conf/application.conf, or ensure that reference.conf is in $GEOMESA_ACCUMULO_HOME/conf/sfts/gtd.

Run the ingest. You may optionally point to a different Accumulo instance using -i and -z options. See geomesa-accumulo help ingest for more detail.

$ geomesa-accumulo ingest -u USERNAME -c CATALOGNAME -s gtd -C gtd gtd-clean.csv