17.3. Delimited Text Converter¶
The delimited text converter handles plain delimited text files such as CSV or TSV. To use the delimited text
converter, specify type = "delimited-text"
in your converter definition.
17.3.1. Configuration¶
The format of the delimited files must be defined using the format
element. GeoMesa uses
Apache Commons CSV for parsing. The available
formats are instances of org.apache.commons.csv.CSVFormat
:
- DEFAULT or CSV:
CSVFormat.DEFAULT
- TDF or TSV:
CSVFormat.TDF
- QUOTED:
CSVFormat.DEFAULT.withQuoteMode(QuoteMode.ALL)
- QUOTE_ESCAPE:
CSVFormat.DEFAULT.withEscape('"')
- QUOTED_WITH_QUOTE_ESCAPE:
CSVFormat.DEFAULT.withEscape('"').withQuoteMode(QuoteMode.ALL)
- EXCEL:
CSVFormat.EXCEL
- MYSQL:
CSVFormat.MYSQL
- RFC4180:
CSVFormat.RFC4180
In addition, GeoMesa supports custom quote, escape and delimiter characters, which can be used to modify the
base format. These can be specified through options.quote
, options.escape
and options.delimiter
.
If the input files have header lines, they can be skipped over by specifying a number of lines to skip
using options.skip-lines
, e.g. options.skip-lines = 1
.
17.3.2. Transform Functions¶
The transform
element supports referencing each field in the record by its column number using $
. $0
refers to the whole line, then the first columns is $1
, etc. Each column will initially be a string, so
further transforms may be necessary to create the correct type. See Transformation Function Overview for more details.
17.3.3. Example Usage¶
Suppose you have a SimpleFeatureType
with the following schema:
phrase:String,dtg:Date,*geom:Point:srid=4326
And you have the following comma-separated data:
first,hello,2015-01-01T00:00:00.000Z,45.0,45.0
second,world,2015-01-01T00:00:00.000Z,45.0,45.0
We want to concatenate the first two fields together to form the phrase, parse the third field as a date, and
use the last two fields as coordinates for a Point
geometry. The following configuration defines an appropriate
converter for taking this CSV data and transforming it into our SimpleFeatureType
:
geomesa.converters.example = {
type = "delimited-text",
format = "CSV",
id-field = "md5($0)",
fields = [
{ name = "phrase", transform = "concatenate($1, $2)" },
{ name = "dtg", transform = "dateHourMinuteSecondMillis($3)" },
{ name = "lat", transform = "$4::double" },
{ name = "lon", transform = "$5::double" },
{ name = "geom", transform = "point($lon, $lat)" }
]
user-data = {
// note: keys will be treated as strings and should not be quoted
my.user.key = "$phrase"
}
}
The id
of the SimpleFeature
is formed from an MD5 hash of the
entire record ($0
is the original data). The simple feature attributes
are created from the fields
list with appropriate transforms (note the
use of intermediate fields ‘lat’ and ‘lon’). If desired, user data for the
feature can be set by referencing fields. This can be used for setting
Accumulo visibility constraints, among other things (see Accumulo Visibilities).