17.11. Transformation Function Overview¶
17.11.1. Control Functions¶
try
withDefault
17.11.2. String Functions¶
stripQuotes
length
trim
capitalize
lowercase
regexReplace
concatenate
substring
toString
emptyToNull
printf
17.11.3. Date Functions¶
now
date
dateTime
basicDate
basicDateTime
basicDateTimeNoMillis
dateHourMinuteSecondMillis
millisToDate
secsToDate
17.11.4. Geometry Functions¶
point
linestring
polygon
geometry
projectFrom
17.11.5. ID Functions¶
stringToBytes
md5
uuid
uuidZ3
uuidZ3Centroid
base64
17.11.6. Type Conversions¶
::int
or::integer
::long
::float
::double
::boolean
::r
stringToInt
orstringToInteger
stringToLong
stringToFloat
stringToDouble
stringToBoolean
17.11.7. List and Map Parsing¶
parseList
parseMap
17.11.8. Functions defined using scripting languages¶
You can define functions using scripting languages that support JSR-223.
This is currently tested with JavaScript only as it is natively
supported in all JREs via the Nashorn extension. To define a JavaScript
function for use in the converter framework, create a file with the
.js
extension and the function definition as the contents of the file.
For instance, if you have defined a function such as
function hello(s) {
return "hello: " + s;
}
you can reference that function in a transform expression as
js:hello($2)
17.11.9. Installing Custom Scripts¶
Custom scripting functions are made available to GeoMesa comamnd line tools or distributed (map-reduce) ingest via including them on the classpath or setting a system property.
For local usage, geomesa defines the system property geomesa.convert.scripts.path
to be a colon-separated list of script files and/or directories containing scripts.
This system property can be set when using the command line tools by setting the
CUSTOM_JAVA_OPTS
environmental variable:
CUSTOM_JAVA_OPTS="-Dgeomesa.convert.scripts.path=/path/to/script.js:/path/to/script-dir/"
A more resilient method of including custom scripts is to package them as a JAR or ZIP
file and add it to the GEOMESA_EXTRA_CLASSPATHS
environmental variable. If using
maven you can simply package them in a folder under src/main/resources/geomesa-convert-scripts/
which will create a folder in your jar file named geomesa-convert-scripts
with
the scripts inside. You can manually create a jar with this folder as well. An easier way
is often to package them as a zip archive with a folder similary named geomesa-convert-scripts
inside the archive containing the scripts:
$ unzip -l /tmp/scripts.zip
Archive: /tmp/scripts.zip
Length Date Time Name
--------- ---------- ----- ----
0 2017-03-09 11:33 geomesa-convert-scripts/
42 2017-03-09 11:33 geomesa-convert-scripts/my-script.js
--------- -------
42 2 files
For either zip or jar files add them to the extra classpaths in your environment to make them available for the tools or map-reduce ingest:
GEOMESA_EXTRA_CLASSPATHS="/path/to/my-scripts.jar:/tmp/scripts.zip"
A example of ingest with a scripts on the classpath is below:
GEOMESA_EXTRA_CLASSPATHS="/tmp/scripts.zip:/path/to/my-scripts.jar" bin/geomesa-accumulo ingest -u <user-name>
-p <password> -s <sft-name> -C <converter-name> -c geomesa.catalog hdfs://localhost:9000/data/example.csv
You can also verify the classpath is properly configured with the tools:
GEOMESA_EXTRA_CLASSPATHS="/tmp/scripts.zip:/path/to/my-scripts.jar" bin/geomesa-accumulo classpath
17.11.10. CQL Functions¶
Most of the basic CQL functions are available as transformations. To use
one, invoke it like a regular function, prefixed with the cql
namespace. For example, you can use the CQL buffer function to turn a
point into a polygon:
cql:buffer($1, 2.0)
For more information on the various CQL functions, see the GeoServer filter function reference.
17.11.11. JSON/Avro Transformations¶
See Parsing Json and Parsing Avro sections
17.11.12. Enrichment Functions¶
The converter framework provides a mechanism for setting an attribute based on a lookup from a cache. The cache can be a literal cache in the system or in an external system such as Redis.
cacheLookup