17.12. Transformation Function Usage¶
17.12.1. Control Functions¶
17.12.1.1. try¶
Description: Execute another function - if it fails, instead use a default value
Usage: try($1, $2)
Example: try("1"::int, 0) = 1
Example: try("abcd"::int, 0) = 0
17.12.1.2. withDefault¶
Description: Replace a value with a default, if the value is null
Usage: withDefault($1, $2)
Example: withDefault('foo', 'bar') = foo
Example: withDefault(null, 'bar') = bar
17.12.2. String Functions¶
17.12.2.1. stripQuotes¶
Description: Remove double quotes from a string.
Usage: stripQuotes($1)
Example: stripQuotes('fo"o') = foo
17.12.2.2. length¶
Description: Returns the length of a string.
Usage: length($1)
Example: length('foo') = 3
17.12.2.3. trim¶
Description: Trim whitespace from around a string.
Usage: trim($1)
Example: trim(' foo ') = foo
17.12.2.4. capitalize¶
Description: Capitalize a string.
Usage: capitalize($1)
Example: capitalize('foo') = Foo
17.12.2.5. lowercase¶
Description: Lowercase a string.
Usage: lowercase($1)
Example: lowercase('FOO') = foo
17.12.2.6. uppercase¶
Description: Uppercase a string.
Usage: uppercase($1)
Example: uppercase('foo') = FOO
17.12.2.7. regexReplace¶
Description: Replace a given pattern with a target pattern in a string.
Usage: regexReplace($regex, $replacement, $1)
Example: regexReplace('foo'::r, 'bar', 'foobar') = barbar
17.12.2.8. concatenate¶
Description: Concatenate two strings.
Usage: concatenate($0, $1)
Example: concatenate('foo', 'bar') = foobar
17.12.2.9. substring¶
Description: Return the substring of a string.
Usage: substring($1, $startIndex, $endIndex)
Example: substring('foobarbaz', 2, 5) = oba
17.12.2.10. toString¶
Description: Convert another data type to a string.
Usage: toString($0)
Example: concatenate(toString(5), toString(6)) = '56'
17.12.2.11. emptyToNull¶
Description: Replace an empty string with null. Useful for setting optional attributes from delimited
text files, where inputs will never be null.
Usage: emptyToNull($0)
Example: emptyToNull('') = null
17.12.2.12. printf¶
Description: Format custom strings. As an implementation detail, this function delegates to Java’s String formatting classes.
Usage: printf('patterns', $arg1, $arg2, ...)'
- Examples:
printf('%s-%s-%sT00:00:00.000Z', '2015', '01', '01') = '2015-01-01T00:00:00.000Z' printf('%2f', divide(-1, 2, 3)) = '-0.17'
17.12.3. Date Functions¶
17.12.3.2. date¶
Description: Custom date parser. The date format is defined by the Java 8 DateTimeFormatter class.
Usage: date($format, $1)
Example:
date('yyyy-MM-dd\\'T\\'HH:mm:ss.SSSSSS', '2015-01-01T00:00:00.000000')
17.12.3.3. dateTime¶
Description: A strict ISO 8601 Date parser for format
yyyy-MM-dd'T'HH:mm:ss.SSSZZ.
Usage: dateTime($1)
Example: dateTime('2015-01-01T00:00:00.000Z')
17.12.3.4. basicDate¶
Description: A basic date format for yyyyMMdd.
Usage: basicDate($1)
Example: basicDate('20150101')
17.12.3.5. basicDateTime¶
Description: A basic format that combines a basic date and time for
format yyyyMMdd'T'HHmmss.SSSZ.
Usage: basicDateTime($1)
Example: basicDateTime('20150101T000000.000Z')
17.12.3.6. basicDateTimeNoMillis¶
Description: A basic format that combines a basic date and time with no
millis for format yyyyMMdd'T'HHmmssZ.
Usage: basicDateTimeNoMillis($1)
Example: basicDateTimeNoMillis('20150101T000000Z')
17.12.3.7. dateHourMinuteSecondMillis¶
Description: Formatter for full date, and time keeping the first 3
fractional seconds for format yyyy-MM-dd'T'HH:mm:ss.SSS.
Usage: dateHourMinuteSecondMillis($1)
Example: dateHourMinuteSecondMillis('2015-01-01T00:00:00.000')
17.12.3.8. millisToDate¶
Description: Create a new date from a long representing milliseconds since January 1, 1970.
Usage: millisToDate($1)
Example: millisToDate('1449675054462'::long)
17.12.3.9. secsToDate¶
Description: Create a new date from a long representing seconds since January 1, 1970.
Usage: secsToDate($1)
Example: secsToDate(1449675054)
17.12.4. Geometry Functions¶
17.12.4.1. point¶
Description: Parse a Point geometry from lon/lat or WKT.
Usage: point($lon, $lat) or point($wkt)
Note: Ordering is important here…GeoMesa defaults to longitude first
Example: Parsing lon/lat from JSON:
Parsing lon/lat
# config
{ name = "lon", json-type="double", path="$.lon" }
{ name = "lat", json-type="double", path="$.lat" }
{ name = "geom", transform="point($lon, $lat)" }
# data
{
"lat": 23.9,
"lon": 24.2,
}
Example: Parsing lon/lat from text without creating lon/lat fields:
# config
{ name = "geom", transform="point($2::double, $3::double)" }
# data
id,lat,lon,date
identity1,23.9,24.2,2015-02-03
Example: Parsing WKT as a point
# config
{ name = "geom", transform="point($2)" }
# data
ID,wkt,date
1,POINT(2 3),2015-01-02
17.12.4.2. linestring¶
Description: Parse a linestring from a WKT string.
Usage: linestring($0)
Example: linestring('LINESTRING(102 0, 103 1, 104 0, 105 1)')
17.12.4.3. polygon¶
Description: Parse a polygon from a WKT string.
Usage: polygon($0)
Example: polygon('polygon((100 0, 101 0, 101 1, 100 1, 100 0))')
17.12.4.4. geometry¶
Description: Parse a geometry from a WKT string or GeoJson.
Usage: geometry($0)
Example: Parsing WKT as a geometry
# config
{ name = "geom", transform="geometry($2)" }
# data
ID,wkt,date
1,POINT(2 3),2015-01-02
Example: Parsing GeoJson geometry
# config
{ name = "geom", json-type = "geometry", path = "$.geometry" }
# data
{
id: 1,
number: 123,
color: "red",
"geometry": {"type": "Point", "coordinates": [55, 56]}
}
17.12.4.5. projectFrom¶
Description: Project a geometry from its native CRS to EPSG:4326. GeoMesa only supports EPSG:4326, so geometries must be transformed when ingesting from another CRS.
Usage: projectFrom('EPSG:3857',$0)
Example: Reprojecting a parsed point from EPSG:3857 to EPSG:4326:
# config
{ name = "geom", transform="projectFrom('EPSG:3857',point($2::double, $3::double))" }
# data
id,x,y,date
identity1,1689200.14,1113194.91,2015-02-03
17.12.5. ID Functions¶
17.12.5.1. stringToBytes¶
Description: Converts a string to a UTF-8 byte array (to pass to md5() or base64()).
Usage: stringToBytes($0)
Example: stringToBytes('row,of,data')
17.12.5.2. md5¶
Description: Creates an MD5 hash from a byte array.
Usage: md5($0)
Example: md5(stringToBytes('row,of,data'))
17.12.5.4. uuidZ3¶
Description: Generates a Z3-based UUID for point geometries.
Usage: uuidZ3($geom, $date, $interval)
Example: uuidZ3(point('POINT (3 2)'), dateTime('2015-01-01T00:00:00.000Z'), 'week')
See Configuring Z-Index Time Interval for details on Z3 intervals.
17.12.5.5. uuidZ3Centroid¶
Description: Generates a Z3-based UUID for non-point geometries.
Usage: uuidZ3Centroid($geom, $date, $interval)
Example: uuidZ3Centroid(linestring('LINESTRING(102 0, 103 1, 104 0, 105 1)', dateTime('2015-01-01T00:00:00.000Z'), 'week')
See Configuring Z-Index Time Interval for details on Z3 intervals.
17.12.5.6. base64¶
Description: Encodes a byte array as a base-64 string.
Usage; base64($0)
Example: base64(stringToBytes('foo'))
17.12.6. Type Conversions¶
17.12.6.1. ::int or ::integer¶
Description: Converts a string into an integer. Invalid values will cause the record to fail.
Example: '1'::int = 1
17.12.6.2. ::long¶
Description: Converts a string into a long. Invalid values will cause the record to fail.
Example: '1'::long = 1L
17.12.6.3. ::float¶
Description: Converts a string into a float. Invalid values will cause the record to fail.
Example: '1.0'::float = 1.0f
17.12.6.4. ::double¶
Description: Converts a string into a double. Invalid values will cause the record to fail.
Example: '1.0'::double = 1.0d
17.12.6.5. ::boolean¶
Description: Converts a string into a boolean. Invalid values will cause the record to fail.
Example: 'true'::boolean = true
17.12.6.6. ::r¶
Description: Converts a string into a Regex object.
Example: 'f.*'::r = f.*: scala.util.matching.Regex
17.12.6.7. stringToInt or stringToInteger¶
Description: Converts a string into a integer, with a default value if conversion fails.
Usage: stringToInt($1, $2)
Example: stringToInt('1', 0) = 1
Example: stringToInt('', 0) = 0
17.12.6.8. stringToLong¶
Description: Converts a string into a long, with a default value if conversion fails.
Usage: stringToLong($1, $2)
Example: stringToLong('1', 0L) = 1L
Example: stringToLong('', 0L) = 0L
17.12.6.9. stringToFloat¶
Description: Converts a string into a float, with a default value if conversion fails.
Usage: stringToFloat($1, $2)
Example: stringToFloat('1.0', 0.0f) = 1.0f
Example: stringToFloat('not a float', 0.0f) = 0.0f
17.12.6.10. stringToDouble¶
Description: Converts a string into a double, with a default value if conversion fails.
Usage: stringToDouble($1, $2)
Example: stringToDouble('1.0', 0.0) = 1.0d
Example: stringToDouble(null, 0.0) = 0.0d
17.12.6.11. stringToBoolean¶
Description: Converts a string into a boolean, with a default value if conversion fails.
Usage: stringToBoolean($1, $2)
Example: stringToBoolean('true', false) = true
Example: stringToBoolean('55', false) = false
17.12.7. Math Functions¶
Usage:
All math functions accept: Integers, Doubles, Floats, Longs and parsable Strings.
All math functions return: Doubles. If another data type is needed, convert the value afterwards. e.g. add($1,$2)::long
Example:
{ name = "value3", transform = "add($value1, multiply($value2, 1.2))::double" }
17.12.7.2. subtract¶
Description: Subtracts two or more values.
Example: subtract($1,$2)
Example: subtract($1,$2,1.0f)
17.12.7.3. multiply¶
Description: Multiply two or more values.
Example: multiply($1,$2)
Example: multiply($1,$2,0.01d)
17.12.7.4. divide¶
Description: Divides two or more values sequentially.
Example: divide($1,$2)
Example: divide($1,$2,"15") is equivalent to ($1/$2)/"15"
17.12.8. List and Map Parsing¶
17.12.8.1. parseList¶
Description: Parse a List[T] type from a string.
If your SimpleFeatureType config contains a list or map you can easily
configure a transform function to parse it using the parseList
function which takes either 2 or 3 args
- The primitive type of the list (int, string, double, float, boolean, etc)
- The reference to parse
- Optionally, the list delimiter (defaults to a comma)
Here’s some sample CSV data:
ID,Name,Age,LastSeen,Friends,Lat,Lon
23623,Harry,20,2015-05-06,"Will, Mark, Suzan",-100.236523,23
26236,Hermione,25,2015-06-07,"Edward, Bill, Harry",40.232,-53.2356
3233,Severus,30,2015-10-23,"Tom, Riddle, Voldemort",3,-62.23
For example, an SFT may specific a field:
{ name = "friends", type = "List[String]" }
And a transform to parse the quoted CSV field:
{ name = "friends", transform = "parseList('string', $5)" }
17.12.8.2. parseMap¶
Description: Parse a Map[T,V] type from a string.
Parsing Maps is similar. Take for example this CSV data with a quoted map field:
1,"1->a,2->b,3->c,4->d",2013-07-17,-90.368732,35.3155
2,"5->e,6->f,7->g,8->h",2013-07-17,-70.970585,42.36211
3,"9->i,10->j",2013-07-17,-97.599004,30.50901
Our field type is:
numbers:Map[Integer,String]
Then we specify a transform:
{ name = "numbers", transform = "parseMap('int -> string', $2)" }
Optionally we can also provide custom list/record and key-value delimiters for a map:
{ name = "numbers", transform = "parseMap('int -> string', $2, '->', ',')" }