19.4. JSON Converter

The JSON converter handles JSON files. To use the JSON converter, specify type = "json" in your converter definition.

19.4.1. Configuration

The JSON converter supports parsing multiple JSON documents out of a single file. In order to support JSON path expressions, each JSON document is fully parsed into memory. For large documents, this may take considerable time and memory. Thus, it is usually better to have multiple smaller JSON documents per file, when possible.

Since a single JSON document may contain multiple features, the JSON parser supports a JSONPath expression pointing to each feature element. This can be specified using the feature-path element.

The fields element in a JSON converter supports two additional attributes, path and json-type. path should be a JSONPath expression, which is relative to the feature-path, if defined (above). For absolute paths, root-path may be used instead of path. json-type should specify the type of JSON field being read. Valid values are: string, float, double, integer, long, boolean, geometry, array and object. The value will be appropriately typed, and available in the transform element as $0. Geometry types can handle either WKT strings or GeoJSON geometry objects.

19.4.2. Handling Complex Elements

JSON can contain complex, nested elements that don’t necessarily map well to the flat attribute structure used by SimpleFeatureTypes. These type of elements can be easily handled using GeoMesa’s support for JSON Attributes. In your SimpleFeatureType schema, indicate a complex JSON string through the user data hint json=true. In your converter, select the outer element and then turn it into a JSON string through the toString transformer function. You will be able to filter and transform the data using JSONPath at query time. See JSON Attributes for more details.

19.4.3. JSON Transform Functions

The transform element supports referencing the JSON element through $0. Each column will initially be typed according to the field’s json-type. Most types will be converted to the equivalent Java class, e.g. java.lang.Integer, etc. array and object types will be raw JSON elements, and thus usually require further processing (e.g. jsonList or jsonMap, below).

In addition to the standard functions in Transformation Function Overview, the JSON converter provides the following JSON-specific functions:

19.4.3.1. jsonToString

This will convert a JSON element to a string. It can be useful for quickly representing a complex object, for example in order to create a feature ID based on the hash of a row.

19.4.3.2. jsonList

This function converts a JSON array element into a java.util.List. It requires two parameters; the first is the type of the list elements as a string, and the second is a JSON array. The type of list elements must be one of the types defined in GeoTools Feature Types. See below for an example.

19.4.3.3. jsonMap

This function converts a JSON object element into a java.util.Map. It requires three parameters; the first is the type of the map key elements as a string, the second is the type of the map value elements as a string, and the third is a JSON object. The type of keys and values must be one of the types defined in GeoTools Feature Types. See below for an example.

19.4.3.4. mapToJson

This function converts a java.util.Map into a JSON string. It requires a single parameter, which must be a java.util.Map. It can be useful for storing complex JSON as a single attribute, which can then be queried using GeoMesa’s JSON attribute support. See JSON Attributes for more information.

19.4.4. Example Usage

Assume the following SimpleFeatureType:

geomesa.sfts.example = {
  attributes = [
    { name = "name",    type = "String"          }
    { name = "age",     type = "Integer"         }
    { name = "weight",  type = "Double"          }
    { name = "hobbies", type = "List[String]"    }
    { name = "skills",  type = "Map[String,Int]" }
    { name = "source",  type = "String"          }
    { name = "geom",    type = "Point"           }
  ]
}

And the following JSON document:

{
  "DataSource": { "name": "myjson" },
  "Features": [
    {
      "id": 1,
      "name": "phil",
      "physicals": {
        "age": 32,
        "weight": 150.2
      },
      "hobbies": [ "baseball", "soccer" ],
      "languages": {
        "java": 100,
        "scala": 70
      },
      "geometry": { "type": "Point", "coordinates": [55, 56] }
    },
    {
      "id": 2,
      "name": "fred",
      "physicals": {
        "age": 33,
        "weight": 150.1
      },
      "hobbies": [ "archery", "tennis" ],
      "languages": {
        "c++": 10,
        "fortran": 50
      },
      "geometry": { "type": "Point", "coordinates": [45, 46] }
    }
  ]
}

You could ingest with the following converter:

geomesa.converters.myjson = {
  type         = "json"
  id-field     = "$id"
  feature-path = "$.Features[*]"
  fields = [
    { name = "id",      json-type = "integer",  path = "$.id",               transform = "toString($0)"                }
    { name = "name",    json-type = "string",   path = "$.name",             transform = "trim($0)"                    }
    { name = "age",     json-type = "integer",  path = "$.physicals.age",                                              }
    { name = "weight",  json-type = "double",   path = "$.physicals.weight"                                            }
    { name = "hobbies", json-type = "array",    path = "$.hobbies",          transform = "jsonList('string', $0)"      }
    { name = "skills",  json-type = "map",      path = "$.languages",        transform = "jsonMap('string','int', $0)" }
    { name = "geom",    json-type = "geometry", path = "$.geometry",         transform = "point($0)"                   }
    { name = "source",  json-type = "string",   root-path = "$.DataSource.name"                                        }
  ]
}