12.8. Ingress Processors¶
12.8.1. GetGeoMesaKafkaRecord¶
The GetGeoMesaKafkaRecord processor provides the ability to read messages written by the GeoMesa Kafka data store
and output them as NiFi records for further processing.
Warning
The GetGeoMesaKafkaRecord has not been tested with multiple processor threads, and may not work as expected.
kafka.consumer.count can be used to configure the number of consumer threads in a given processor.
Property |
Description |
|---|---|
kafka.brokers |
The Kafka brokers, in the form of |
kafka.zookeepers |
The Kafka zookeepers, in the form of |
kafka.zk.path |
The zookeeper discoverable path, used to namespace schemas |
Type Name |
The simple feature type name to read |
Kafka Group ID |
The Kafka consumer group ID, used to track messages read |
Record Writer |
The NiFi record writer service used to serialize records |
Geometry Serialization Format |
The format to use for serializing geometries, either text or binary |
Include Visibilities |
Include a column with visibility expressions for each row |
Include User Data |
Include a column with user data from the SimpleFeature, serialized as JSON |
Record Maximum Batch Size |
The maximum number of records to output in a single flow file |
Record Minimum Batch Size |
The minimum number of records to output in a single flow file |
Record Max Latency |
The maximum delay between receiving a message and writing it out as a flow file. Takes precedence over minimum batch size if both are set |
Consumer Poll Timeout |
The amount of time to wait for new records before writing out a flow file, subject to batch size restrictions |
Kafka Initial Offset |
The initial offset to use when reading messages from a new topic |
kafka.consumer.count |
The number of consumers (threads) to use for reading messages |
kafka.consumer.config |
Configuration options for the kafka consumer, in Java properties format |
Note that any processors with the same Kafka Group ID will split messages between the processors, as per standard Kafka consumer group behavior. Generally this is not desirable, and a unique group ID should be used for each processor.
12.8.1.1. Attributes¶
The GetGeoMesaKafkaRecord will set the following NiFi expression attributes, for use in the configured record writer:
Attribute |
Description |
|---|---|
|
The name of the Feature ID column in the output record |
|
The name and types of any geometry columns in the output record, comma-separated |
|
The name of the default date column in the output record |
|
The name of any JSON-type string columns in the output record, comma-separated |
|
The name of the visibilities column in the output record |
These properties correspond to the default configuration of the GeoMesa Record Processors, so generally no additional configuration is needed to read from Kafka and write to another data store.