User Manual

Table of Contents

• 1. Introduction
  • 1.1. What is GeoMesa?
  • 1.2. Community and Support
  • 1.3. License
• 2. Versions and Downloads
  • 2.1. Release Distributions
  • 2.2. Maven Integration
  • 2.3. Source Code
  • 2.4. Upgrading
  • 2.5. Legal Review
• 3. Installation
  • 3.1. Upgrading
• 4. Getting Started
  • 4.1. Quick Start
  • 4.2. GeoDocker: Bootstrapping GeoMesa Accumulo and Spark on AWS
  • 4.3. GeoMesa Kafka
  • 4.4. Storm Analysis
  • 4.5. GeoJSON
• 5. Architecture Overview
  • 5.1. GeoMesa and GeoServer
  • 5.2. Integration with GeoMesa
  • 5.3. Key-value stores and Z-curves
  • 5.4. GeoMesa Indexing Key
• 6. GeoMesa Data Stores
  • 6.1. GeoTools Feature Types
    • 6.1.1. Available Types
  • 6.2. Index Overview
  • 6.3. Index Basics
    • 6.3.1. Spatial Index (Z2/XZ2)
    • 6.3.2. Spatio-temporal Index (Z3/XZ3)
    • 6.3.3. ID Index
    • 6.3.4. Attribute Index
  • 6.4. Index Versioning
  • 6.5. Index Configuration
    • 6.5.1. Setting Schema Options
    • 6.5.2. Setting Attribute Options
    • 6.5.3. Setting the Indexed Date Attribute
    • 6.5.4. Customizing Index Creation
    • 6.5.5. Configuring Feature ID Encoding
    • 6.5.6. Configuring Z-Index Shards
    • 6.5.7. Configuring Z-Index Time Interval
    • 6.5.8. Configuring XZ-Index Precision
    • 6.5.9. Configuring Attribute Index Shards
    • 6.5.10. Configuring Attribute Cardinality
    • 6.5.11. Configuring Index Splits
    • 6.5.12. Configuring Cached Statistics
    • 6.5.13. Mixed Geometry Types
  • 6.6. Runtime Configuration
    • 6.6.1. Common Properties
  • 6.7. Query Planning
    • 6.7.1. Filter Decomposition
    • 6.7.2. Index Selection
  • 6.8. Explaining Query Plans
    • 6.8.1. GeoServer
  • 6.9. Query Properties
    • 6.9.1. Setting Query Hints
    • 6.9.2. Loose Bounding Box
    • 6.9.3. Exact Counts
    • 6.9.4. Query Index
    • 6.9.5. Query Planning Type
  • 6.10. Analytic Querying
    • 6.10.1. Feature Sampling
    • 6.10.2. Density Queries
    • 6.10.3. Statistical Queries
    • 6.10.4. Arrow Encoding
    • 6.10.5. Binary Encoding
  • 6.11. Authorizations
  • 6.12. Query Auditing
  • 6.13. Moving and Migrating Data
  • 6.14. Reserved Words
    • 6.14.1. Override
• 7. Command-Line Tools
  • 7.1. Environment and Status Commands
    • 7.1.1. classpath
    • 7.1.2. env
    • 7.1.3. help
    • 7.1.4. version
    • 7.1.5. version-remote
  • 7.2. Schema Commands
    • 7.2.1. create-schema
    • 7.2.2. delete-catalog
    • 7.2.3. describe-schema
    • 7.2.4. gen-avro-schema
    • 7.2.5. get-sft-config
    • 7.2.6. get-type-names
    • 7.2.7. keywords
    • 7.2.8. remove-schema
  • 7.3. Ingest Commands
    • 7.3.1. delete-features
    • 7.3.2. ingest
  • 7.4. Query and Export Commands
    • 7.4.1. convert
    • 7.4.2. explain
    • 7.4.3. export
  • 7.5. Analytic Commands
    • 7.5.1. stats-bounds
    • 7.5.2. stats-count
    • 7.5.3. stats-histogram
    • 7.5.4. stats-top-k
  • 7.6. Defining Simple Feature Types
    • 7.6.1. Included Feature Types
    • 7.6.2. Classpath Loading
    • 7.6.3. Specification String
    • 7.6.4. Configuration String
    • 7.6.5. File Name
  • 7.7. Defining Simple Feature Converters
    • 7.7.1. Included Converters
    • 7.7.2. Classpath Loading
    • 7.7.3. Configuration String
    • 7.7.4. File Name
  • 7.8. Logging Configuration
  • 7.9. Remote File System Support
    • 7.9.1. Enabling S3 Ingest
    • 7.9.2. Enabling Azure Ingest
  • 7.10. GeoMesa Scala Console
    • 7.10.1. Example Usage
• 8. Accumulo Data Store
  • 8.1. Installing GeoMesa Accumulo
    • 8.1.1. Installing from the Binary Distribution
    • 8.1.2. Building from Source
    • 8.1.3. Installing the Accumulo Distributed Runtime Library
    • 8.1.4. Setting up the Accumulo Command Line Tools
    • 8.1.5. Installing GeoMesa Accumulo in GeoServer
    • 8.1.6. Upgrading
  • 8.2. Accumulo Configuration
    • 8.2.1. Batch Writer Properties
    • 8.2.2. Map Reduce Input Splits Properties
    • 8.2.3. Zookeeper Session Timeout
  • 8.3. Using the Accumulo Data Store Programmatically
    • 8.3.1. Creating a Data Store
    • 8.3.2. Accumulo Data Store Parameters
  • 8.4. Using the Accumulo Data Store in GeoServer
  • 8.5. Accumulo Command-Line Tools
    • 8.5.1. General Arguments
    • 8.5.2. Commands
  • 8.6. Command Line Examples
    • 8.6.1. Feature Type Management
    • 8.6.2. Ingesting Data
    • 8.6.3. Exporting Features
  • 8.7. Accumulo Visibilities
    • 8.7.1. Data Store Level Visibilities
    • 8.7.2. Feature Level Visibilities
    • 8.7.3. Attribute-Level Visibilities
  • 8.8. Data Management
    • 8.8.1. Accumulo Attribute Indices
    • 8.8.2. Accumulo Logical Timestamps
    • 8.8.3. Table Sharing
    • 8.8.4. Upgrading Existing Indices
    • 8.8.5. Accumulo Index Versions
  • 8.9. GeoMesa Jobs
    • 8.9.1. Building Instructions
    • 8.9.2. GeoMesa Input and Output Formats
    • 8.9.3. Map/Reduce Jobs
  • 8.10. GeoMesa Raster
    • 8.10.1. Installation
    • 8.10.2. Ingest
    • 8.10.3. GeoServer
  • 8.11. Internals
    • 8.11.1. Indexing Strategies
    • 8.11.2. Iterator Stack
  • 8.12. Kerberos
    • 8.12.1. Development & Testing
  • 8.13. Age-Off Iterators
    • 8.13.1. Installation
    • 8.13.2. Requirements
    • 8.13.3. Configuration
    • 8.13.4. Statistics
    • 8.13.5. Forcing Deletion of Records
• 9. HBase Data Store
  • 9.1. Installing GeoMesa HBase
    • 9.1.1. Installing the Binary Distribution
    • 9.1.2. Building from Source
    • 9.1.3. Configuration and Classpaths
    • 9.1.4. Deploying the GeoMesa HBase Distributed Runtime JAR
    • 9.1.5. Register the Coprocessors
    • 9.1.6. Setting up the HBase Command Line Tools
    • 9.1.7. Installing GeoMesa HBase in GeoServer
    • 9.1.8. Connecting to External HBase Clusters Backed By S3
    • 9.1.9. Configuring HBase on Azure HDInsight
  • 9.2. HBase Configuration
    • 9.2.1. geomesa.hbase.config.paths
    • 9.2.2. geomesa.hbase.remote.filtering
  • 9.3. Using the HBase Data Store Programmatically
    • 9.3.1. Creating a Data Store
    • 9.3.2. HBase Data Store Parameters
  • 9.4. Using the HBase Data Store in GeoServer
  • 9.5. HBase Command-Line Tools
    • 9.5.1. General Arguments
    • 9.5.2. Commands
  • 9.6. HBase Index Configuration
    • 9.6.1. Setting File Compression
  • 9.7. HBase Index Versions
  • 9.8. HBase Heatmaps
    • 9.8.1. Install Heatmap SLD
  • 9.9. HBase Visibilities
    • 9.9.1. Setup and Configuration
    • 9.9.2. Data Store Level Visibilities
    • 9.9.3. Feature Level Visibilities
    • 9.9.4. Known Issues
  • 9.10. Kerberos
    • 9.10.1. Development & Testing
    • 9.10.2. Managing Hadoop and HBase configurations on the classpath
    • 9.10.3. Enabling Kerberos on HDP
• 10. Bigtable Data Store
  • 10.1. Installing GeoMesa Bigtable
    • 10.1.1. Installing the Binary Distribution
    • 10.1.2. Installing GeoMesa Bigtable in GeoServer
  • 10.2. Using the Bigtable Data Store Programmatically
    • 10.2.1. Creating a Data Store
    • 10.2.2. Bigtable Data Store Parameters
  • 10.3. Using the Bigtable Data Store in GeoServer
  • 10.4. Bigtable Command-Line Tools
    • 10.4.1. General Arguments
  • 10.5. GeoMesa Spark SQL on Google Cloud Dataproc
• 11. Cassandra Data Store
  • 11.1. Installing GeoMesa Cassandra
    • 11.1.1. Connecting to Cassandra
    • 11.1.2. Installing from the Binary Distribution
    • 11.1.3. Building from Source
    • 11.1.4. Setting up the Cassandra Command Line Tools
    • 11.1.5. Installing GeoMesa Cassandra in GeoServer
  • 11.2. Using the Cassandra DataStore Programmatically
    • 11.2.1. Creating a Data Store
    • 11.2.2. Cassandra Data Store Parameters
  • 11.3. Using the Cassandra Data Store in GeoServer
  • 11.4. Cassandra Command-Line Tools
    • 11.4.1. General Arguments
  • 11.5. Cassandra Heatmaps
  • 11.6. Cassandra Index Versions
  • 11.7. Cassandra Configuration
    • 11.7.1. Connection Properties
• 12. Kafka Data Store
  • 12.1. Installing GeoMesa Kafka
    • 12.1.1. Installing from the Binary Distribution
    • 12.1.2. Building from Source
    • 12.1.3. Setting up the Kafka Command Line Tools
    • 12.1.4. Installing GeoMesa Kafka in GeoServer
  • 12.2. Using the Kafka Data Store Programmatically
    • 12.2.1. Creating a Data Store
    • 12.2.2. Kafka Data Store Parameters
  • 12.3. Using the Kafka Data Store in GeoServer
  • 12.4. Kafka Command-Line Tools
    • 12.4.1. General Arguments
    • 12.4.2. Commands
  • 12.5. Data Producers
  • 12.6. Data Consumers
  • 12.7. Listening for Feature Events
• 13. Lambda Data Store
  • 13.1. Overview of the Lambda Data Store
    • 13.1.1. Basic Architecture
    • 13.1.2. Alternate Solutions
    • 13.1.3. Integration with Other Data Stores
  • 13.2. Installing GeoMesa Lambda
    • 13.2.1. Installing from the Binary Distribution
    • 13.2.2. Building from Source
    • 13.2.3. Installing the Accumulo Distributed Runtime Library
    • 13.2.4. Setting up the Lambda Command Line Tools
    • 13.2.5. Installing GeoMesa Lambda in GeoServer
    • 13.2.6. Upgrading
  • 13.3. Using the Lambda Data Store Programmatically
    • 13.3.1. Creating a Data Store
    • 13.3.2. Lambda Data Store Parameters
  • 13.4. Using the Lambda Data Store in GeoServer
  • 13.5. Lambda Command-Line Tools
    • 13.5.1. General Arguments
  • 13.6. Lambda Data Store Configuration
    • 13.6.1. Data Store Parameters
    • 13.6.2. Miscellaneous Properties
  • 13.7. Advanced Topics
    • 13.7.1. Parallelism
    • 13.7.2. Installation Tips
    • 13.7.3. Manual Persistence
    • 13.7.4. Monitoring State
• 14. FileSystem Datastore (Amazon S3, HDFS Datastore)
  • 14.1. FileSystem Datastore Architecture
    • 14.1.1. FileSystem
    • 14.1.2. Partition Schemes
    • 14.1.3. Storage Formats
  • 14.2. Partition Schemes
    • 14.2.1. Leaf vs Bucket Storage
    • 14.2.2. JSON Configuration
  • 14.3. Installing GeoMesa FileSystem
    • 14.3.1. Installing from the Binary Distribution
    • 14.3.2. Building from Source
    • 14.3.3. Setting up the FileSystem Command Line Tools
    • 14.3.4. Installing GeoMesa FileSystem in GeoServer
  • 14.4. FileSystem DataStore Configuration
    • 14.4.1. File Writers
  • 14.5. Using the FileSystem Data Store Programmatically
    • 14.5.1. Creating a Data Store
    • 14.5.2. FileSystem Data Store Parameters
  • 14.6. Using the FileSystem Data Store in GeoServer
  • 14.7. FileSystem Command-Line Tools
    • 14.7.1. General Arguments
    • 14.7.2. Commands
  • 14.8. FileSystem Datastore Example
  • 14.9. GeoMesa FileSystem Data Store with Spark SQL
    • 14.9.1. Ingesting GDELT
    • 14.9.2. Querying GDELT with Spark SQL
• 15. GeoServer Plugins
  • 15.1. Installation
  • 15.2. Creating a Data Store
  • 15.3. Publish a GeoMesa Layer
  • 15.4. Preview a Layer
  • 15.5. Analysis with WPS
  • 15.6. Configuration
    • 15.6.1. WMS Plugin
    • 15.6.2. Logging Explain Query Planning
    • 15.6.3. Auditing GeoMesa DataStores
  • 15.7. GeoMesa GeoServer Extensions
• 16. GeoMesa Spark
  • 16.1. Architecture
  • 16.2. Spark JTS
    • 16.2.1. Example
    • 16.2.2. Configuration
    • 16.2.3. Geospatial User-defined Types and Functions
    • 16.2.4. GeoJSON Output
    • 16.2.5. Building
  • 16.3. Spark Core
    • 16.3.1. Example
    • 16.3.2. Configuration
    • 16.3.3. Simple Feature Serialization
    • 16.3.4. Usage
  • 16.4. Spatial RDD Providers
    • 16.4.1. Accumulo RDD Provider
    • 16.4.2. HBase RDD Provider
    • 16.4.3. FileSystem RDD Provider
    • 16.4.4. Converter RDD Provider
    • 16.4.5. GeoTools RDD Provider
  • 16.5. SparkSQL
    • 16.5.1. Example
    • 16.5.2. Configuration
    • 16.5.3. Usage
    • 16.5.4. Geospatial User-defined Types and Functions
    • 16.5.5. In-memory Indexing
    • 16.5.6. Spatial Partitioning and Faster Joins
  • 16.6. SparkSQL Functions
    • 16.6.1. Geometry Constructors
    • 16.6.2. Geometry Accessors
    • 16.6.3. Geometry Cast
    • 16.6.4. Geometry Editors
    • 16.6.5. Geometry Outputs
    • 16.6.6. Spatial Relationships
    • 16.6.7. Geometry Processing
  • 16.7. GeoMesa PySpark
    • 16.7.1. Prerequisites
    • 16.7.2. Installation
    • 16.7.3. Using Geomesa PySpark
    • 16.7.4. Jupyter
  • 16.8. Deploying GeoMesa Spark with Jupyter Notebook
    • 16.8.1. Prerequisites
    • 16.8.2. Installing Jupyter
    • 16.8.3. Installing the Toree Kernel
    • 16.8.4. Configure Toree and GeoMesa
    • 16.8.5. Vegas for Data Plotting
    • 16.8.6. Running Jupyter
  • 16.9. Deploying GeoMesa Spark with Zeppelin
    • 16.9.1. Installing Zeppelin
    • 16.9.2. Configuring Zeppelin with GeoMesa
    • 16.9.3. Data Plotting
• 17. GeoMesa Convert
  • 17.1. Converter Basics
    • 17.1.1. Defining SimpleFeatureTypes
    • 17.1.2. Defining Converters
  • 17.2. Parsing and Validation
    • 17.2.1. Validators
    • 17.2.2. Error Mode
    • 17.2.3. Parse Mode
    • 17.2.4. Logging
    • 17.2.5. Transactional Considerations
  • 17.3. Delimited Text Converter
    • 17.3.1. Configuration
    • 17.3.2. Transform Functions
    • 17.3.3. Example Usage
  • 17.4. JSON Converter
    • 17.4.1. Configuration
    • 17.4.2. Transform Functions
    • 17.4.3. Example Usage
  • 17.5. XML Converter
    • 17.5.1. Configuration
    • 17.5.2. Transform Functions
    • 17.5.3. Example Usage
    • 17.5.4. Handling Namespaces with Saxon
  • 17.6. Avro Converter
    • 17.6.1. AvroPath
  • 17.7. Fixed-Width Text Converter
    • 17.7.1. Configuration
    • 17.7.2. Transform Functions
    • 17.7.3. Example Usage
  • 17.8. JDBC Converter
    • 17.8.1. Configuration
    • 17.8.2. Transform Functions
    • 17.8.3. Command Line Ingestion
    • 17.8.4. Example Usage
  • 17.9. Using Caches for Enrichment
  • 17.10. Prepackaged Converter Definitions
    • 17.10.1. Global Database of Events, Language, and Tone (GDELT)
    • 17.10.2. GeoLife GPS Trajectory Data
    • 17.10.3. GeoNames
    • 17.10.4. Global Terrorism Database (GTD)
    • 17.10.5. NYC Taxi
    • 17.10.6. OpenStreetMap Data
    • 17.10.7. OpenStreetMap GPX Data
    • 17.10.8. T-Drive Trajectory Data
    • 17.10.9. Twitter
  • 17.11. Transformation Function Overview
    • 17.11.1. Control Functions
    • 17.11.2. String Functions
    • 17.11.3. Date Functions
    • 17.11.4. Geometry Functions
    • 17.11.5. ID Functions
    • 17.11.6. Type Conversions
    • 17.11.7. List and Map Parsing
    • 17.11.8. Functions defined using scripting languages
    • 17.11.9. Installing Custom Scripts
    • 17.11.10. CQL Functions
    • 17.11.11. JSON/Avro Transformations
    • 17.11.12. Enrichment Functions
  • 17.12. Transformation Function Usage
    • 17.12.1. Control Functions
    • 17.12.2. String Functions
    • 17.12.3. Date Functions
    • 17.12.4. Geometry Functions
    • 17.12.5. ID Functions
    • 17.12.6. Type Conversions
    • 17.12.7. Math Functions
    • 17.12.8. List and Map Parsing
    • 17.12.9. Enrichment Functions
  • 17.13. Using SFT and Converter Definitions with Command-Line Tools
  • 17.14. Loading Converters and SFTs at Runtime
  • 17.15. Extending the Converter Library
    • 17.15.1. Adding Scripting Functions
    • 17.15.2. Adding New Transformation Functions
    • 17.15.3. Adding New Data Formats
    • 17.15.4. Adding Functions to the Geomesa Classpath
• 18. GeoMesa Processes
  • 18.1. Installation
  • 18.2. Processors
    • 18.2.1. ArrowConversionProcess
    • 18.2.2. BinConversionProcess
    • 18.2.3. DensityProcess
    • 18.2.4. DateOffsetProcess
    • 18.2.5. HashAttributeProcess
    • 18.2.6. HashAttributeColorProcess
    • 18.2.7. JoinProcess
    • 18.2.8. KNearestNeighborProcess
    • 18.2.9. Point2PointProcess
    • 18.2.10. ProximitySearchProcess
    • 18.2.11. RouteSearchProcess
    • 18.2.12. SamplingProcess
    • 18.2.13. StatsProcess
    • 18.2.14. TrackLabelProcess
    • 18.2.15. TubeSelectProcess
    • 18.2.16. QueryProcess
    • 18.2.17. UniqueProcess
    • 18.2.18. Chaining Processes
• 19. GeoMesa GeoJSON
  • 19.1. GeoJSON API
    • 19.1.1. Adding and Updating Features
    • 19.1.2. Querying Features
    • 19.1.3. Programmatic Access
    • 19.1.4. REST Access
    • 19.1.5. Methods
  • 19.2. JSON Attributes
    • 19.2.1. JSONPath CQL Filter Function
• 20. GeoMesa NiFi Bundle
  • 20.1. Installation
    • 20.1.1. Build and Install the Processors
    • 20.1.2. Install the SFTs and Converters
  • 20.2. Processors
    • 20.2.1. Input Configuration
    • 20.2.2. PutGeoMesaAccumulo
    • 20.2.3. PutGeoMesaHBase
    • 20.2.4. PutGeoMesaFileSystem
    • 20.2.5. PutGeoMesaKafka
    • 20.2.6. PutGeoTools
    • 20.2.7. ConvertToGeoAvro
  • 20.3. NiFi User Notes
    • 20.3.1. GetHDFS Processor with Azure Integration
  • 20.4. Reference
• 21. GeoMesa Blob Store
  • 21.1. FileHandler Interface
  • 21.2. BlobStore Interface
  • 21.3. Provided File Handlers
    • 21.3.1. GeoMesa-BlobStore-EXIF
    • 21.3.2. GeoMesa-BlobStore-GDAL
  • 21.4. GeoMesa-BlobStore-Accumulo
  • 21.5. REST API
  • 21.6. Servlet Configuration Options
• 22. GeoMesa Native API
  • 22.1. Usage
• 23. GeoMesa Metrics
  • 23.1. Instrumentation of GeoServer
  • 23.2. Installation
    • 23.2.1. Ganglia Reporting
    • 23.2.2. Delimited File Reporting
  • 23.3. Configuration
  • 23.4. Session Tracking
  • 23.5. Configuration of Reporters
  • 23.6. Standard Configuration
    • 23.6.1. Console Reporter
    • 23.6.2. Slf4j Reporter
    • 23.6.3. Delimited Text Reporter
    • 23.6.4. Graphite Reporter
    • 23.6.5. Ganglia Reporter
    • 23.6.6. Accumulo Reporter
• 24. GeoMesa Stream Processing
  • 24.1. Modules
  • 24.2. Usage
  • 24.3. UDP
• 25. GeoMesa Web Data
  • 25.1. Building Instructions
  • 25.2. Installation Instructions
    • 25.2.1. Tomcat Installation
    • 25.2.2. Jboss Installation
  • 25.3. Advanced Configuration
    • 25.3.1. Distributed Jars
    • 25.3.2. Distributed Classloading
  • 25.4. Analytic Web Service
    • 25.4.1. Endpoints
    • 25.4.2. Response Formats
    • 25.4.3. Example requests
  • 25.5. Appendix A: GeoServer Jars
• 26. Upgrade Guide
  • 26.1. Compatibility Across Versions
    • 26.1.1. Semantic Versioning
    • 26.1.2. Compatibility
    • 26.1.3. Compatibility Matrix
  • 26.2. Version 2.0.0 Upgrade Guide
    • 26.2.1. GeoTools 18 and GeoServer 2.12
    • 26.2.2. Data Store Parameters
    • 26.2.3. Removal of Joda Time
    • 26.2.4. Saxon XML Parser
    • 26.2.5. Kafka Data Store
    • 26.2.6. Accumulo Standardization
    • 26.2.7. Table Splitters
    • 26.2.8. System Properties

Appendix

• GeoMesa Utils
  • Simple Feature Wrapper Generation
  • Geohash