ApacheCon North America 2017

A common theme in the IoT space is the need for large volume data streaming, ingestion and storage, and post-ingestion processing and analytics all of which depend on an efficient, scalable and well-performing data model.

With intelligent transportation picking up traction as an IoT showcase, this presentation will take the usecase of vehicle-to-infrastructure (V2I) data exchange for intelligent vehicle systems, and walk through a high level data schema and datastore design approach to support billions of vehicles and hundreds of billions of daily data events. It should be noted that at these volumes, effective and efficient schema-level indexing is not practical.

The proposed design borrows a page from the venerable Unix Filesystem inode structure and can be implemented on datastores like Apache Cassandra and Apache HBase.