On this workshop, I will show how to set a simple IoT architecture, similar that the one I helped to develop for a global automotive corporation, using Apache Flume as a configurable pipeline. In this case, instead of car data, we'll consume real-time civil aeroplane radio messages.
we'll set the smallest architecture to process a real-time streaming of events. From its creation, using MQTT protocol, until its processing with Apache Kafka, HDFS and a live NodeJS reporting system.
we'll set up all the pieces of the system, one by one, using docker images.

What we'll learn:

- Minimal parts that compose an IoT-based architecture.
- How to use a new-old player in data ingestion architectures that it's not so widely known: Apache Flume, that will help us to consume, enrich and adapt MQTT messages. We'll give an overall vision of its pipelining capabilities and how could we route messages to the other elements of the system.
- Thus, we will see how our messages are sent to Kafka, saved to HDFS and visualised through a NodeJs dashboard.
[The workshop will be git-repository guided. So all the initial setup information, steps and exercises will be placed in different branches, to be able to follow the progression even if an exercise wasn't finished]


Comments are closed.

Provocador as usual

Angel Rayo at 15:55 on 30 Sep 2018

Desconocía por completo la cantidad de datos que tenemos a nuestro alcance utilizando una simple antena. Ver cómo con una Raspberry, una antena y datos de vuelos podemos construir una app que consuma y procese datos me ha parecido muy interesante :-)

Muchas gracias.

Sergio Ferraz at 18:27 on 30 Sep 2018


Fran García at 15:39 on 10 Oct 2018

The topic was pretty interesting however we only had time to cover the first point.