I would like to demonstrate how to use this simple statistical rule to capture outliers represented as invalid file loads in your etl process and pass them as documents into elasticsearch.setup will consist of Sql server hosting your etl Logging database, scheduled, stored procedure calculating The empirical rule and raising events into The windows event Log ,winlogbeat passing these events to elasticsearch and also kibana will be used for visualization. During The presentation, I would like to show how to get this setup running on your local machine to try out and talk further about The pros / cons of this solution.

Comments

Please login to leave a comment

Whole system explained here is just an example (on windows platform) but similar approach is doable on linux and with various databases and monitoring systems. I like idea about statistical approach for alerting instead of fixed boundaries, so maybe presenter should have spend more time on useful statistic methods here then on winlogbeat and specific technologies.