Data Pipelines

Once we built APIs. OK, we still build APIs, but that's not all we build. We build streams and queues and other things too, also technically interfaces, but they start to take on a different tone when we use them to feed our applications, to feed our reports. We start to call these data pipelines, because it expresses the important thing here: that the data gets delivered.

I like to do a little more with my data. I also like to make it reliable, so when it shows up messy, or when my work on it makes a mistake, I can recover. I also like to handle common issues so that I don't have to attend all the meetings to make excuses for things I forgot to do.

This talk builds a foundation for the fundamentals you'll need to build or refine your own data pipelines. Whether you're batching, streaming, or MapReducing your data, you can setup basic principles to ensure you're starting strong. [36]

Comments

Comments are closed.