Baking Quality into Your Data Pipeline with Ali Khalid

@aali.khalid is taking to the TestBash New Zealand stage to talk to us about big data pipelines :grin: What data pipelines look like, how we can test them and how do we automate them.

If you’ve got questions for Ali about the talk, add them here :point_down:

Remember to :heart: questions that you like to show appreciation for others and help @jamesespie see what questions are popular to ask live.

For a lot of the Quality checks you discussed. Are these things typically run in a test environment or more in production?


What are the most common bugs with data pipelines? Is there a particular section that is more fragile than the others?


In your experience @aali.khalid, does the business model get defined clearly for you to perform your tests? If not, how do you tackle on-the-fly scenarios?