Upgrading to Airflow 2.0We ran into a couple of gotchas during the update that weren’t covered by the excellent migration document that the Airflow team put…Dec 18, 2020Dec 18, 2020
Airflow > Inject REST API Parameters Into Docker/Kubernetes OperatorsThis is a rough draft, but I wanted to get my thoughts out into the ether because I wasn’t able to find a solution on Google. Hopefully…May 19, 2020May 19, 2020
Snowflake Deployment Quick StartWhen deploying a new Snowflake instance these are some of the steps that I typically perform to stand up the instance (e.g. clean up…Apr 22, 2020Apr 22, 2020
Airflow at LirioLirio uses Airflow extensively to define and automate data engineering, data science and machine learning jobs across 4 environments. We…Feb 4, 2020Feb 4, 2020
Thoughts on scaling from 10GB to 10TB with Snowflake.This story has been on my mind for awhile now. I wanted to write this up around my one year anniversary at Lirio and look back on the…Dec 16, 2019Dec 16, 2019
BERT-serving worker fails (silently) to start within Kubernetes pod.Silently failing is bad. Don’t do that.Jul 26, 2019Jul 26, 2019
Data, Privacy and Stewardship of Consumer DataI had an interesting conversation at lunch today about data, privacy and the responsible stewardship of consumer data. These are important…Jul 19, 2019Jul 19, 2019
Airflow > Kubernetes Pod Operator & Resource LimitsI ran into an issue this week with a job consuming all available resources and crashing the host so I started exploring the operator’s…Jul 16, 2019Jul 16, 2019
