The metadata problem at scale Automattic’s data ecosystem is large and highly interconnected: thousands of Iceberg tables and views queried via Trino, thousands of Airflow tasks producing and updating them via Spark jobs, and a growing catalogue of Looker and Superset dashboards and charts on top. Most of the information about these assets exists somewhere, scattered…
Data for Breakfast 