Data Engineering

In my recent data engineering quest, I had to wrangle data from seven sources and fill our data warehouse using Azure Data Factory. The end result? Twenty-one pipelines doing a synchronized dance, populating twenty-six tables. Back in the pre-ChatGPT era, this Herculean task would’ve demanded three full-time heroes for six months. Enter ChatGPT: in just five months, with the power of 1.25 FTEs, Mission Accomplished. But that’s not all! ChatGPT Playground’s prompts turned me into a coding sorcerer....

Declarative programming will expedite the Return on Investment in your modern data warehouse implementation. I recently watched Michael Armbrust’s insightful presentation at the Data + AI Summit 2023, where he initiated the discussion by emphasizing the advantages of Declarative Programming. Throughout his talk, he delved into the innovative features of Delta Live Tables, DLT Pipelines, HMR, serverless computing, Enzyme, and more. His primary message, “Declarative programs specify what should be done, not how to do it” left a lasting impression....