Engineering

When developing software, there is a natural tension between “speed” and “quality”. They are often seen as opposing goals. Delivery teams are under pressure to ship new features as fast as possible, but the reliability and functionality of their systems can not be sacrificed. In fields like data engineering, this is especially apparent. The need to quickly gain new insights is real, but sacrificing on quality can result in the wrong insights, which is worse than no insights at all!. Does this relationship between speed and quality have to be a dichotomy? What if we could move fast, but also make sure we are delivering the kind of work we can be proud of as professionals? ...

Creating software for a living and keeping up with the industry and latest trends can be overwhelming. Every day brings a new framework, a new technology, a new architectural pattern. One of my favourite parodies of this fact is Days Since Last JavaScript Framework (spoiler, the answer is 0 days, every day). In the data space, the latest thing appears to be the data lakehouse pattern, which although not new has had a resurgence because of Apache Iceberg and the idea of distributed compute. And, anyone not under a rock has seen the explosion of Generative AI across the industry, with everyone trying to shoehorn AI into their product, whether that’s what their users want or not. With the pace of change, it is hard not to think that you’re being left behind. For some, this results in a practice I recently heard described as Resume Driven Development. ...