Quality

In 1973, the science fiction author Arthur C. Clarke wrote what would become perhaps the most quoted observation in the history of technology: “Any sufficiently advanced technology is indistinguishable from magic.” I have always liked this quote, because he was right. Do you know how a quartz clock works? I’ll give you a hint: it’s vibrating crystals. Turns out the holistic hippies were right, crystals are magic. Right now in my pocket I have at my fingertips essentially all human knowledge and information. Somewhere in a server farm in Virginia, a computer the size of a wardrobe decides that I need to be enraged at a video of a monkey being bullied. This thinking silicon (a rock) converts this information into pulses of light, fires them down a glass wire thinner than a human hair, sends them under the Atlantic Ocean at roughly the speed of light, up through a cable landing station on a beach in Portugal that looks completely unremarkable from the outside, across Europe, under more oceans, up through the seafloor off the coast of Queensland, through a series of increasingly unimpressive concrete buildings, through the air invisibly as radio waves, through my wall, and into a small rectangle of glass and metal in my hand. The monkey has no idea any of this happened. ...

It’s 2025, and the last few years has seen an explosion of GenAI-driven tools in knowledge work like software engineering. Technologies like GitHub Copilot, Cursor and large language models like ChatGPT now allow developers to generate entire applications with just a few keystrokes. If you spend any time on LinkedIn or other haunts of tech influencers, AI tools are often presented like a magic wand: type a prompt, hit enter, and tada! An entire application is built, fully integrated, with no need for you as a developer to ever do pesky things like read docs or write tests. Just ask the right question, and the code appears as if by magic. Gone are the days where engineering required knowledge and experience to do a good job. We have outsourced that to the machine. ...

When developing software, there is a natural tension between “speed” and “quality”. They are often seen as opposing goals. Delivery teams are under pressure to ship new features as fast as possible, but the reliability and functionality of their systems can not be sacrificed. In fields like data engineering, this is especially apparent. The need to quickly gain new insights is real, but sacrificing on quality can result in the wrong insights, which is worse than no insights at all!. Does this relationship between speed and quality have to be a dichotomy? What if we could move fast, but also make sure we are delivering the kind of work we can be proud of as professionals? ...