What’s possible today that was unthinkable a decade ago? And how might we start imagining the possibilities a decade from now?
On the latest episode of Collaborative Craft, I met over Zoom with Brad Ediger, 8th Light's Head of Technology, about the state of data engineering — past, present, and future. I took the opportunity to scratch beneath the surface and understand what drives Brad’s notorious optimism.
Relentless Optimism
Kicking off a new series on all things data, Brad and I dive into common challenges businesses face when dealing with data, before doing a bit of time-traveling — looking at what’s possible today that was unthinkable 10 years ago, and speculating on where the industry might be a decade from now.
Whether or not machine learning ends up supplanting handwritten algorithms in some lofty way in 10 or 20 years, whether or not we're still writing code — the lessons from this process, the lessons from understanding what machine learning needs of data, what it needs of data architectures and of data pipelines, and what sort of new concepts it puts into our heads, is going to be incredibly useful to the industry.
Through trial, error, and informed repetition, we’re still in the process of building the software that shapes our world. That perspective encourages the idea that we’re still building the fundamental blocks of our everyday digital lives, and a healthy growth mindset is all that’s required for optimism.
I love legacy systems. I've built a career working on legacy systems, and I have a great deal of respect for them and for the value that they deliver to organizations. But the reality is that they tend towards entropy. Organizations that are working on successful systems, which are also known as legacy systems, tend to this place of keeping the lights on and have to exert active effort toward pushing against that. Engineers have to be thinking, leadership has to be thinking about how to systematically improve the things that might go wrong here.
Looking to the future, Brad believes we’re ready for a paradigm shift in how we think about and work with data.
In five to 10 years, I hope that it's going to be as declasse to not version your data as it is today not to version your code. … My hope is that we have abstractions that make the way that we're talking about this now sound primitive.
Despite all of these reasons for optimism, Brad still cautions patience as we continue stumbling forward on this path.
I think that the time scale over which these new database paradigms succeed and fail is years to decades. I think it took a long time for relational data to become established. We don't see much of the hierarchical object databases, the kind of things that were popular when relational was a thing that was being argued for. And so I think it takes a little while. It takes some retrospect. It takes some time to figure out what has succeeded and failed.
Subscribe to Collaborative Craft
If you'd like to receive new episodes as they're published, please subscribe to Collaborative Craft in Apple Podcasts, Google Podcasts, Spotify, or wherever you get your podcasts. If you enjoyed this episode, please consider leaving a review in Apple Podcasts. It really helps others find the show.