Discover more from On Engineering
How the Agile Manifesto changed nothing.
After over 20 years, in an eerily similar tech market, the ideals of the Agile Manifesto still seem novel and strange. What happened?
When the Agile Manifesto first appeared in 2001, the industry had just finished reeling from the catastrophic collapse of the dot-com bubble. Massive amounts of capital had swarmed into the tech market, every company was “adopting” the internet, and, when the inevitable economic downturn arrived, many software engineers lost their jobs.
Flush with cash due to the low interest rates of 1998 and 1999, the late 90s gave rise to a new kind of startup: the internet startup. Blinded by the possibilities of this entirely new and booming market, many entrepreneurs (often without the ability to actually execute on their ideas) where able to successfully pitch their businesses to venture firms. While there were some massively successful unicorns that came out of this bubble (like Google’s internet search engine, Ebay’s online auction site, and Amazon.com’s internet book store) many others failed, inevitably going out of business and tanking the market with them.
Regardless of this terrible tech-market, the world of software in 2001 was a very messy, cut-throat, and “process-heavy” place. Many engineers were subject to the dreadful Waterfall methodologies of software development: a system of creating software that used ironclad requirements which were very difficult to change. And fixed deadlines on delivery of software products left little room for iteration or integration testing.
In other engineering fields, Waterfall works just fine: if you’re building a bridge, I sure hope you have a list of requirements, measurements, materials, and structural calculations before you start building.
But in software, where things change quickly, integrations break often, and customer needs shift without warning, a shorter and simpler development methodology is often needed. You could be months into developing some piece of software using Waterfall and find out some critical API integration being delivered by another team, in another department is completely incompatible with what you’ve built (requiring more lengthy Waterfall cycles). Or that your project is getting de-prioritized, if not canceled completely, by the business (if only we had gotten user feedback months ago so we knew what the right thing to build was … ).
Companies in the late 90s and early 2000s that attempted to “adopt the internet” found themselves bogged down by the old lengthy engineering processes and dev cycles. This, at least in part, contributed to the economic collapse of the dot-com bubble: huge sectors of the economy that had stagnated and failed to execute on inflated evaluations of their internet business’s value and stability. I wonder what the world would be like if businesses during the early technology revolutions had taken the simpler agile approach: would fewer businesses have failed? Would we have progressed the state of technology even further today?
Thankfully, agile proposes a different approach.
Let’s imagine your team is tasked with building and delivering a software car. In Waterfall development, you might first build the frame, then the wheels, eventually the engine, and so on. Alot can go wrong: what if your customers needs aren’t met by the final product? What if your integrations with the road software suite is broken upon delivery? Or you find out that the engine components never actually worked as intended causing the entire product to break. The plans and requirements for the car are handed down to the dev team at the very beginning of development, often by “the business”, and there is little flexibility for change or ability to get early feedback from real users.
Conversely, using agile to build this software car, you’d always and continuously deliver working and tested software, working closely with the people who know the customer’s needs. You might first deliver a skateboard type thing. Easy enough to build; it has four wheels and works right out of the gate. Then, you may make the frame bigger. You may add an engine for a small go-cart-type vehicle. Add enough seats for a car. Configure the steering wheel. Add a radio. Eventually, the project will be completed and, all along the way, the development team is able to get feedback from real users and integrate with existing systems continuously, testing as they go (the team would catch that failing integration with the road systems very early on!).
Agile software development proposes principles that may seem obvious to engineers today in 2023, but, at the time, these ideas were intriguing, nuanced, and powerful: always deliver working software, test your code, integrate continuously, have face-to-face conversations, prioritize relationships with people on your team, adapt quickly to changing requirements, work closely with the business people, and let teams self organize for the best results.
Like any good “declaration”, all this was wrapped up in a simple and elegant document: The Agile Manifesto. This wasn’t something that was handed down from pundits within the industry; it was written and signed by software engineers doing real work. Its inception actually sounds like something out of a novel: in the winter of 2001, 17 software developers gathered in Snowbird Utah to ski, eat, drink, and discuss the software industry (but, unfortunately, not to go on some epic quest). People like Kent Beck (who would later go on to establish Extreme programming), Andrew Hunt & David Thomas (who co-authored The Pragmatic Programmer), and Jeff Sutherland (a fore-father of the scrum project management methods). They were all present. The output; a single, simple document outlining the engineering processes for an ideal, lean, and efficient software organization.
Agile development would eventually become the bedrock for other systems of software development, like Scrum, DevOps, Extreme-programming, and Platform Engineering. These subsequent systems of development heavily emphasized different prinviples within agile; like continuous delivery, continuous integration automation suites, relationships between individual contributors, and easily deploying and delivering optimized dev environments to individual contributors and dev teams. But throughout all of these methodologies is agile. Agile remains the backbone. Agile is what gave life to these methods.
Despite agile seeming like it’s all about process and how to get stuff done, the Agile Manifesto is, first and foremost, a response to an industry that was falling apart, cannibalizing itself from the inside out, and burning out its talent.
At its heart, The Agile Manifesto is
… a set of values based on trust and respect for each other and promoting organizational models based on people, collaboration, and building the types of organizational communities in which we would want to work.
To buy into the agile engineering processes also means that you buy into the cultural ideals that surround and encompass agile. The engineering processes of test often, ship working code, integrate continuously, etc. are only in service of a model that supports the kinds of engineering organizations where self-organizing teams thrive, people love the work they are doing, and trust in your fellow engineer is paramount.
Today, most software organizations would say they’ve adopted agile (or at least some bastardized form of agile). Yet, I would argue that the industry has missed the mark on adopting the true heart of agile.
We find ourselves in a very similar situation to the 2001 dot-com bubble collapse; increasing interest rates, massive droves of software talent being let go in some of the worst ways possible, priorities shifting away from collaboration and psychologically safe engineering organizations and moving more towards efficiently delivering products.
It seems that the industry is pushing back on the agile ideals.
And my worst fear is that the Agile Manifesto changed nothing with more and more of the sector being somewhere I wouldn’t really want to work.
I’ll be the first to admit: agile takes time, energy, and dedication. It’s not always easy. Retrospectives, planning meetings, user research (and so on) all take time and engineering resources. Time not spent coding or working directly on products. But if there’s anything the last 20 some years of this tech boom market have shown us, where agile was adopted by the industry very broadly, it’s that agile works. Happy engineers who love their work deliver amazing solutions and, in the long term, make for more sustainable organizations that can continue to ship stable, and innovative products that customers love.
If you’ve been reading this so far and find yourself saying “Huh, the dot-com bubble sort of looks like the tech-market today”, that’s because it is. From an economic standpoint, culture perspective, and engineering process view. It seems when the economic goings get rough, engineering organizations get worse.
A primary, high profile example of this is the Elon Musk takeover of Twitter: most engineers have been laid off, there have been multiple lengthy outages, rumors of badly broken internal infrastructure systems, and a new “extremely hardcore” culture, all for the crusade to find profitability and ship software requirements.
But Elon is not solely to blame. The problems at Twitter existed long before the takeover:
Soon after joining Twitter in 2019, Dantley Davis gathered his staff in a conference room at the company’s San Francisco headquarters … He asked employees to go around the room, complimenting and critiquing one another. Tough criticism would help Twitter improve, he said. The barbs soon flew. Several attendees cried during the two-hour meeting, said three people who were there.
That sure doesn’t sound like “the type of organizational community in which we would want to work”.
This is significant because the tech market is a self feeding, always cannibalizing beast: whatever the big players do, typically companies like Google, Meta, and Amazon, the rest of the tech market will follow. These trends in engineering cultures, compensation, interviews, and so on will always trickle down to the rest of the industry. So, without you even knowing it, the Elon takeover of Twitter has probably already affected you.
Layoffs and downsizing in 2023 may not yet be over. And many economists believe we are heading into a recession (if not already there) which could accelerate cultural and engineering organization changes at many companies. And like before, in 2001, when the dot-com bubble had its reckoning, the software and tech industry of today must face the economic music.
But my hope for the future is that engineering organizations and leadership recognize the history that is being repeated here, change course, and continue to focus on lean, agile processes that work for them. Otherwise, we may see more companies like Twitter with a failing business model, collapsing infrastructure, deplorable stability, and maybe worst of all, an engineering organization that no one wants to be apart of.
Thanks for reading On Engineering! Subscribe for free to receive new posts and support my work.