Agile

When I google "why agile is...", I get the following suggestions:

Why Agile is... bad/failing

That does not exactly make Agile look good, does it? Why does Agile have such a bad reputation among developers when The Agile Manifesto seems to support developer's favorite way of working?

Why companies shift to Agile

Brief reminder: companies exist to make money. As much as it seems like C-level executives always get their bonus + a golden parachute when they jump ship, some of their compensation is actually tied to the company's performance. Moreover, after leaving a company, they are more likely to get hired at a higher salary if they did well in their previous jobs, just like anyone else.

In a company that is shifting to Agile, the mantra of "Waterfall bad, Agile good" will be repeated often enough to make anyone sick. Since everyone seems to be talking about Waterfall without a clear definition of what Waterfall is: a method of developing a product where the outcome is fixed, changes in requirements are extremely unlikely and development progresses through phases which completely define all necessary prerequisites to start with the next phase, without revisiting decisions made in previous phases.

That sounds bad when working on an iteratively updated product, e.g., a web platform or an app. When making toasters, the Waterfall method sounds very reasonable. Yet Waterfall is replaced by Agile even in companies that make toasters, which points towards the fact that

sometimes management is chasing the latest trend and
bad implementations of Waterfall

and thus, these companies will not fare better with Agile, either. For the 90% of companies that fail to meet timelines and budget predictions for the simplest projects, the choice of process framework will not make any difference. Unfortunately, most people work in these companies and therefore, any process deployed there will get a bad reputation over time.

Nonetheless, for the companies that may sensibly shift to Agile (i.e., not making toasters), let's establish why they are doing it:

Agile is deployed to make more money with less resources.

Specifically, Agile is not introduced

to make developers' lives easier,
to make developers' lives more fun,
to make code more elegant/beautiful,
to have more Bullshit Job positions that can be filled with friends and family.

If you thought differently, you either forgot why the company is able to pay your salary or you work in a project that is not driven by the need to make money (and you either work for free or make way less money than you could). With that being said, it is quite obvious that companies that introduce Agile do not give a flying squirrel's pinky toenail about The Agile Manifesto. Their goal is to make money, and if The Agile Manifest can be (ab)used to that end, they will.

What Agile does for the company

Now that we have established why Agile is deployed - namely, for the company to make more money, not for you, the developer - we may look at what companies hope to get from Agile.

Developers want to write code, and keep writing code until it is, in their eyes, perfect. There are two issues with this:

the average developer is unable to tell you reliably when they will be done and
the average developer is unable to realize that their "perfect" solution requires far more resources than is required for a product that will sell.

Many projects are not delivered on time and within budget and few developers decide to not refactor the code for the nth time to get rid of "technical debt" because the current implementation is sufficient for the customer. It does not make sense to dream of a reality where companies deploy Agile for altruistic purposes. Repeat: Agile is deployed to make more money with less resources.

What do companies get from Agile, in that case? Developers hate "daily standups", "sprint reviews", "story points", etc. and they hate "Agile" for it. Yet the company gets

predictable project timelines, e.g., via Story Points estimations,
features whose technical details are completely understood, e.g., via Refinements or PI Planning.

Again, the average developer is unable to predict a completion date with any sufficient precision. A manager having to explain to higher-ups again that their nth project is not ready well past the deadline will look for solutions and if that solution is "Agile" in the form of SCRUM, which developers hate, they will enforce it.

A develper will not care about burndown charts and will not understand why they should invest time in "estimating Story Points" or "reviewing the Sprint" that could have been spent frantically writing code. For management, however, seeing that there are roughly 3000 Story Points remaining for a goal that should be hit within three months to yield a profitable product while the team is delivering at a rate of 30 Story Points per Sprint is a valuable tool to, e.g., terminate a project early and save the company money and resources that could be deployed elsewhere. Getting rid of all the SCRUM ceremonies would perhaps speed up development, but probably not by the 1600% that would be required to get it done in time.

Let's sum it up:

Developers get paid.
The company expects them to deliver
- solutions and
- a trustworthy estimate of when that solution will be available.
With that, the company makes decisions that ultimately
- maximize profit and
- ensures it is able to pay future salaries.

Since it is difficult to make predictions, especially about the future, and especially without any prior, the company will prescribe their favorite method of getting the data. The method currently in fashion is Agile, specifically SCRUM (and in certain cases, Kanban or DevOps).

That is what Agile is all about in companies. To make money. Nobody cares how beautiful the code is (have you seen drivers for networking hardware of certain companies that are in the Linux Kernel and power tera- or petabit of global data transfers every second?), or whether a developer thinks that the code needs to be refactored to get rid of technical debt, for the nth time, mostly without any business value behind it.

Developers and management often do not know about each others' goals. A product manager usually has a much better view into which features should be prioritized given customer needs and market value, while developers have a good view of what is easy to implement and how "ugly" architecture may prevent certain solutions from being implemented.

This is where The Agile Manifesto proposes a solution: let the developers talk to the customers. This works in certain, small-scale settings but in any larger company, this will not work. Imagine, e.g., a company where the product is created by many teams that do not share skills, e.g., companies making mobile phones. There will be many teams with different responsibilities, e.g., design, digital hardware, RF hardware, kernel and OS, application software, marketing. It does not make sense to have giant meetings where people from 20 different teams bombard potential customers with questions. Most employees will never talk to customers in such companies. It takes a good process to align all teams behind common goals such that each teams' objectives, both near-term and long-term are aligned and yield a timely delivery. Needless to say, everyone will have to participate in "boring" tasks that are required to maintain project coordination and on-time delivery, whether they like it or not - it is not a hobby project, they are employees and get paid for it.

Basically, sometimes work sucks: that's the reason you get paid for it.

Agile process frameworks

As pointed out above, some development efforts require a great deal of coordination between multiple teams. The unorganized, "classical" way to solve that, often unfairly labeled Waterfall, is by having lots of meetings. Meetings might work when there are two or three teams working on something, but often dependencies are everywhere and the number of communication links grows not linearly with the number of teams, but with N^2. This is why many companies fail to deliver the first large product even if they were doing perfectly fine delivering many smaller products in parallel before.

Unfortunately, going from zero process, "it just works [because we have 5 people working on it and they sit next to each other in the same office]", to "full-fledged multi-team product development with predictable, limited timeline and effort" does not work in zero time. This is where companies hurriedly introduce "Agile", mostly without having anyone onboard knowing anything about it, checkmark the "we need a process" item on their list and and wonder why it does not work.

Remember how Agile is deployed to make more money with less resources? Well, now it is time to get management expectations about Agile back to reality:

Agile methods are solutions for certain sets of requirements.

When companies that used to do fine with smaller projects run into trouble with their first big, multi-team project and panically deploy SCRUM or Kanban or DevOps, the management doing so most often did not research the requirements their project has with respect to a process. Thus, whatever Agile process is deployed often does not cover all requirements and thus fails.

Here is another hint for managers reading this:

None of SCRUM, Kanban or DevOps are sufficient for multi-team projects.

All of them define how a single team shall perform work to deliver on business goals effectively and in a predictable fashion. None of them define how multiple teams with possibly different intra-team processes shall stay aligned towards common business objectives, cooperate effectively and predictably.

The mistake that happens in many companies at this point is that they start to invent processes. This "innovation" is mostly done by high-level management who will inadvertently design processes that are optimal for them, but do not take into account other role's needs, such as how much effort a process causes elsewhere, how many meetings result from the proposed process, how much manual, boring, repetitive work is expected from certain roles.

The good news is that there are process frameworks out there that will fulfil the needs of any multi-team project. There is no need to re-invent the wheel (and please do not pick up fancy words from a bunch of existing process frameworks and then use them for something else in Your Custom Process™ - terms have established meanings, and assigning totally different ones just makes your company look like a fool to any potential candidate who has received training in any widely established framework). Yes, all of these frameworks will try to sell their consultancy services but that does not make them less well-researched or less widely deployed than they are. When looking for the ideal framework to solve your company's needs,

collect requirements,
stay problem-oriented,
avoid solutioning early,
check which types of companies successfully employ the framework,
contact them if possible.

It is the same as looking for a software library for a typical problem, such as UI-design or linear algebra. No company in their right mind would write these from scratch unless that is their business. Why re-invent the wheel? No company is that special (I know you think yours is, but trust me, it is not) and working processes for large scale software/hardware/systems development have long been established.

When the decision for a particular framework has been made,

support for that decision from highest level of management is absolutely required and
external experts or new hires with extensive experience are required to drive implementation throughout the whole company, on every level.

Otherwise, the process introduction will fail and people will get the mindset of "oh yet another new process that will solve nothing". Having working Agile teams (SCRUM, Kanban, DevOps...) is a prerequisite for most Agile process frameworks, so getting started with them early is a good idea.

It may happen that the most suitable framework for a particular project or company is Waterfall, or something that feels - to parts of the organization - like Waterfall, or an "abomination of Agile". It does not matter. Remember, Agile is deployed to make more money with less resources. It is the job of Agile Coaches to help people saying things like "But The Agile Manifesto says People over Processes!!!" by making them aware that coordinating 20 teams is not going to work on a "people" level. It is more efficient to track progress in a common tool (such as the much-hated Jira) so that anyone can at any time track progress towards the multiple, tightly dependent goals that exist in such huge projects. This allows for fast and effective planning without the endless meetings that everyone hates so much. If I had a dollar for every minute I spent in meetings that just existed to update everyone on the state of progress of everyone else - something that is easily solved by consequent tracking of tasks or Stories in a tool such as Jira - I would be retired on a yacht.

Needless to say coordinating 20 teams - even with the best tooling and training in the world - will not be as agile as five people working on a small piece of a pure software solution. When customers voice concerns or wishes, it will take quite a while to get these into the product. The reason for that is that even tiny changes often cause lots of work across multiple teams. A good example for this is the trend towards larger, more efficient engines for aircraft. These larger engines take more space and thus are closer to the ground. At some point, they become too large to be installed in place of the older, smaller engines and some other solutions are requried, such as increasing the height of the landing gear, which will completely change the layout of large parts of the fuselage. Or they are mounted a bit upwards and to the front, which comes with its own set of issues. Such a "tiny" change of increasing the size and efficiency of an engine by a few percent took years to implement correctly.

Large projects handle like oil tankers.

It is not possible to change direction every day. The difference between Waterfall and Agile is that with Waterfall, the route and target port is fixed from the start while with Agile, they may be altered during the voyage, but only such that the general direction and momentum are maintained.

Tooling and People

Let me re-iterate the need for tooling. The Agile Manifesto does not forbid the use of tools. People are quick to cite it once they find that they have to do work they do not enjoy. I repeat: basically, sometimes work sucks and that's the reason you get paid for it. There are dependencies that might not be obvious which require the use of certain tools and work that might not make sense or might not be enjoyable. It is up to the Agile Coach to clarify why things are needed and why certain tools and processes exist: people are more cooperative when they understand why things need to happen.

Apart from whatever tooling is required to implement the Agile process (such as Jira, Confluence, Jira Align, GitHub Projects...), at least one other tool is required in any project: a requirement tracing system.

I always hated requirement tracing.

With that out of the way, I also see no alternative to it. In large projects with inter-dependent teams, an automated, integrated requirement tracing system is the only way to avoid tons of unproductive meetings:

to reliably track what has been implemented,
what has been tested,
what has been integrated,
what is deployed in which version,
to estimate effort of change requests,
to track which parts of the code are required for what,
to plan large refactoring required for upcoming features,
to create meaningful documentation,
and more.

Unfortunately, there does not appear to be any good requirement tracing solution that fulfils the requirements of large-scale projects with diverse teams.

Without requirement tracing systems, projects find themselves discovering requirements at runtime, which makes it impossible to predict delivery, avoid conflicting development in different teams, etc.

Given a requirement tracing system, some pre-work has to be done on architecture. Just because something is an Agile process framework does not mean everything that Waterfall did gets to be thrown out of the window. Thus, an architectural roadmap that has sufficiently high definition in the short term and sufficient flexibility in the long term is necessary. In small teams, architecture can evolve organically. In large projects with tens of teams, explicit top-level architecture must be designed and nailed down in requirements to create alignment. This architecture will be high level and not define everything down to the last API call (that would be Waterfall). It will also almost never be optimal in some sense of the word.

However, when carefully done, it will be sufficient and good enough for a while. Programmers hate both sufficient and good enough, but if people knew what kind of code powered the games and phones and movie streaming that they enjoy so much, they might understand that for better or worse, good enough is what powers the world around us. And not just that: good enough products actually being sold are what is used to pay your salary, not the perfect ones that never made it to release.

One final thing is required for everything to work, even with the best tooling and the best Agile process framework: people. Excellent people.

Architecture does not come from mediocre programmers. Architecture is done by architects. Architects are people who have mastered multiple programming languages, understand business needs and have a good feeling for effort that implementation of a particular architecture might require.

Requirements do not grow on trees. They come from requirements engineers and top level system architects, from product managers and technical experts. These people need to exist and have sufficient expertise in each others' fields to be able to design a consistent outline of top-level requirements that will enable implementation of a suitable product.

Maintainable, testable code also does not spawn for free. If your company is "writing C++ code" that is full of raw pointers and where every class method is always marked 'virtual', the best process framework and the best architecture will not save you. Your teams consist of programmers. What is required for large-scale projects are software engineers. They are not the same.

For each role, the rule is: garbage in - garbage out. Generally, you get what you pay for them. There might be a reson that the highest-valued companies in the world employ software engineers that earn more than upper level management in your company. Not every single person needs to be a top level expert (or there would be no place for the average person), but the fewer there are, the more unforeseen expenses will pop up in the form of delays, bugs and budget overruns.

What has Agile ever done for us

In a company where an Agile process framework has been successfully deployed, there are benefits even for regular developers. A process framework needs committment from everyone in the company. Its rules must be strictly followed - there are no exceptions for upper management. Imagine if the rules of the road were optional for luxury vehicles: it would not work. If that is the case in your company I am sorry to say, what you are doing is not Agile or any other framework, it is a giant pile-up on the motorway. Do not hate Agile for your management's incompetence. If the rules are being followed by everyone, there are benefits over other forms of working, however.

The much hated SCRUM Sprint then means that management cannot change their mind about the highest-priority item every three days. The SCRUM Master will ensure that teams are able to focus on committed Sprint content and that any external pressure and escalations will be diverted into future planning sessions.

The prescribed tools and the rules around the use of these tools will prevent Product Managers from bringing vague ideas with poorly defined business value and priority to the table. They have to provide figures to highlight the importance of customer requests and make it possible for management to decide, e.g., based on WSJF or similiar methodologies, which features are going to be implemented next. Product management committs to the business value, the teams commit to the effort (Story Points) and management commits to the decision of doing it or not doing it. Such committments create clear and documented accountability, where everyone is responsible only for what they can promise. This propagates fair treatment and evidence-based career development.

Employees are able to point towards agreed-upon metrics when the yearly review is due. Since due dates and delivery content is documented as something either committed to by the team or gambled on by management without team support, there are no finger-pointing exercises and at the end of the year. Bad management - that gambled and lost - will be replaced while management whose teams had most success will be promoted.

The Daily Standup self-establishes in SCRUM teams because the team realizes that syncing once for ten minutes is more efficient than tons of tiny meetings throughout the day. They focus on exchanging information required to get things done that they committed to, achieving their goals and earning their favorable yearly review. They are not discussing for hours how to do something (the Architecture, which was clear before work on a feature was started) or even worse, backtracking on existing work because it turned out it was not the right path to a solution.

All the Story Point estimates and refinement sessions have given management enough insight into possible delivery dates and enabled them to make decisions that result in optimal delivery of products and features at the right time for the best price. Regular employees tend to overlook this as their impression mostly is that this just results in bigger bonuses for the C-suites, but it also means that the company is probably going to be around for a bit longer and able to pay salaries. And that is quite desirable for the regular employee.

Takeaways

Agile is deployed to make more money with less resources.

Companies introduce agile to have better plannability, not to make developer's lives more fun.

Developers are notoriously bad at giving reliable estimates regarding a timeline.

Large projects handle like oil tankers. As part of the company, you are part of the ship. Agility means something else in these large projects.

The company will require Agile Ceremonies as part of work to make efforts more visible and plannable.

Some of these ceremonies may be boring or exhausting.

Sometimes work sucks: that's the reason you get paid for it.

This blergh post represents my personal views and is based on experience from multiple roles in many projects. It is meant as a counterpoint to the many articles and videos depicting Agile in a negative way and instead tries to provide a realistic look at the motives behind introduction of Agile methods.