5 Requirements in the Software Lifecycle
This chapter looks at the roles of requirements in the software life-cycle. We will talk about requirements in waterfall and agile projects, and about the relations between requirements, architecture, and quality assurance.
5.1 Requirements in the Waterfall Model
The waterfall model presents software development as a linear sequences of activities that starts with defining the requirements, followed by designing the architecture, developing the code, testing the code, and operating the system. This model is illustrated in Figure 5.1.
The waterfall model was first described in a 1970 paper by Winston W. Royce who wanted to warn people against using that model. Just below a figure similar to Figure 5.1, he writes:
“I believe in this concept, but the implementation described above is risky and invites failure”.
The paper continues by discussing the pitfalls of a linear approach and argues for a more iterative approach to software development. Despite these warnings, the waterfall model became the dominant model for the procurement and management of software project for many years. Ironically, Winston W. Royce became known as the ‘inventor’ of the waterfall model.
For requirements engineering, the important point in the waterfall model is that it requires all requirements to be defined at the start of the project. Working as a requirements engineer is a waterfall project is not an envious situation. You have to write a detailed requirements specification document, and the requirements in that document need to be complete, consistent and unambiguous because any omissions and errors will be very hard and expensive to correct later. All changes to your initial requirements specification will have to go through a formal change request process where you have to write a change request explaining what requirements you would like to add or change, why you want to change them, and how it would affect the rest of the system. If the request is approved, applying the changes may involve a lot of rework on the software architecture, code, test cases, and other documents such as user manuals and product documentation. There is thus high pressure to get it right the first time, but this is nearly impossible.
Assumptions and Limitations
The waterfall model relies on two assumptions:
it is possible to fully understand the stakeholder goals, domain assumptions and machine requirements at the start of the project, and
these goals, assumptions and requirements will not change during development.
For many projects, these assumptions will not be true. It is rarely possible to fully understand the stakeholder goals, domain assumption and machine requirements before the machine is built and used. Furthermore, stakeholder goals and the context are almost certain to change during the design and implementation of the machine.
The waterfall model has further important drawbacks for stakeholders and the development team:
The clients and other stakeholders must wait a long time before the software is delivered. The project must go through the full sequence of requirements, architecture, development, and testing before the software is finally operational and of benefits to its stakeholders. During this time, the development project incurs cost but does not bring any value to its stakeholders.
The goals, requirements and assumptions cannot be validated until the whole system is operational. If the requirements engineers made a mistake in understanding the goals or made incorrect assumptions about the domain, it will take a long time before these mistakes can be detected. Furthermore, since a lot of code and tests may already have been written, fixing these mistakes may generate a lot of rework.
Changes are difficult and expensive because they need to be propagated through all these phases. Any changes to the requirements involves identifying all places in the architecture, code and tests that are impacted by the changes.
5.2 Requirements in Agile Development
An alternative to the waterfall model is agile software development. In software engineering, the term agile has become an umbrella term covering a wide range of software development processes and practices. In a broad sense, agile simply means iterative and incremental:
- Incremental means that we are not trying to develop the full system at once. We first develop a very small set features, then we gradually add new features.
- Iterative means that when we develop a feature, we don’t try to make it perfect at once. We first develop a very simple feature, then we gradually improve it.
Iterative and incremental methods have been around since the 1960s, much before they were called agile, but they were not mainstream practices until the publication of the Agile Manifesto in 2001 that started a gradual shift from waterfall to agile processes across the IT industry.
Figure 5.2 illustrates the agile development process.
In an agile process, you build and deliver a product over several iterations. The product may be a piece of software or a larger system; it corresponds to what we called the Machine in Chapter 2. In the first iteration, you perform requirements and architecture activities to understand your stakeholder goals and create an initial vision and development plan for the product. Depending on the kind of product, this first iteration may take a few days to a few months. In the next iteration, you build and deliver a first usable version of the product. This iteration may again take a few weeks to a few months but is much shorter than the first iteration in a waterfall project because you are not trying to deliver the full system at once. In the following iterations, you gradually expand and improve the product. Each iteration involves some requirements engineering activities to decide what new features to build and how to improve existing features. A benefit of delivering the software in multiple iterations is that you can now use stakeholders’ feedback about earlier versions to refine your understanding of stakeholder goals, domain assumptions and software requirements.
Benefits and Limitations
The main benefits of agile software development comes from using short iterations. Using short iterations means that:
You deliver value earlier to the business and your stakeholders. Instead of having to wait a long time before any software is delivered, stakeholders benefit from an initial version of the software much earlier.
You can validate your requirements and assumptions much earlier than if you deliver your system after a long waterfall development cycle.
Because you don’t need to define all requirements upfront, you can more easily adapt to changes in stakeholders’ needs and context.
On the downside, agile methods rely on assumptions that are not always easy to satisfy.
A first assumption is that an iterative and incremental approach is possible and acceptable to stakeholders. In some contexts, delivering an initial product that may contain defects may not be acceptable. Safety-critical or business-critical defects would not be acceptable. Similarly, delivering an initial product with only a restricted set of features may not be acceptable. In agile projects, product owners have to find a way to split a large product into small increments that are acceptable and valuable to the stakeholders; this is not easy and not always possible.
A second assumption is that the development team will have good collaborations with the client and other stakeholders. That is the ideal situation but sometimes key stakeholders are not sufficiently engaged in the project, are unable to provide the right information when needed, or don’t have enough time to devote to the project. In such situation, it is not uncommon for relations with clients to deteriorate and lead to arguments about contracts, costs, schedule, and the quality of the work. These arguments can lead to expensive court cases.
A third important assumption is that the software development team is able to constantly change the product at a sustained pace, over multiple iterations. For this assumption to be true, the development team must be skilled in software engineering practices that enable frequent changes to the code. Notably, they must have strong practices for continuous integration, automated testing, modular architecture and refactoring. Attempting agile development with a development team that lacks these skills will quickly lead to an impasse.
The Evolution of Agile Requirements
The first generation of agile methods tended to ignore requirements engineering. They left responsibility for requirements engineering to the client or a client representative. Modern agile practices, however, have brought requirements engineering back into responsibilities and skillset of software engineers. Modern practices, represented by techniques such as user story mapping and impact mapping, emphasise that the responsibility of software engineers is not merely to deliver software; our responsibility is to deliver value to the product’s stakeholders. Helping clients and other stakeholders to understand their goals, formulate their requirements, and decide what software to build is part of our responsibilities. Remember the analogy with restaurant waiters and medical doctors in Chapter 1.
A Different Focus
Requirements engineering has a different focus in agile and waterfall projects. In a waterfall project, the focus is on writing high-quality requirements documents and managing formal change requests. In agile development, the focus is on prioritizing what to build next and on just-in-time understanding, analysis and communication of testable requirements.
5.3 Requirements and Architecture: the Twin-Peaks Model
Let’s now look at the relation between requirements and architecture. In the waterfall model, the relation is sequential: first, we define the requirements, then we define an architecture that will support the requirements. In practice, the process is more iterative and incremental than sequential.
This iterative and incremental process, known as the Twin Peaks model, is shown in Figure 5.3.
The left peak represents the requirements description, the right peak the architecture description. The two descriptions are separate because requirements descriptions are descriptions of the World and architecture descriptions are descriptions of the Machine. They have a common intersection which is a description of the interface between the World and the Machine. The arrows represent the specification process. This process starts with a brief partial description of stakeholder goals, domain assumptions and machine requirements. The initial requirements description informs the identification of one or more candidate architectures that may satisfy the goals and requirements. Exploring candidate architectures reveals costs, tradeoffs, omissions and missing opportunities in the requirements. This leads to revising and adding more details to the requirements description, which in turns leads to adding more details to the architecture description. The process of adding details to the requirements and architecture descriptions continues until we are confident enough that the envisioned architecture is adequate to address the stakeholder needs in the given context.
5.4 Requirements and Quality Assurance: the V-Model
Requirements have strong relations with quality assurance, testing and verification. The V-model in Figure 5.4 shows the relations between the different levels of requirements and tests.
The important point in this model is that requirements and tests are strongly linked. Acceptance tests evaluate whether the Machine satisfies the stakeholders’ goals in the application domain. System tests evaluate whether the Machine satisfies its requirements. Integration tests evaluate whether components in the software architecture satisfy their requirements. Unit tests evaluates whether smaller unit of development, for example classes and functions, satisfy their requirements.
Requirements are also essential for any verification activity. Verification is always with respect to some higher-level requirements. We verify code with respect to unit requirements, unit requirements with respect to components requirements, components requirements with respect to Machine requirements, and Machine requirements with respect to stakeholder goals.
When we talk about software quality, all levels of testing and verification are important, but ultimately the quality of a system is determined by the extent to which it meets its stakeholder goals. A software that has bugs but satisfies its stakeholders goals is better than a system that has no bug but does not satisfy its stakeholder goals. In other words, stakeholder goals are the ultimate criteria for evaluating the quality of software systems.
5.5 Notes and Further Readings
Winston W. Royce’s 1970 paper was the first to contain a diagram of the waterfall model (Royce 1970) with a clear description of its principles and limitations.
The discussion on requirements in agile in Section 5.2 borrows ideas from Henrik Kniberg’s excellent talk “What is Agile?” (Kniberg 2013). Craig Larman and Victor Basilli have written a brief history of iterative and incremental development processes before they were called agile (Larman and Basili 2003). Agile requirements engineering techniques such as impact mapping (Adzic 2012) and user story mapping (Patton 2014) will be covered Chapter 11.
Bashar Nuseibeh introduced the Twin-Peaks model on the relation between requirements and architecture (Nuseibeh 2001) adapting ideas from an earlier paper on the relation between specification and implementation (Swartout and Balzer 1982). The process of creating and evolving a software architecture is described in more details in excellent books on the topic (Rozanski and Woods 2012; Bass, Clements, and Kazman 2012; Richards and Ford 2020).
Paul Rook introduced the V-model of software development (Rook 1986). The V-model figure in Section 5.4 is adapted from Jeremy Dick, Elizabeth Hull and Ken Jackson’s book on systems requirements engineering (Dick, Hull, and Jackson 2017). How to generate tests cases from requirements is described in books on software testing (Pezzè and Young 2008; Aniche 2022).