Workshop Summary REFSQ ’98*
by Andreas L. Opdahl & Klaus Pohl
The proceedings of the workshop can be ordered via email: pohl@informatik.rwt*h-aachen.de
The Fourth International Workshop on Requirements Engineering: Foundation for Software Quality (REFSQ’98) was held in conjunction with CAiSE*98 in Pisa, Italy on June 8th and 9th 1998. The workshop was organised by Eric Dubois, Andreas L. Opdahl and Klaus Pohl.
In the following we give an overview of the presentations and of the numerous fruitful discussions that took place during the workshop.
Introduction and Workshop Structure
Achieving high quality means to realise the needs of users and customers. These needs are elicited, negotiated, documented and validated during requirements engineering (RE). RE is therefore the most crucial phase of software development. The goal of RE is not just to produce a specification that is as consistent and complete as possible: RE is also a cooperative and iterative learning process. According to Pohl [2] the goal of RE is to produce a traceable requirements specification that is as complete as possible and that is documented using various appropriate representation formats, and to establish a sufficient common understanding among the stakeholders involved. Conventional RE methods normally support only parts of this process or help stating only specific kinds of requirements.
The goal of the REFSQ workshop series, which has been held in conjunction with the Conference on Advanced information Systems Engineering (CAiSE) since 1994, is to improve the understanding of the relations between requirements engineering and software quality. More precisely, REFSQ aims at having intensive discussion provoked by brief presentations about:
In response to the Call for Papers for REFSQ ’98 which addressed the above problems we received 31 submissions. After the reviewing process we accepted 12 full papers of high-quality, five position/preliminary result papers, and four industrial problem statements. Consequently, the workshop comprised 21 presentations with around two-thirds of the workshop time set aside for discussion of individual papers as well as general issues.
With respect to the described research problems and the contents of the accepted papers, we divided the workshop into four full-paper sessions: Goals in RE (chair: Andreas Opdahl); the RE process (chair: Klaus Pohl); specification and validation (chair: Philip Morris); scenarios in RE (chair: Eric Yu). In addition, there were two sessions of position statements and preliminary results chaired by Neil Maiden, as well as a session of industrial problem statements chaired by Klaus Pohl.
To ensure the effectiveness of the workshop a full paper presentation was restricted to 15 minutes and followed by a 20 minutes presentation-related discussion. Each discussion was initiated by the two other speakers of the session acting as discussants. At the end of each session, the major topics raised by the talks or the talk related discussions were elaborated. The results of the discussions are summarised in the session summaries (see below). Presentations of position papers and preliminary results were restricted to 10 minutes with 10 minutes set aside for discussion.
The presenters had to make the context of each talk explicit by drawing one or more labelled arrows between a given set of concepts (cf. Figure 1). This set of concepts comprised people (or stakeholders) which participate in the software development process and products (or documents) established by the process. The arrows indicated the relations between the concepts addressed by the paper. While arrows between people indicate some kind of social interaction, an arrow between documents describes technical and logical relationships. Arrows between people and documents usually indicate production/usage relationships.
Each talk was summarised by the presenter and the discussants by answering the following questions:
The workshop was closed by a general discussion which aimed at integrating the session summaries and identifying important research topics. This led to a final workshop summary and a research agenda of topics worth elaborating further to make RE processes more efficient and effective, and thereby improve software quality in the future.
Figure 1: Framework for making the context of each talk explicit.
Session Summaries
Session Summary: Goals in RE
Overview of Talks
This session continued along the lines of a similar session on Goals and Agents from REFSQ’97 [1]. The presentations in this session proposed to use goals as a high-level abstraction medium for structuring and abstracting the content of scenarios and requirements.
In the first presentation, Eric Yu (co-author: J. Mylopoulos) emphasised the increasing use of goals in RE. Eric characterised existing goal-based RE approaches. He argued that the increasing consideration of goals during RE requires a better integration of goals with the various others products produced, e.g., specifications, viewpoints, argumentation, rationales, as well as with the RE methods and processes applied (see discussions for details). Eric suggested three major potential integration paths, namely ontological analysis, synthesis and/or the integration with existing RE frameworks. The problem with the proposal was the missing clear differentiation between goals and requirements. The question if industry should apply goal modelling techniques in addition to requirements modelling was controversially discussed.
The presentation by Peter Haumer (co-authors: K. Pohl and K. Weidenhaupt) focused on the improvement of the quality of current state (goal) models. Peter argued that important facets of the existing system should be captured using rich media, e.g. video. In addition, parts of the captured observations should be related to the goals of current state model which have been defined/modified based on the captured observation. Thereby, so called extended traceability is established. Among others, extended traceability improves the comparison of different observations, reviewing of conceptual models, explanation of the models to, e.g., new project members, and the discussion of the models with stakeholders not familiar with the modelling technique. In the paper related discussion the validation of the approach in real industrial projects was suggested. The integration of patterns, e.g. goal patterns, could support the elicitation of goals from the captured observation as well as the validation of goals against the observations.
The last presentation in this session by Camille Ben Achour (co-authors: C. Roland and C. Souveyet) proposed the organisation of large sets of (textual) scenarios using goal hierarchies. By assigning goals to scenarios and organising the goals using three types of relations (refine, and, or) a structure for managing scenarios is established. The refinements levels are thereby restricted to three possible layers which define system internal scenarios, system interaction scenarios, and contextual scenarios (from specific to more generic). The management and evolution of the resulting goal structure is supported by heuristics which, e.g., support the change or the introduction of a new goal in the hierarchy. In the discussion the comparison of the proposed goal structure to and-or refinement trees and other goal tree proposal was requested. As a major criticism, the organisation of goals in tree-structure was questioned. Experience seems to indicated that the relation between goals span more a net than a hierarchical (and-or) tree.
Discussion Summary
There was a general agreement that goals should be considered during RE. Among others, they could and should serve as basis for defining more detailed requirements, i.e. a goal should provide the rationale for introducing a (set of) requirements. As requirements, goals only exist in a given context.
The introduction of goals in the RE process is especially helpful if the project at hand is somewhat fussy, the stakeholders involved are diverse, and/or if a complex interacting system is being developed. To empower the definition and use of goal models in RE, a much better integration of goals with existing RE models and RE modelling techniques/methods is required.
Existing goal modelling approaches are quite diverse. For example, the term goal is used to refer to different things, e.g., high-level business goal, performance goals, stakeholder goals, individual goals. Towards a common understanding a framework is needed which proposes a classification of the various kinds of goals. On the other hand, such a framework should define the relations between the various goal types and other RE artefacts, e.g. requirements and scenarios.
There was thus a general agreement that a reconciliation of existing goal oriented RE approaches is required. Among the participants there was no preference if such a reconciliation should be driven by the integration of goals with existing RE frameworks, by ontological analysis, by synthesis or by a combination of the three possibilities.
Session Summary:
Position Papers and Preliminary Results 1
Overview of Talks
The presentation by Āngels Hernández (co-author: N. Castell) dealt with the derivation of requirements specifications from natural-language texts as realised in the SAREL system and with using SAREL to compare specifications. The discussion pointed to the problem of explicitly defending the cost of doing natural language analyses and to the need for explicit semantic and/or conceptual models underlying such an approach.
In his talk, Vincenzo Gervasi (co-authors: F. Fabbrini, M. Fusani, V. Gervasi, S. Gnesi and S. Ruggieri) presented a linguistic quality framework for natural-language requirements based on the framework of Krogstie, Lindland, Sindre and Sølvberg. Their framework comprised four quality types and 7 quality factors with associated quality criteria and supporting activities. The discussion raised the question of how much the application of such a framework would depend on domain knowledge and whether the framework would be useful for specifications that are not in natural language, but for instance in the form of goals or scenarios.
Bernd Deifel then discussed RE issues in the development of complex COTS using Pohl’s classification of four main RE activities: elicitation, negotiation, specification and validation/verification. The main discussion topic was the extent to which - and in what ways - RE for COTS differed from RE for a single customer. Bernd also pointed out that a COTS product is not only a piece of software, but that it also has organisational implications in that it needs to be tailored to and implemented in the user organisation and that this should have consequences for how their requirements are established.
Session Summary: The RE Process
Overview of Talks
The session comprised three presentations which all reflected the increasing concern for process in addition to product issues within both RE and the broader systems engineering and software development communities.
In the first talk, Peter Sawyer (co-authors: I. Sommerville and S. Viller) presented a "good practice guide" for RE developed in close co-operation with industry. A main driving force behind their work was the problem of disseminating RE research results to industry. In response, they have devised and described a set of 66 good practices and organised them into a 3-level maturity model for RE inspired by the capability maturity model for managing software development processes. Another discussion topic was whether research results can be disseminated in general form or if they must first be adapted to each specific application domain. The participants also questioned to what extent "good practices" can be identified and described across application domains and even between individual organisations.
In his talk, Wing Lam (co-authors: V. Shankararaman and S. Jones) also focused on "good practices", but with emphasis on management of changing requirements based on work within the DESIRE project. They identified six practices and associated improvement actions for each case. Furthermore, the practices were organised into an overall change improvement framework. The discussion pointed out that requirements changes during RE are different from later changes, for instance, with respect to contracts. It was stressed that over-constraining requirements changes might be just as dangerous as being too permissive. It was also indicated that a classification of various types of changes might be needed.
In the final talk of the session, Timo Käkölä (co-author: A.A. Salo) discussed requirements for groupware-support for RE processes which aim at developing new products. A list of main issues were identified and discussed based on experiences from large-scale development projects within Nokia. The ensuing discussion focused on the particularities of RE as an application area for CSCW and on which types of groupware tools that were most applicable. Asked to prioritise his list of issues, Käkölä responded that bridging the communication gap between people from marketing, R&D and production was particularly important in the projects studied. Also, the participants compared the Lotus-based approach taken at Nokia — based on replicated repositories — with less tightly integrated alternatives, e.g., based on viewpoints.
Discussion Summary
The session discussion focused on two topics:
Several ways of identifying good practice was suggested by the participants, including:
In the discussion of characterisation, participants suggested that the process should:
The session discussion then concluded with a discussion of whether compliance-checking is feasible for RE processes.
Session Summary:
Position Papers and Preliminary Results 2
Lee Young (co-author: N. Mehandjiev) presented requirements engineering problems when developing management information systems, pointing to a knowledge, responsibility and communications gap between three groups of actors: strategic managers, tactical managers and systems analysts. The solution outlined was an exploration process for requirements decisions which centered around a multi-perspective model of strategic requirements. Lee also presented early experiences from an industrial case study.
In the second talk of the session, Joao Carvalho (co-author: I. Santos) presented a conceptual framework for studying how people understand the organisations they work in, the work they do, as well as their expectations for new software systems, challenging the common RE focus on technical issues. Joao pointed out that RE work has both structural, social, political and symbolic dimensions, and outlined how the framework could inspire an RE method. A research design for validating the framework was also outlined.
Session Summary: Specification and Validation
Overview of Talks
The session started with the presentation by Pär J. Ågerfalk (co-author: Göran Goldkuhl). Pär outlined a language action approach for RE. He argued that it is vital to take the communicative aspects of information systems into account when talking about and defining the requirements of the systems. The main focus of the talk was on the perspectives under which the RE process is being performed. As main contribution he proposes to view the operationalisation of information systems as actions. In the discussion a more detailed comparison of the proposed approach with existing action oriented approaches, e.g., speech act approaches and business process approaches was requested. It was questioned if the resulting model is better suited to communicate with the stakeholders than existing models. In addition, a clearer statement of the achievements would help to understand and relate the proposed approach better.
Søren Lauesen (co-author: Houman Younessi) proposed six important facets to be considered when defining usability requirements for a system: the time it should take for an (experienced and inexperienced) user to perform the task (performance style); the rate for serious usability defects in performing a task detected by the user (defect style); the number of prototype cycles to be performed for building the product (process style); the satisfaction rate of users for learning and using the system (subjective style); the screen lookouts to be used by the system which are usability tested during RE (design style); the style guideline to be used, e.g. Windows have at most three levels. To define the usability requirements for a particular system function the six facets must be combined in an appropriated way. The approach has been successfully applied to real applications. In the paper related discussion the six facets were seen as a good guideline to improve the definition of usability requirements during RE. The guidelines provided could be improved by associated well-known HCI techniques to the facets to be used for eliciting the requirements.
The third talk in this session was given by Neil Maiden (co-authors: M. Cisse, H. Perez and D. Manual). Neil proposed a pattern language for socio-technical system design to inform the validation of system requirements. Patterns are used to relate scenarios and requirements. A pattern, e.g., an input of essential information to the system, is defined by a situation in which the pattern should be applied, and a set of scenarios by which the actual defined requirements can be tested against "common good" knowledge expressed in the scenarios. Using structured walkthroughs, not only the requirements are validated against the scenarios, but also reflection on the patterns itself is enabled, e.g., improving the scenarios, the situation of the patterns, is enabled. Neil sees patterns as an ideal means for managing organisation knowledge about good RE practice, i.e., to facilitate the reuse of experience in new projects. In the discussion, it was questioned if the use of patterns to organise and enable the reuse of experience is better than other experience based approaches used for several years. The approach should be applied to real projects to validate, among others, if the time needed to check the requirements against the patterns pays of and if patterns per se are well suited to organise, manage and improve organisational knowledge.
Discussion Summary
The general discussion focused on the general issue of industrial uptakes of research results. The absence of well-trained people in the area of RE in industry was seen as one factor that makes industrial uptake of research results complicated. Researchers should thus invest more time and effort in the training of people.
Industrial uptakes, in general, takes place incrementally, i.e., it is quite unlikely that industrial uptake takes place in one big jump. Prerequisites to enable industrial uptake are the application of research results to real life cases, the integration with the methods used in industry, and the definition of easy-to-learn and easy-to-use guidelines (methods) which empower the application of the results in real projects.
The discussion of some examples seems to indicate, that there is an increasing number of researchers which downsize their research results to make them industrially applicable. On the other hand, the results are often not promising enough, i.e., industry is most often not willing to apply (parts of) the results in trial applications.
Nevertheless, the participants from industry and research agreed that closer co-operation are needed to improve RE practice, to provide feedback about research results, and to raise/enforce practice oriented research directions; lets get such co-operation started.
Session Summary:
Industrial Problem Statements
Overview of Talks
To ensure that RE researchers react to the needs of practitioners REFSQ’98 included a special session of industrial problem statements. The session turned out to be a fruitful experience which will be continued at future workshops. The problem statements dealt with a diverse range of topics indicating an abundance of industrially relevant RE areas which are open for future research.
The first statement, made by Sjaak Brinkkemper, discussed requirements management in the context of development of packaged software. The talk emphasised the need for academic RE research within an area characterised by many customers, many release cycles, platform independence and concurrent engineering of several aspects of the final products (software, documentation, training, etc.). Particular research issues suggested by Sjaak are: bundling of requirements, impact analysis after requirements changes, estimates of changed resource needs, interference between requirements and problem tracking, managing relationships between requirements, the separation of specific from standard requirements, specification of requirements by non-technical persons, as well as requirement management tools.
In his statement, Paris Bayias (co-authors: T. Hadzilacos and P. Velonias) presented problems with applying existing RE methods when the new system is heavily constrained by legacy systems. The talk reported experience from the OASIS automated electronic stock exchange system and outlined the RE approach used to establish requirements for OASIS. In the discussion, it was pointed out that the project seemed to have particular timeliness and stability requirements. It was also pointed out that the lack of an underlying representation framework might represent a problem.
Dieter Landes emphasised the importance of non-functional requirements based on experience from applying the Goal-Question-Metric paradigm to several RE projects at Daimler-Benz. Dieter stressed the importance of tightly coupling and assessing non-functional requirements in parallel with functional ones. The discussion pointed to the need for cost assessments in association with a methodology. However, since cost tends to be fixed, the practical consequence is more often a need to bundle and prioritise among the derived requirements. The usefulness of distinguishing between functional and non-functional requirements was also briefly discussed.
Finally, Marcelo Masera (co-author: P. Morris) reported the results from two workshops organised by the EC’s Joint Research Centre. The workshops aimed at identifying industrial RE research needs related to development of embedded systems. The extent to which existing industrial RE approaches and standards - mainly developed by and for large organisations with "high-end" RE needs - would scale to the majority of enterprises was controversially discussed.
Session Summary: Scenarios in RE
Overview of Talks
The last paper session of the workshop focused on the use of scenarios in RE. It continued along the session on scenarios and use cases of last year’s REFSQ workshop.
Björn Regnell (co-author: Per Runeson) presented a number of possible combinations of scenario-based RE with static verification and dynamic testing. He especially focused on the opportunity of using prioritised scenarios for guiding inspections and the combination of scenarios with reliability testing. The ability to derive test cases from scenarios clearly depends on the representation formalism used. Which scenario presentation is better for which type of test cases has to be evaluated. Björn stated the hypotheses that usage-based reading using prioritised scenarios will elicit the defects which matters most, that reliability tests are less difficult and less expensive to construct if they are based on scenarios, and that EP-analysis gives a more complete set of scenarios compared to ad-hoc scenario elicitation. Future work will test these hypotheses by defining concrete scenario-based test approaches and applying them to real case studies. In the discussion, the need for relating scenarios and test cases was controversially discussed. On the one hand, test generation approaches could be much more productive. On the other hand, proving to the customer that the developed software can deal with certain scenarios (use situations) in the defined way can better be achieved with scenario-based test approaches. It is an open research issue for which test situations (purposes) which approach should be applied.
The second talk of the session by Barbara Paech dealt with extensions to use cases to empower richer and thus better use case description. Barbara identified four levels of information to be included in a use case description, namely work division together with application data, application-specific system functions, work-specific system functions and dialogue definitions. The goal of extending use cases along the lines suggested, is to improve the fitness for use by integrating HCI, RE, conceptual modelling and OO development ideas. The proposed extensions should allow to express and interrelate different views on "use cases". In the discussion, guidance on how to define those level was requested. Moreover, the benefits gained by defining use cases using this more complex notation should be validated by applying the approach to real life problems. Thereby it could be tested if the four levels are sufficient and if they improve the use case usage.
In the last presentation of the workshop, Shailey Minocha (co-author: Alistair Sutcliffe) presented an approach for scenario-based analysis of non-functional requirements. The main goal of the work is the guided elicitation of non functional requirements (NFRs) using scenario templates. Based on an extensive literature survey and existing classification frameworks for NFRs, an comprehensive list of interrelations between NFRs and quality criteria was defined. Based on the relation to the quality criteria the interference of different NFRs was elaborated. For seven NFRs (usability, reliability, reusability, portability, security, scalability, performance efficiency) templates were proposed which aggregate advice on process and representation for analysing the NFRs in concrete projects. Those templates were then embedded in an NFR analysis method. The approach was seen as very ambitious, since each NFR is a major area by its own and thus requires a major effort to deal with it successfully. Thus, a restriction to a more narrow theme, i.e. to one NFR (e.g., security requirements) and an in-depth elaboration with domain experts was suggested. It was questioned if templates (patterns) really help to identify and specific NFRs.
Discussion Summary
The session discussion took the form of a brainstorm on open and/or problematic issues in relation to scenarios. The following issues were mentioned:
General Discussion
The starting point of the general discussion was the need to validate RE research in industrial settings. Although it was pointed out that REFSQ has clearly evolved in this direction over the years, it should be a goal of next year’s REFSQ to have even more presentations that reflect experience from practical examples.
It was pointed out that it is difficult to make commercial companies use new and yet unproven methods, techniques and tools. Hence the situation calls for new methodological approaches to validate research results.
Industrial participants pointed out that the scarce resource for industry is human time due to the scarcity of domain knowledge, not money! From the academic side it was stated that it is difficult to get clear problem statements from industry. The available surveys on states-of-practice are mostly done by academics.
As a preparation for REFSQ’99, a list of RE problem areas which had been collected throughout the workshop was presented and discussed. The list especially emphasised industrial problems and needs:
Pisa, Italy, June 1998
Andreas L. Opdahl and Klaus Pohl
References:
[1] Eric Dubios, Andreas Opdahl, Klaus Pohl, "Workshop Summaries REFSQ ‘97"; in: Proceedings of the REFSQ ’97 '97Workshop, F.U.N.D.P, Namur, Belgium, 1997
[2] Klaus Pohl; "Process-Centered Requirements Engineering"; RSP/Wiley and Sons, UK, 1996