Keynote – The Journey of modelling in testing: from the past until today

Modeling has a long history. State diagrams were invented by David Harel in the 1980s. From the beginning, the core idea of modeling was to manage complexity. Today, the Unified Modeling Language (UML) is widely accepted. However, despite all the efforts of the International Requirements Engineering Board (IREB) to establish models in requirements engineering, its main application area remains system architecture and design.

The idea of model-based testing (MBT) emerged in 1999. It was originally conceived as an automated, tool-based approach. Take a test case generator, feed it a UML model from the system design, and obtain test cases from it with little to no effort.

Unfortunately, this approach has not proven successful for several reasons. One of them is the fact that testing is more than just checking against a design specification. Today, we understand MBT more broadly. The ISTQB defines model-based testing as “testing based on or involving models.”

In her keynote, Anne will take you on a journey from the early days of MBT to modern agile visual test design approaches. She will share her own experiences of good practices and common pitfalls that she has gathered over more than 15 years of practical application. As passionate as she is about MBT, it would be surprising if Anne didn’t end up convincing you as well. 

Keynote – The future of models in testing

The utilization of models in product development has seen a significant surge over the past years. For instance, models are employed to generate code for subsystems or produce interface code. Furthermore, they provide substantial opportunities for innovative testing methods, opening new avenues for efficient product development.

In software testing the trend of automation is clearly visible. More and more test steps are being automated, integrating seamlessly into the CI/CD pipeline. However, the primary challenge lies in the maintenance of these automated test cases. The absence of good programming practices can lead the test suite into a maintenance hell.

Another emerging trend is the incorporation of formal models during the product design phase. They serve as the blueprint for generating code, rendering handwritten codes obsolete. This technique has found notable success in high-tech environments, indicating a potential paradigm shift in the software development process.

As software development begins to embrace formality, test development too can adapt to a similar formal approach. This transformation can be realized with the help of model-based testing. It allows for the generation of test cases directly from a formal model that outlines the system’s behavior. From a maintenance perspective, this approach appears promising, although it also presents its own set of challenges.

In the foreseeable future, it is likely that we will see an extensive use of modeling for not just the system-under-test (for instance, an autonomous car) but also its surrounding environment. This includes factors like weather conditions, pedestrians, and other vehicular traffic, leading to a comprehensive virtual ecosystem. With such advancements, the future of testing is looking extremely promising, paving the way for more reliable and efficient product development.

Flaky Testing: The CI/CD Silent Killer

In this presentation, Erik will take you into the fascinating world of Flaky Testing. What exactly are these? Why are they so dangerous for the CI/CD pipeline? This presentation will also address common root causes and explore what we as testers can do to avoid becoming victims of this silent killer for the CI/CD pipeline.

Finding your way into Model Based Testing: models and tools 

In my experience, I have seen many definitions for Model Based Testing (MBT). Indeed, there are many different ways of using models in testing! What definition applies to a given situation? What is Model Based Testing actually? Are there tools that support these specific needs? In this talk, I would like to explore with you some answers to these questions. Together, we will look at different types of models and their associated tools and reflect about how they relate to testing objectives. At the end of the talk, we should together have a better understanding of the different approaches to model-based testing and be more efficient at using models for our testing activities.

Large Language Models in the scriptless test loop
 

Automated ent-to-end testing of UI’s is not an easy challenge. Although exploratory testing techniques as provided by Testar are helpful, three important challenges remain. The first is action selection, which determines which UI element is eligible to be interacted with. Determining the ‘best’ choice at a given state is a hard decision to make, especially when you want to replicate human behaviour as much as possible. The second is generating input for the aforementioned UI element (i.e., action selection). This should take the journey of the user thus far into account (i.e., the context) and provide meaningful input for the purpose of exploring the application in the best way possible, according to predefined heuristics. The third is state abstraction, which can be used to reason about groups of similar tests at once. This is achieved by removing irrelevant details of (historical) state, retaining only the key characteristics to detect state similarity.

In this talk, we will discuss how LLM’s can be used as a way to improve action selection by consuming the application state as its context, and suggestion which action to take. We will also explore how we can use the models to generalise the concrete application states, simplifying the challenge of creating state abstractions. Finally, we discuss how an LLM can be used to generate the actual input that is sent to the selected UI component that has been selected. Together, these steps explain how we can use large language models in the test loop.

You are cleared to use a model

When thinking about models in testing, many people think about ‘model based testing’. A model is made of the system and from that the cases are derived, preferably with elaborate tooling which also enables easy management of cases when the model changes. Many testers think this is the only ‘allowed’ context of ‘model based testing’. But…one can also think about ‘testing based on models’ and those models can be mental models, models to simplify complex systems, flow- and datamodels to be used for test designs… When a plane flies to its destination it gets cleared many times during its journey. It means it gets the ‘OK’ to go to the next part of its departure, flight or landing. In this talk I’m going to give you the clearance (do you actually need it?) to use models anywhere during your tests, by sharing various examples of the models I used – and still use- now I’m testing the Dutch Air Traffic system and meanwhile I share some insights in the air traffic system as well.

Sensemaking in testmodelling with TestCompass

In the dynamic landscape of software development, testing has emerged as the most commonly used technique for measuring software quality. Despite its
significance, software testing often receives insufficient attention in computer science education, leading to suboptimal testing practices among students and graduates.

Teaching software testing is a complex intellectual activity for which students need to allocate multiple cognitive resources at the same time. A systematically developed body of knowledge of didactic approaches, effects of educational settings, and learning outcomes is lacking.

As a first step in determining how software testing education can be improved, we have studied the sensemaking process of students designing test cases using an online modelling tool. We continue this research by studying the approaches taken by test experts when designing test cases. By comparing students to these experts, we will know where the deficiencies lie for them and what we need to train them on. We can use this knowledge to develop instructional designs that can be used in different educational contexts.

To address this issue, the presentation will shed light on the intricate process of teaching software testing.

Key Highlights:

• Unveiling the challenges associated with teaching software testing in educational settings.
• Exploring the multifaceted cognitive demands placed on students during software testing education.
• Examining the existing gap in didactic approaches and learning outcomes in software testing education.
• Insightful comparison between student-generated test cases and those created by seasoned test experts.
• Identifying deficiencies and opportunities for improvement in software testing education.
• Future prospects of developing innovative instructional designs to enhance software testing proficiency.

This presentation marks a significant stride toward bridging the gap between theory and practice in software testing education. Join us as we uncover valuable insights that will pave the way for a more effective approach to teaching this essential skill.

From BDD Scenarios to Test Case Generation 

Model-based testing (MBT) offers the possibility of automatic generation and execution of tests. However, it is not yet widely used in industry due to the difficulty in creating and maintaining models. On the other hand, Behavior Driven Development (BDD) is becoming more popular in the agile development process to achieve a common understanding of the system under development among stakeholders and to automate testing. However, BDD scenarios are written in human language and are usually not precise enough. Moreover, tests extracted from BDD scenarios are too short and incomplete; they only cover a very small part of the system. Our goal is to combine these two approaches to benefit from the usability of BDD and the test automation capabilities of MBT. In this paper, we first define a formal model of scenarios that we call BDD Transition Systems, second, we create more complete tests by composing scenarios (model composition), and finally, we generate and execute tests automatically. We demonstrate the applicability of this approach in a real-world example: an industrial printer.

Model-based User Experience Assessment
 

For many software applications user experience is important, yet assessing how the users would experience an application before the application is released is very hard. We can indeed do a Beta testing with a group of real users, but it is not always practical to do this between releases. In this talk we will discuss a model-based approach to model users’ emotional experience. The approach is rooted in a well-known theory from Psychology proposed by Ortony, Clore, and Collins. Compared to a machine learning approach, a more classical model based approach has the benefit that it is comprehensible (and explainable), but the challenge is indeed how to craft such a model. In the talk we will first discuss the structure of the underlying theory, and then we will discuss how a user model looks like and how we can use it to perform an automated user experience analysis.