KEYNOTE: Testing Challenges for Cyber-Physical Systems​

KEYNOTE: Complexity of Simplicity – Testing a Microservices Architecture

Testing in an always on situation

During my career I have been part of testteams that had to test systems that shouldnot have down time.

How do you test such systems functionally and non-functionally. And more important how do you make sure there is no downtime, when you are testing mission critical systems. And how to bring the message that you should or shouldn’t go live because you have doubts. And what to do when you go live but you shouldn’t have gone.

With real life examples I will make clear that sometimes you have to take a risk

How many virtual users should I test? A virtual approach to designing performance test load plans

Many performance tests start with a simple question: “How many virtual users should I test with?”

User count alone rarely predicts bottlenecks. Congestion emerges from where users go, how long they stay, and where journeys overlap.

In this presentation, we explore a visual map of a fictional holiday park application to demonstrate how user journeys can be transformed into a realistic workload model. Following the journeys through hotels, campsites, cabins, and viewpoints, we gradually build a model by identifying behavioural groups, estimating arrival rates and dwell times, and revealing how concurrency creates hotspots.

Along the way, the audience is invited to predict outcomes and challenge assumptions, illustrating how different modelling choices lead to different testing strategies.

Attendees will leave with a practical framework for designing realistic performance test workloads, enabling them to justify their load-testing assumptions and identify performance risks before creating their load tests.

PICKLES: a natural language framework for requirement specification and model-based testing

This talk introduces PICKLES, a structured natural-language extension of Gherkin designed to bridge the gap between human-readable specifications and Model-Based Testing. It enables bi-directional translation between scenarios and formal models, allowing tests to remain readable while still supporting automated model-based test generation. By combining multiple scenarios into a unified “master model,” standard MBT techniques can be applied directly, and the generated formal test cases can be then expressed back in natural language, using consisting terminology. This way, we can improve automation and coverage without sacrificing specification clarity.

How to survive in the AI jungle: Addressing reasoning risk in autonomous systems

“100 percent correct does not exist in AI. So what does quality mean?”

AI is redefining how we build software and how we test it. In this talk, Nicole introduces Evaluaite, a practical framework for governing quality in AI-driven systems. She explains why traditional pass or fail testing no longer works in probabilistic environments and how testers can evolve their role in the AI era. Through real-world examples and practical approaches such as LLM evaluations, she demonstrates how QA professionals can move beyond binary thinking and make AI quality measurable, manageable, and strategically governed.

Quick bug detection through black-box checking: a systematic evaluation

Combinations of active automata learning, model-based testing and model checking have been successfully used in numerous applications, e.g., for spotting bugs in implementations of major network protocols and to support refactoring of embedded controllers.

However, in the large majority of these applications, model checking is only used at the very end, when no counterexample can be found anymore for the latest hypothesis model. This contrasts with the original proposal of black-box checking (BBC) by Peled, Vardi & Yannakakis, which applies model checking for all hypotheses, also the intermediate ones. We present the first systematic evaluation of the ability of BBC to find bugs quickly, based on 77 benchmark models from real protocol implementations and controllers for which specifications of safety properties are available.

Our main findings are: (a) In cases where the full model can be learned, BBC detects violations of the specifications with just 3% of the queries needed by an approach in which model checking is only used for the full model. (b) Even when the full model cannot be learned, BBC is still able to detect many violations of the specification. In particular, BBC manages to detect 96% of the safety property violations in the challenging RERS 2019 industrial LTL benchmarks. (c)

Our results also confirm that BBC is way more effective than existing MBT algorithms in finding deep bugs in implementations.

The environment is the challenge, not the test object

We often talk about the challenges of the test object itself: high availability, high risk, or legacy systems. Yet, there is one common challenge that underlies them all: the people around you and the organization in which you operate. While organizations

have been made more agile through frameworks like Agile Scrum, why hasn’t predictability increased? Why is quality still under pressure? As a local saying goes: “We drink a glass, we take a leak, and everything stays the same.”;

I believe that we, as testers, need a different mindset and must shift further left. Especially with the rise of AI, we need to validate quality much earlier. This ensures agility isn’t reduced to “mopping the floor with the tap running”, but instead enables us to adapt agilely to implement the right feature (without the need for extensive rework), and ensuring that integration and end-to-end testing can simply take place as scheduled.

In my presentation, I will show what extra preparation you can do as a team to make your work genuinely fun again. Using testing to prove that the system simply does what it is supposed to do. That is where the real challenge lies, making it practically irrelevant whether you are working on a legacy system, a high-availability network, or a mission-critical solution.

Takeaway

Garbage in is garbage out. Learn what you can do to prevent it, so you can refocus your energy on the true challenges of the test object!

Agent-based Test Generation with Large Language Models and Stryker

Software Testing is an essential but time-intensive discipline within Software Engineering. Once a software test is in place, we can evaluate its quality using, among other metrics, mutation testing. This method of analysis uses artificial source code bugs (mutations), and measures how many of them cause test failure. It is itself notoriously time consuming but provides feedback on the essential part of Software Testing: potential fault detection.

Recently, Large Language Models (LLMs) have shown promising results in automating a wide range of classical Software Engineering problems. Additionally, these models can be trained to use tools to form so-called agents, capable of completely autonomous work. In this thesis, we extend research in this direction by providing an agent with various tools before prompting it for the generation of entire JavaScript test suites. In order to provide feedback signals on generated test quality, we integrate the JavaScript mutation framework StrykerJS at various points in the test generation process.

We evaluate the effectiveness and cost of the agentic test generation approach by comparing against an LLM-based JavaScript test generation framework that integrates test quality feedback but does not enable autonomous tool calls. Additionally, we measure the effectiveness of the mutation feedback by splitting our experiments into groups with and without detailed mutant information. We find that our solution can create stronger tests while re ducing LLM token costs. Mutation analysis proves to be a useful signal, but only when scoped correctly

Actual Intelligence: a path into the future