KEY NOTE SPEAKER

Known and unknown unknowns

Applying Behaviour-Driven Development for Game Creation

How to test computer games? In computer games, a player interacts with advanced (AI) agents, and deals with extensive game worlds. While computer games can be immensely complex, and bugs show up in well-known games, testing has not been picked up as much in the game software engineering community, as it has in traditional software engineering. In this talk I will show how Behavior-Driven Development, which is a popular technique for specification and testing in traditional software engineering, can be applied in game software engineering as well. Specifically, I will present the highlights of (i) a framework to help express game behaviors as BDD scenarios, (ii) a method to apply BDD in game development, and (iii) tooling to apply BDD in Unity 3D, a major game development platform.

In these disruptive times, when code and tests can be created in a few minutes by generative AI, students are less than ever motivated to learn. In an attempt to alleviate the situation, we developed VU-BugZoo, an innovative teaching platform that instead of asking students to create test plans on paper, for hypothetical code, it engages them in exciting bug detective games in executable, standalone- and embedded- code. While grading, we praise an elegant test strategy above just finding the bugs. The results show that the approach succeeds in increasing the excitement and engagement in today’s software testing classrooms.

We push the button and let our tests run. But what really happens? What machinery is put into motion when we test and what are the environmental consequences of our quality assurance process? In this talk, I will take you through some of the insights that we have made while closely examining the potential environmental impact of building and testing 204 Java open source projects. The outlook is that testing is perhaps not as green as we might have thought.

Het gebruik van model-based testen in het test proces voor medische software

TBD

Today’s world depends on many digital services and the communication between them. To facilitate this communication between applications, standardised and well-specified application programming interfaces (APIs) are often used.

In particular, the use of well-defined representational state transfer (REST) architectural constraints for APIs is popular. As an entry point to many applications, these APIs provide an interesting attack surface for malicious actors. Furthermore, since APIs often control access to business logic, a security lapse can have high-impact undesirable consequences.

Thorough testing of these APIs is therefore essential to ensure business continuity. Manual testing cannot keep up, so automated solutions are needed. In this talk, we introduce and demonstrate WuppieFuzz, an open-source, automated testing tool that makes use of fuzzing techniques and code coverage measurements to find bugs, errors and/or vulnerabilities in REST APIs.

Today’s world depends on many digital services and the communication between them. To facilitate this communication between applications, standardised and well-specified application programming interfaces (APIs) are often used.

In particular, the use of well-defined representational state transfer (REST) architectural constraints for APIs is popular. As an entry point to many applications, these APIs provide an interesting attack surface for malicious actors. Furthermore, since APIs often control access to business logic, a security lapse can have high-impact undesirable consequences.

Thorough testing of these APIs is therefore essential to ensure business continuity. Manual testing cannot keep up, so automated solutions are needed. In this talk, we introduce and demonstrate WuppieFuzz, an open-source, automated testing tool that makes use of fuzzing techniques and code coverage measurements to find bugs, errors and/or vulnerabilities in REST APIs.

Take away

This presentation explores how automated white-box validation following the LLM-as-a-Judge paradigm can be used to assess and improve RAG pipelines, offering practical insights into maintaining the trustworthiness and effectiveness of RAG-based applications over time.

Abstract

With the advent of foundation models and generative AI, especially the rapid rise in Large Language Models (LLMs), we see our students and companies around us build a popular type of AI-enabled systems: private document chatbots based on Retrieval-Augmented Generation (RAG). From these systems, we learned that it is not hard to build something that works from a technical point of view, but that the true challenge lies in building solutions that are thrustworthy and capable of delivering consistently high-quality results.

Validating RAG pipelines in practice is challenging for several reasons: 1) the document retrieval step can be configured in many ways and is sensitive to the structure and quality of input documents; 2) inference of high-quality answers depends on the relevance of the retrieved documents; 3) assessing answer quality is subjective and often require domain expertise; 4) improving RAG pipelines involves experimentation with new techniques and configurations, which demands structured and consistent evaluation methods.

Most existing validation frameworks treat RAG systems as black boxes, evaluating only the final output without insight into how that output was produced. In contrast, our approach takes a white-box perspective: by leveraging knowledge about the inner components of the RAG architecture and evaluating the outputs of intermediate stages, we gain a deeper understanding of system behavior and failure points.

To support this, we are developing an automated validation framework, that combines both traditional, calculable metrics with more nuanced, LLM-based evaluation techniques following the LLM-as-a-Judge paradigm. This provides more granular insights, supports targeted improvements, and allows for more reliable quality assurance.

In this presentation, we share key insights from our work and show how automated white-box validation can be applied in practice to assess and improve RAG pipelines. We explore the benefits and challenges of this approach and illustrate how it enables ongoing quality monitoring to help RAG-based applications remain trustworthy and effective over time.

Take away

This presentation explores how automated white-box validation following the LLM-as-a-Judge paradigm can be used to assess and improve RAG pipelines, offering practical insights into maintaining the trustworthiness and effectiveness of RAG-based applications over time.

Abstract

With the advent of foundation models and generative AI, especially the rapid rise in Large Language Models (LLMs), we see our students and companies around us build a popular type of AI-enabled systems: private document chatbots based on Retrieval-Augmented Generation (RAG). From these systems, we learned that it is not hard to build something that works from a technical point of view, but that the true challenge lies in building solutions that are thrustworthy and capable of delivering consistently high-quality results.

Validating RAG pipelines in practice is challenging for several reasons: 1) the document retrieval step can be configured in many ways and is sensitive to the structure and quality of input documents; 2) inference of high-quality answers depends on the relevance of the retrieved documents; 3) assessing answer quality is subjective and often require domain expertise; 4) improving RAG pipelines involves experimentation with new techniques and configurations, which demands structured and consistent evaluation methods.

Most existing validation frameworks treat RAG systems as black boxes, evaluating only the final output without insight into how that output was produced. In contrast, our approach takes a white-box perspective: by leveraging knowledge about the inner components of the RAG architecture and evaluating the outputs of intermediate stages, we gain a deeper understanding of system behavior and failure points.

To support this, we are developing an automated validation framework, that combines both traditional, calculable metrics with more nuanced, LLM-based evaluation techniques following the LLM-as-a-Judge paradigm. This provides more granular insights, supports targeted improvements, and allows for more reliable quality assurance.

In this presentation, we share key insights from our work and show how automated white-box validation can be applied in practice to assess and improve RAG pipelines. We explore the benefits and challenges of this approach and illustrate how it enables ongoing quality monitoring to help RAG-based applications remain trustworthy and effective over time.

AI Based Testing versus Model-Based Testing