Evaluation

How XR2ACH evaluates its learning environment – purpose, instruments, questionnaire design and the scales used, with sources.

Why we evaluate

Evaluation in XR2ACH is formative: it shows what works in the learning environment and what needs to improve – in operation, comfort, collaboration and technical content. Trials take place with small groups in single sessions. Claims about learning effectiveness need a study design of their own, with a comparison group and independent performance data; where we have that, we say so.

Apprentice wearing a VR headset in front of a screen showing his view, during a trial

The instruments

PeriodTrialInstrument
November 2025DKV conferenceFeedback form on the first version of the learning environment
February–April 2026Large-group tests, Bundesfachschule LeonbergQuestionnaire based on the Cognitive Affective Model of Immersive Learning (Makransky & Petersen, 2021), with a VR group and a comparison group; analysed by Neu-Ulm University of Applied Sciences
September 2026Multiplayer trial, LeonbergQuestionnaire on operation, collaboration, comfort and technical content
September 2026Train-the-trainer workshop, MaintalThree-part questionnaire for trainers: before, after the stations, after free exploration
from autumn 2026Further trialsQuestionnaire with validated scales (see below)

Answers are linked to the application’s pseudonymized learning data (xAPI) through a participant ID. This puts self-assessment and actual behaviour in the scenario side by side.

How the questionnaires are designed

  • COMFORT BEFORE AND AFTER Symptoms such as headache or dizziness are asked about before and after the session. Asking only afterwards assumes everyone started symptom-free, an assumption that does not hold empirically. Brown, Spronck & Powell (2022); Bimberg, Weissker & Kulik (2020)

  • PRESENCE IS NOT ONE QUANTITY Spatial presence, involvement and social presence are treated separately; our own items are not reported as a presence score. Bareišytė et al. (2024); Usoh et al. (2000); Degenhard et al. (2025)

  • USABILITY NEEDS TASKS A questionnaire measures satisfaction with operation; usability itself shows in concrete tasks that we observe. Balzerkiewitz & Stechert (2024); Webster & Dues (2017)

  • UNAMBIGUOUS ITEMS Our own items avoid reverse-worded statements because in small groups they led to contradictory answers.

  • FREE TEXT NEXT TO EACH BLOCK Not only at the end.

  • OWN ITEMS ONLY WHERE NO SCALE EXISTS There are no published instruments for leak detection or CAD visualization. Here, our own items are the appropriate choice.

Validated scales from the next trials

The session questionnaires so far consisted mostly of our own items. From the next trials, validated scales form the core, so that results become comparable with published benchmarks and between trials:

What is measuredScaleSource
Symptoms (before and after)Simulator Sickness Questionnaire (SSQ)Kennedy et al. (1993)
UsabilitySystem Usability Scale (SUS)Brooke (1996); Lewis (2018)
User experienceUser Experience Questionnaire, short version (UEQ-S)Schrepp, Hinderks & Thomaschewski (2017)
Workload for a taskNASA Task Load Index, unweighted versionHart & Staveland (1988); Hart (2006)
Presence (in research sessions)igroup Presence Questionnaire (IPQ)Schubert, Friedmann & Regenbrecht (2001)

Technical questions on CAD visualization and leak detection remain our own items.

Results

We publish results once they have been analysed and agreed with the partners involved.

View news and trial reports

References

  • Balzerkiewitz, H.-P., & Stechert, C. (2024). How to assess the usability of virtual reality (VR) systems for implementation in product development processes. Procedia CIRP, 128, 460–465. doi.org/10.1016/j.procir.2024.03.027
  • Bareišytė, L., Slatman, S., Austin, J., Rosema, M., van Sintemaartensdijk, I., Watson, S., & Bode, C. (2024). Questionnaires for evaluating virtual reality: A systematic scoping review. Computers in Human Behavior Reports, 16, 100505. doi.org/10.1016/j.chbr.2024.100505
  • Bimberg, P., Weissker, T., & Kulik, A. (2020). On the usage of the Simulator Sickness Questionnaire for virtual reality research. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE.
  • Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester & I. L. McClelland (Eds.), Usability Evaluation in Industry (pp. 189–194). Taylor & Francis.
  • Brown, P., Spronck, P., & Powell, W. (2022). The simulator sickness questionnaire, and the erroneous zero baseline assumption. Frontiers in Virtual Reality, 3, 945800. doi.org/10.3389/frvir.2022.945800
  • Degenhard, A., Askari, A., Rietzler, M., & Rukzio, E. (2025). When do we feel present in a virtual reality? Towards sensitivity and user acceptance of presence questionnaires. In CHI ‘25. ACM. doi.org/10.1145/3706598.3714204
  • Hart, S. G. (2006). NASA-Task Load Index (NASA-TLX); 20 years later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(9), 904–908.
  • Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183.
  • Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology, 3(3), 203–220. doi.org/10.1207/s15327108ijap0303_3
  • Lewis, J. R. (2018). The System Usability Scale: Past, present, and future. International Journal of Human–Computer Interaction, 34(7), 577–590.
  • Makransky, G., & Petersen, G. B. (2021). The Cognitive Affective Model of Immersive Learning (CAMIL): A theoretical research-based model of learning in immersive virtual reality. Educational Psychology Review, 33, 937–958. doi.org/10.1007/s10648-020-09586-2
  • Schrepp, M., Hinderks, A., & Thomaschewski, J. (2017). Design and evaluation of a short version of the User Experience Questionnaire (UEQ-S). International Journal of Interactive Multimedia and Artificial Intelligence, 4(6), 103–108.
  • Schubert, T., Friedmann, F., & Regenbrecht, H. (2001). The experience of presence: Factor analytic insights. Presence: Teleoperators and Virtual Environments, 10(3), 266–281. doi.org/10.1162/105474601300343603
  • Usoh, M., Catena, E., Arman, S., & Slater, M. (2000). Using presence questionnaires in reality. Presence: Teleoperators and Virtual Environments, 9(5), 497–503. doi.org/10.1162/105474600566989
  • Webster, R., & Dues, J. F. (2017). System Usability Scale (SUS): Oculus Rift DK2 and Samsung Gear VR. Purdue University. docs.lib.purdue.edu/soetfp/24