Using machine learning to model trace behavioral data from a game-based assessment

Published in International Journal of Selection and Assessment, 2022

Recommended citation: Auer, E. M., Mersy, G., Marin, S., Blaik, J., & Landers, R. N. (2022). Using machine learning to model trace behavioral data from a game-based assessment. International Journal of Selection and Assessment, 30, 82–102. https://doi.org/10.1111/ijsa.12363 https://doi.org/10.1111/ijsa.12363

Download paper here

Suggested citation:

Auer, E. M., Mersy, G., Marin, S., Blaik, J., & Landers, R. N. (2022). Using machine learning to model trace behavioral data from a game-based assessment. International Journal of Selection and Assessment, 30, 82–102. https://doi.org/10.1111/ijsa.12363

BibTeX citation:

@article{auer_2022, author = {Auer, Elena M. and Mersy, Gabriel and Marin, Sebastian and Blaik, Jason and Landers, Richard N.}, title = {Using machine learning to model trace behavioral data from a game-based assessment}, journal = {International Journal of Selection and Assessment}, volume = {30}, number = {1}, pages = {82-102}, keywords = {data science, game-based assessments, games, machine learning, microbehaviors, psychometrics, trace data}, doi = {https://doi.org/10.1111/ijsa.12363}, url = {https://onlinelibrary.wiley.com/doi/abs/10.1111/ijsa.12363}, eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1111/ijsa.12363}, abstract = {Abstract In the context of game-based assessments (GBAs), we examined the potential of trace data modeling to supplement or replace an existing GBA’s scoring approach. We used data science tools and “big data” practices, such as feature engineering and a series of machine learning algorithms, to predict traditionally measured cognitive ability and conscientiousness scores from the copious trace data generated by a theory-driven GBA designed to measure cognitive ability. Several types of predictors were developed from the raw trace data from 621 participants, including counts of game objects that the player interacted with, the amount of time spent doing so, and mouse movement data across a variety of meaningful intervals. Broadly, we found promising evidence for trace data modeling of cognitive ability, including incremental contribution to the prediction of a criterion grade point average (GPA), but less promising evidence for trace data modeling of conscientiousness, suggesting that trace data modeling like this may be more valuable for assessing traits in games that were developed to target those traits.}, year = {2022} }