Design process and preliminary psychometric study of a video game to detect cognitive impairment in senior adults

Design of the video game

The video game, named Episodix, was designed according to a methodology based on participatory design focused on the end user. The vision of senior adults captured in participatory design workshops and focus groups (Diaz-Orueta et al., 2012; Liamputtong, 2011) drove the design process from a technical and aesthetical perspective. Besides, experts in neurology, neuropsychology and geriatrics, and patients’ associations (cf. Acknowledgments below) provided the expertise required to ensure the validity of the new tool for cognitive assessment. The collaboration with health professionals (Brox et al., 2011) from an early design stage is necessary to guarantee both content and clinical validity.

In relation to the design approach, paper-based tests were taken as a reference trying to be challenging and fun at the same time, with the objective of creating a digital game that has the same construct validity as the original test. Existing recommendations (Brox et al., 2011; Nap et al., 2014) to develop games attractive to senior players were followed. Then, focus groups would confirm whether the game developed was challenging and fun to play, but also if it had a purpose or value, providing a perceived benefit and, eventually, was acceptable and usable by senior adults. Moreover, we tried to replicate real-life situations using virtual reality to reach an ecological solution that reflects the real complexity of daily tasks involving episodic memory. Table 1 summarizes the main structural elements of the game developed.

As mentioned above, the digital game implemented is based on the gamification of CVLT. In CVLT, a list of words is presented as a shopping list for Monday, and the subject taking the test has to remember as many elements in the shopping list as possible. Then, a second interference list is presented as the Tuesday shopping list. After the recall phase, a third list is produced including items from Monday and Tuesday together with new items, and the subject has to recognize elements from the Monday list (Delis et al., 1987).

In turn, the game developed consists on a virtual walk through a medium-sized town (cf. Fig. A1 for a street map of the game). During the walk, objects are presented to the participants—in visual and audio format—integrated naturally in the environment. These objects belong to three different collections of objects, namely List-A or main learning list; List-B or interference list; and List-C or yes/no recognition list, which have an average frequency equal to the original pen-and-paper test. In the same way as the conventional test, Episodix consists of the same set of phases, that is, “Learning” or stimuli presentation and “Recall” or remembering of objects. Players have to learn and recall as many objects as possible across several phases.

Given the importance of the concept of semantic category in CVLT, the translation of these categories to the Episodix game is further discussed below. In the pen-and-paper test, the learning lists A and B consist of four semantic categories, two of them being shared among lists (i.e., fruits and herbs). In the case of Episodix, the same happens, only that more ecological categories were chosen to be integrated in an urban-like walk. In Episodix, List A categories are urban furniture, vehicles, professions and shops; while categories in List B are urban furniture, vehicles, buildings, and tools (i.e., urban furniture and vehicles are shared). Most importantly, the average frequency of each list is the same in the pen-and-paper test and the video game. For this, word frequencies where checked taking as a reference a Spanish dictionary of word frequencies (cf. “Diccionario de frecuencias de la unidades lingüísticas del castellano” Alameda & Cuetos, 1995) and the corpus offered by the Google n-Gram service (Google books Ngram Viewer: https://books.google.com/ngrams). Next, sets of words were fetched belonging to the new categories having the same average usage frequency as the original test. For this purpose, a computer program was developed to automatically produce sets of four words per category maintaining an average frequency similar to those in the original test. This process is described graphically in Fig. 1. Note that this approach allows us to obtain, in a dynamic and automated way, different versions of the lists to avoid the practice effect across successive administrations.

Furthermore, the temporal administration pattern of Episodix is similar to the original pen-and-paper test (i.e., CVLT). The main difference corresponds to the number of trials and the intermediate break’s duration in order to force interferences during subject’s recall. Regarding the number of trials, as summarized in Table 2, the administration of List-A is repeated three times, while List-B and List-C, are administered only once. As the game offers a visual reinforcement, it is not necessary to repeat five times the Learning Phase of the first word list. The impact of this approach in the evaluation of the subjects’ learning ability is further discussed below (cf. ‘Results and Discussion’ sections). In relation to the waiting time between short and long term recall, in the original test it is recommended to wait for twenty minutes. This resting time is typically used to administer other neuropsychological tests. In the case of Episodix, it will be suggested to propose the subject to play two or three small games that evaluate other cognitive areas (e.g., attention, executive functions, gnosias, etc.), to guarantee enough waiting time to evaluate long-term recall. These areas, and more specifically attention and executive functions, may serve as early cognitive markers (Facal, Guàrdia-Olmos & Juncos-Rabadán, 2015; Juncos-Rabadán et al., 2012) for the detection of MCI or initial AD. Finally, apart from the actual administration time, pen-and-paper tests require an additional processing time that depends, among other aspects, on the previous experience of the person administering the test. In the case of the video game, this processing is transparent to both the subject under cognitive assessment and the administrator.

Another important design aspect is related to variables or data to be measured in the game. Three categories can be defined according to the granularity or level of detail offered: (i) high level, such as final game scores; (ii) medium level, such as the total number of movements or completed levels; and finally (iii) low level of detail, such as individual responses, accuracy, speed, right actions, errors and omissions. Eventually, low level variables were assessed to be the most appropriate for cognitive evaluation. Moreover, cognitive variables (e.g., yes/no recognition, free recall, short/long delayed recall, recency, primacy, semantic clustering, response to inhibitions, etc.) were also considered in a similar way to the pen-and-paper test (cf. list of variables in Appendix B, Table A1). The analysis of these data will eventually facilitate an assessment of episodic memory, and therefore to obtain some indication on the existence of early cognitive impairments (e.g., MCI or AD).

Development and implementation of the video game

Concerning the technical development of the video game, an agile and iterative methodology was applied (Singh, 2008). This methodology is widely used for the implementation of efficient software solutions. In relation to the development platform, we have chosen Unity (Lv et al., 2013), which has a reputable background in the development of 3D games to be run on multiple systems and devices. At the time of writing this paper the sixth development iteration (Third iteration available: http://episodix.gist.det.uvigo.es/.) was completed. The video game is being designed to run on several computer platforms (i.e., Windows, Linux, iOS and Android) and touch devices. It also supports accessible control interfaces such as joysticks, gaming mice, or gesture-controlled devices. Furthermore, Episodix has multilingual design and at the time of writing this paper it supports Spanish, Galician and English. The speed and rate of presented stimuli is adapted to each user according to their profile, and playing instructions and indicationsare provided in audio and text formats. To sum up, both design and implementation are aimed to facilitate usability and accessibility for any target user, and more specifically for senior adults.

On the other hand, to avoid the learning effect across administrations, Episodix supports multiple scenarios and different collections of objects. A fresh collection is displayed in each game session: List A, B or recognition.

Validation of the video game

The validation of Episodix is instrumental to guarantee the validity and reliability of the cognitive assessment performed. It involves evaluating the psychometric properties of the video game, which in turn are essential to achieve clinical validity. As a reference for the validation process, we relied on the Standards for Educational and Psychological Testing (AERA, APA & NCME, 2014), which state two key attributes for any test: reliability and validity. Reliability indicates the consistency of the test; that is, it should give the same results consistently, as it is supposed to measure the same construct. In turn, validity refers to how well a test measures what it is intended to measure, and includes content validity, criterion validity, face validity, ecological validity, external validity, and construct validity.

Several exploratory experiments were already conducted to evaluate the first four aspects of validity enumerated above (cf. ‘Outcomes from focus group sessions’):

• Content validity—Episodix’s content is based on the gamification of CVLT, which has been already validated to study episodic memory. Experts in neuropsychological evaluation participating in the design of the game and focus group sessions were instrumental to assess how episodic memory will be evaluated with the game by means of the tasks and variables in it to carry out this evaluation.

• Criterion validity—In our case, the aim was to assess whether or not subjects can be correctly classified only from their interaction with Episodix. For this, data collected from a first pilot experiment was processed using statistical techniques to classify participants in healthy individuals or MCI/AD impaired. Furthermore, we have conducted a correlation study between Episodix’s variables and classical tests administered, in order to prove convergent validity a subtype of criterion validity.

• Face validity—In this case, the objective was to check if the game actually appears to measure episodic memory. Meetings with experts in neuropsychological evaluation and the focus group sessions introduced above were used to assess face validity. During these meetings, different solutions for the look and feel of the game were proposed claiming that in all cases the final aim was to evaluate the participants’ memory capabilities.

• Ecological validity—Indicates what extent an evaluation instrument approximates real world conditions. Video games based on virtual reality are a reliable instrument to efficiently replicate situations from everyday life. In our case, focus groups were used to gather perceptions about the real-world appearance of Episodix. Moreover, participants in the pilot experiment discussed below were also questioned about this matter among other aspects such as the degree of usability and engagement.

To sum up, these experiences (two focus group sessions and a first pilot experiment), together with several meetings with experts in neuropsychological evaluation, dramatically facilitated the validation of the design of the video game and a preliminary psychometric study of this proposal. The study procedure was approved by the Galician Ethics Committee (Spain) (i.e., code 2016/236).

Data analysis

Data collected from interactions with Episodix was analyzed using machine learning techniques. Each observation is made up of generic and static data (e.g., age, gender, educational level, socialization level, physical exercise level, frequency of use of technologies, frequency of use of video games, etc.) and the variables obtained in a complete game for each participant. Analysis focused on predictive validity (i.e., predicting performance or behavior from collected data in order to detect cognitive impairment). For that purpose, support vector machine, linear regression and random forest techniques were applied (Maroco et al., 2011) as they are widely used in the biomedical field.

For each algorithm an 80/20 data split for training and testing was performed (Fan et al., 2008), and the experiment was repeated 5 and 100 times. A higher number of repetitions facilitated the obtaining of stable results, and therefore more representative. Then, the one-left-out (Soyka et al., 2012) approach was applied for training/testing each algorithm is trained with the whole sample except one, and then the same algorithm is used to predict the classification of the one left out; and this is repeated as many times as the number of samples available. In this case, the algorithm may be applied to all variables or to a particular subset including the most informative ones. For this, we applied random forest (Archer & Kimes, 2008), which facilitates the automatic selection of the most informative variables.

In order to obtain an accuracy value for each method, typical quality metrics about prediction ability were computed (Sebastiani, 2002); namely, precision (the ratio of relevant information over total information obtained during the prediction process); F1 (which offers an indication of the significance of predicted and recovered data after classification); sensitivity (the game’s ability to identify actually impaired people to avoid false negatives); and specificity (the game’s ability to identify healthy people to avoid false positives). ${\displaystyle P\left(\mathrm{Precison}\right)=TP-\left(TP+FP\right)}$ ${\displaystyle F1\left(F1score\right)=2\ast P\ast R-P+R}$ ${\displaystyle Sensitivity\left(Recall\right)=TP-\left(TP+FN\right)}$ ${\displaystyle Specificity=TN-\left(TN+FP\right).}$

Finally, during the pilot experiment a set of classical tests was administered to participating subjects (i.e., CVLT—Spanish version of California Verbal Learning Test (Delis et al., 1987), MMSE—Mini Mental State Examination (Cockrell & Folstein, 1987); MAT—Memory Alteration Test (Rami et al., 2010); and IQCODE—Informant Questionnaire on Cognitive Decline in the Elderly (Jorm, 2004)), in order to gather golden standard data to correlate with Episodix’s variables. Classical testing also served to guarantee that participants were correctly classified according to standard clinical practice, especially the MCI cases. CVLT and MAT were not administered to AD patients to avoid frustration. Moreover, to validate the convergent validity we have conducted a correlation study between Episodix’s data and those classical tests. Particularly with CVLT, we can show how the scores in the game world were benchmarked with the results of such cited pen-and-paper test.

Outcomes from the selection of the new collection of objects

In order to carry out a first pilot experiment with Episodix, we elaborated three collection of objects equivalent to the original pen-and-paper test in terms of the average frequency of use (cf. Fig. 2), according to the procedure described in ‘Design of the video game’, Eventually, we compiled a fully functional prototype with the following collections (cf. Tables A2 and A3 in Appendix C):

• List A—composed of 16 objects including urban furniture, vehicles, professions and shops. Their average frequency¹ is fr_A−Episodix = 0.000368, (fr_{A−CV  LT} = 0.000364 in the case of CVLT).

• List B—16 objects belonging to categories urban furniture, vehicles, buildings, and tools. This collection has an average frequency of fr_B−Episodix = 0.0000845, in comparison to fr_{B−CV  LT} = 0.0000745 in the case of CVLT.

• List C or recognition—it was constructed following the same criteria as in CVLT, that is, all objects (16) in List A; 8 objects from List B—2 for each common category, and 2 objects from the rest of the categories; 4 prototype objects from List A; 8 objects whose names had a phonetic relationship with elements in List A; and finally, 8 without any phonetic relationship. In this case, frequencies are fr_C−Episodix = 0.000855 vs. fr_{C−CV  LT} = 0.00120.

To determine the presentation sequence of the objects in each list along the Episodix’s virtual city walk, a matching was defined between categories in Episodix and CVLT. Specifically, the matching for List A associates tools with vehicles; herbs with shops; fruits with professions; and clothes with urban furniture. In the case of List B, cooking utensils was bound to vehicles; fruits to buildings; fishes to urban furniture and herbs to tools.

Outcomes from focus group sessions

The initial goal of focus groups was to gather seniors’ opinion on the use of video games in cognitive evaluation in general, and particularly of the digital game developed in this research. The two focus group sessions took place in March–April 2016 (cf. Fig. 3). The first group involved participants in the lifelong learning program of the University of Vigo² (three females and two males involved) and the second group was conducted in the premises of the Association of Family Members of Alzheimer Patients and other Dementias of Galicia-Spain (AFAGA in Spanish; http://afaga.com/en/) (six female and one male participants). All participants met the requirements of being +55 years old and active individuals. A high technological or educational level was not required. However, participants had to declare that they did not experience technological rejection or phobia. All participants read the provided patient information sheet and signed informed consent before participation in the sessions. In both documents, consent was requested to record audio speech and take photographs.

Both sessions had a similar duration of around two hours. Participants were offered coffee and pastries in order to get a comfortable and relaxed atmosphere to facilitate proactive participation. The protocol followed during both sessions (cf. Appendix D, Table A4) consisted in an initial introduction of all participants; a brief intervention expressing their opinion about video games and information technologies; a discussion on their understanding of cognitive assessment; a presentation describing the administration of CVLT; a first contact with a very early version of Episodix; and a debate on classical testing vs. the digital game. Finally, they were asked about the perceived usefulness of utilizing games to carry out cognitive evaluation. The outcomes of these sessions included audio files, photographic material and the informed consents signed by all participants. Text transcriptions were obtained from audio files.

The main results are summarized below:

• Content validity—Participants appreciated the idea of the city walk, the new semantic categories and the visual nature of the game rather than a person verbally producing word lists to remember.

• Face validity—Participants perceived the digital game as a useful tool. They described the idea of moving from a pen-and-paper test to a gaming format as interesting and original. However, they would not use it at home because they fear strong concerns about a possible negative outcome. They claimed that it should be a tool to be used by experts in clinical settings. Besides, they seemed motivated and would prefer to play the game instead of taking the pen-and-paper test in the case both instruments were eventually demonstrated to perform the same cognitive assessment. Incidentally, no matter the game presented was not a digital version of the original test but an actual game, they would prefer the term test to refer to it, as the term video game was perceived to have negative and non-serious connotations.

• Ecological validity—Focus group participants considered both the digital tool and its objects close to real life. In addition, and differently to the pen-and-paper test, the new tool was not perceived as being intrusive.

• Usability: the initial version of Episodix shown was controlled using keyboard and mouse. Participants found it easily administrable, even to use it on their own. However, they raised some concerns about the interfacing mechanisms, as they claimed that the gaming experience was influenced by users’ technological and physical abilities.

Outcomes from the initial pilot study

The pilot experiment involved a total of 16 subjects, 12 women and four men, living in the southwest area of Galicia (Spain) during May and June 2016. The sample was selected according to a cross-sectional approach, and was divided into three groups: (1) eight people without cognitive impairments or healthy control group (average age of 68.3 ± 8.88 years); (2) five AD patients (average age of 75.8 ± 5.36 years); and finally, three MCI patients (average age of 75 ± 6.08 years). In particular, 12 participants were provided by AFAGA 5 diagnosed with AD, two diagnosed with MCI and five healthy subjects (HC). The four remaining subjects (one MCI patient and three healthy subjects) belonging to the social circle of the authors, were selected randomly within the same geographic area. As in focus group sessions, all participants signed informed consent before participation in this study. The study procedure was approved by the Galician Ethics Committee (Spain) (i.e., code 2016/236). The pilot was conducted at participants’ homes (cf. Fig. 4) with the aim of facilitating a relaxing, non-intrusive and close to real-life environment. In other words, cognitive evaluation was carried out in the most ecological and natural environment possible. For that purpose, a more advanced prototype was used than the version introduced during the focus group sessions, more ecological and fully functional.

The pilot experiment consisted of two distinct parts. First, as we have already indicated, a set of classical tests was administered to participating subjects, in order to gather golden standard data to correlate with game, and also, to guarantee that participants were correctly classified (i.e., HC, MCI or AD group). Again, CVLT and MAT were not administered to AD patients to avoid frustration, as suggested by AFAGA.

Participants played a series of games following the time schedule of the original test: the first part of Episodix (to assess immediate and short-term recall), two short-games that evaluate other cognitive areas different from episodic memory; and finally, the second part of Episodix (to gather data about long-delayed recall and recognition ability). Moreover, an initial and final survey was also carried out to capture the user’s perception on cognitive games and its possible evolution after playing the digital game.

General and cognitive characteristics of subjects

Participants were characterized according to the parameters below (characteristics are expressed as median values and their standard deviation, due the reduced size of the sample, cf. Fig. 5).

• Frequency of use of ICTs (1[nothing] to 5[a lot]): 3 ± 1.58 for HC subjects; 3 ± 1.53 for MCI subjects and for 1 ± .0 AD patients.

• Frequency of use video games (1[nothing] to 5[a lot]): 1 ± .74 for HC subjects; 3 ± .58 for MCI subjects and for 1 ± .55 AD patients.

• Educational level. 25% of HC subjects completed university studies, 37.5% primary studies, and only 12% have not attended school. In the case of MCI subjects, one of them had university studies, another one attended primary school, and finally one had only reading and writing abilities. AD patients had the lowest educational level: 80% primary and 20% only reading/writing abilities.

• Socialization level (1[being alone most of the time] to 5[socializing a lot): 4.5 ±  .74 for HC subjects; 3 ± 1.00 for MCI subjects and for 3 ± .89 AD patients.

• Physical exercise (1[nothing] to 5[a lot]): 3.50 ± 1.41 for HC subjects; 3.50 ± 1.00 for MCI subjects and for 2 ± 1.09 AD patients.

• Classical test: results are summarized in the three graphics at the bottom of Fig. 5.

Blue box plots in Fig. 5 are distributed by cognitive groups, while red dashes indicate median values.

Preliminary psychometric study

Firstly, the initial pilot experiment allowed us to obtain relevant information to assess ecological validity, as 80% of participants found this approach closer to real life than the classical test. Similarly, subjects considered the Episodix game less intrusive than CVLT (i.e., 3.67 out of 5). Both aspects were confirmed by the healthy control group (cf. Table 3).

Secondly, with regard to criterion validity or the game’s ability to predict cognitive impairments, data captured from the Episodix game was processed using statistical techniques to classify participants in healthy individuals or MCI/AD impaired subjects. First, data captured from Episodix was considered, excluding data obtained from short games played during the break between short and long-term recall. As indicated above, linear regression, random forest and support vector machines were applied (Maroco et al., 2011) as predictive techniques, including a study of their quality metrics (cf. Table 4).

Table 4:

Experiments to assess of prediction accuracy of episodix using linear regression, random forest and support vector machines.

	Linear regression				Random forest				Support vector machine
	Precision	F1	Sensitivity (Recall)	Specificity	Precision	F1	Sensitivity (Recall)	Specificity	Precision	F1	Sensitivity (Recall)	Specificity
Episodix [80%–20%] executions = 5	0.73	0.70	0.68	0.83	0.75	0.50	0.38	0.92	0.29	0.25	0.22	0.55
Episodix [80%–20%] executions = 100	0.42	0.40	0.39	0.57	0.74	0.69	0.65	0.82	0.40	0.41	0.41	0.51
Episodix [100%]	0.58	0.56	0.54	0.44	0.69	0.76	0.85	0.44	0.55	0.50	0.46	0.44
Episodix [best 50%]	0.58	0.56	0.54	0.44	0.69	0.76	0.85	0.44	0.57	0.59	0.62	0.33
Episodix (best %) LR (best %) = 5%–6% RF (best %) = 12.5%	0.92	0.88	0.85	0.89	0.86	0.89	0.92	0.78	0.79	0.81	0.85	0.67
Episodix +CVG [100%]	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.90	0.83	0.77	0.90
Episodix +CVG [best 50%]	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.85	0.83	0.81	0.83
Episodix +CVG (best %) LR (best %) = 50% RF (best %) = 10%	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.90	0.83	0.77	0.90

DOI: 10.7717/peerj.3508/table-4

Notes:

Higher quality, when accuracy values were closer to the unit.

CVG

cognitive video games during break of Episodix (similar to the break of CVLT)

Using a 80/20 split for training and testing (Fan et al., 2008), and repeating the experiment 5 and 100 times, linear regression and random forest offered best performance, especially in precision and specificity values in the case of five repetitions. For the second case, using 100 executions, results are more stable but relevant only in the random forest case (i.e., precision = 0.74; F1 = 0.69; sensitivity = 0.65 and specificity = 0.82).

On the other hand, when a training/testing ratio based on the one-left-out technique (Soyka et al., 2012) is applied, the most favorable results are obtained when applied to the best subset of most informative variables selected using random forest (Archer & Kimes, 2008). In this case, the best outcomes are obtained using linear regression using the 5%–6% of most informative variables (i.e., all accuracy metrics are higher than 0.85). However, the performance in general is very positive (i.e., all metrics above 0.85, except specificity = 0.67 for SVM).

Finally, the same technique (i.e., one-left-out) was applied on the data extracted from all games, that is, first part of Episodix, two short games to assess semantic and procedural memory, and the second part of Episodix. A data set of 89 independent samples was obtained, which enabled the correct classification of all participants independently of the subset of the most informative variables selected, using linear regression and random forest. In other words, all the quality indicators achieved their maximum value of 1.00 (cf. 6th, 7th and 8th rows in Table 4). In the case of SVM, accuracy obtained is remarkable as well, higher than 0.88 using the whole data set.

Finally, convergent validity was studied through the correlation between Episodix’s variables and classical tests. As Episodix is focused on episodic memory assessment, this correlation study was focused in CVLT. As indicated above, to avoid frustration in participants with cognitive impairments, CVLT was administered to the healthy control group only. As depicted in Fig. 6, there exists a high correlation between CVLT variables and the following of Episodix variables: failures, guesses and omissions during short delay recall phases with clues; omissions from immediately recall phase during the first walk (i.e., RI-A1, RI-A2 and RI-A3) and during the second one (i.e., RI-B1); as well as time duration of short delay clued recall and free recall. Direct correlations in Fig. 6 are depicted in dark red, while indirect correlations are indicated by dark blue marks.