Relating biometric sensors with serious games results

. With the growing amount of data, it is necessary to create methods that facilitate its processing and and that allow a more efficient knowledge extraction from them. The main objective of the project here discussed is to correlate the performance of a patient while playing a serious game with the data collected, at the same time, by an eye tracker and the emotions detected by the Face API of Microsoft Azure suite after processing the photos collected by a webcam. Using analysis algorithms and adequate data visualisations, it is intended to discover new relationships that connect the game results to biometric data and create new knowledge. Through the eye movement and facial expressions, collected during the execution, it is possible to show that children with memory deficit have a slower processing speed, fix more times, make more regressions, and they have more disposal to express an emotion of sadness or surprise.


Introduction
Due to the rapid growth of technology today there is no shortage of data, which has created the need to make methods and tools that facilitate data processing and knowledge discovery [1]. Data mining (DM) is a process of analysing large amounts of data, which aims to discover patterns, relationships, and trends that are useful to extract new knowledge [2,3]. Nowadays, DM is applied in several areas, such as industry, health, and business. In health, DM can be applied to categorise diagnoses, treatments, and monitoring. In this area, DM is a big, is a big challenge because it is often necessary to gather data coming from different datasets with many attributes simultaneously with different time sequences [2].
This study intends to extract new knowledge through data coming from several sensors, collected during execution and a task. Initially, it would use data collected during a serious game that, in the health area, is used as an alternative to traditional therapy [4]. These help a patient to remain interested and motivated during therapy and make the process more enjoyable and dynamic. Subsequently, it is decided to use the results from diagnostic tests of attention and memory capacity, as well as a rapid naming test. With this study, it is expected that by applying a DM process, it is possible to extract new knowledge that makes it possible to make the best possible decision for the patient both in diagnosis and in therapy. In this study, sensors such as the eye tracker and webcam will be used to collect data. With the data coming from these sensors, it is intended to analyse the eye movement of the test and the facial expressions through a Microsoft API for emotion recognition.
The document follows the following structure: in Section 2, Background, the basilar concepts, necessary to understand the rest of the paper, are introduced. Then in Section 3, the experiments carried out in different schools with children are described. Results obtained from the analysis of data collected along the referred experiments are discussed in Section 4. Finally, the paper is concluded with Section 5.

Eye Movements
Eye moments can be divided into two categories, stabilisation movements and saccadic movements. The first fixes the images on the retina; including fixation and smooth pursuit movements. The second brings objects into view; these include saccades and vergence movements [5].
Fixation corresponds to the time interval that the object of interest is stable on the retina, that is, the eyes are fixed on an object [5,6]. A fix can last between 100ms to 1000ms, but it usually lasts between 180ms to 330ms. A single fixation is not enough to acquire quality information about the entire field of vision, which causes the eyes to move repeatedly [7]. Saccadic movement is the most common and the most important to understand the perception of the scene. These are fast and precise movements that occur between one fixation and the next, and they serve as a support for visual exploration during the readjustment between the eye fovea and the areas of interest in the optical space [5,8]. The duration and speed of a saccade vary with the distance covered [6,7]. In reading, a saccade lasts at least 30ms, which corresponds to a rotation of 2 degrees, while a scene perception saccade corresponds to 5 degrees and lasts between 40ms and 50ms [10,11].

Emotions
The theme of emotions is a subject on which there is no unanimity among the different disciplines of knowledge [12]. There is no general proposition accepted around the world.
Emotions are an instant reaction to a specific event, these can be confused with a feeling or mood [13]. According to the Paul Ekman Group, an emotion does not last more than an hour without interruption and, if it persists for a long period, it can be considered a humour state 1 .
The two most accepted theories for emotions classification are the theory of basic emotions and the dimensional theory [14]. In the first, the main idea is that humans have a limit of emotions, which are associated with behavioural elements with recurring patterns. Although this theory is accepted by many, there is still some controversy about the number of basic emotions. Ekman initially proposed a list with six basic emotions (angry, fear, sadness, disgust, happiness, surprise), represented by facial expressions as shown in Figure 1a 2 , where marks that separate disgust from contempt were later observed [15]. There were also those who proposed 8 basic emotions: anger, joy, fear, disgust, sadness, confidence, surprise, and anticipation [14]. In a recent study [16],it was shown that anger, disgust and surprise share similar expressions, eventually proposing that humans only have 4 basic emotions: fear, anger, joy, and sadness.

Sensors
Eye-tracking is the process of registering eye movement and locating the eye position when a person performs a specific task [5,6]. In the 20 th century, several approaches to measuring eye movements were developed, but these are expensive and laborious, which is an obstacle for researchers. Nowadays, with the advancement of technology, tracking eye movements has become easier [6]. The eye trackers illuminate the eyes with a light source, generally infrared because it is unnoticeable to our eyes. The light is reflected off the cornea and through the software calculates the position of the eye [6,17]. Exist different eye trackers devices in which each has its advantages and disadvantages, such as speed, accuracy and cost [17], despite using the same principle 3 . The eye tracker can measure eye movement as the position of the eye can be counter-plotted several times per second. These can collect different measures that help you understand the behaviour of the interpretive process 3 . The most common metrics are the number of fixations, the number of regressive fixations, duration of fixations, amplitude, the peak speed of saccade, blink rate, blink amplitude and duration, phasic, and tonic pupil diameter [17]. The detection of emotions is possible using the Microsoft Azure Cognitive Services to read the image captured by the webcam. Azure Face Services is part of the cognitive system Services. This service contains AI algorithms to detect, analyse and recognise faces in images. From the service, the Face API was used to detect emotions. This API (Application Programming Interface) uses 27 facial points simple to recognise on a face as a reference, such as pupils, eyebrows, or the tip of the nose (Figure 1b 4 ). Using this API, we get for each face detected in the image the following assigns, like a FaceID, an age estimate, a list of emotions, the gender, whether you wear glasses and what kind of glasses, among others.

Methodology
In this study, two diagnostic tests were performed on children aged 6 to 12 years. One of the tests measures attentional capacity and the other measures memory capacity. The children also performed a rapid naming test to assess neuro-cognitive function (reading of geometric figures and colours along 5 lines). In addition to recording the result of this test, eye movement and facial expressions were registered to detect emotions throughout the test. The collection of this data was done through a platform, which uses an eye tracker and a webcam, and stores this data for later analysis. This study involved 121 children, where 27 children belong to the age group from 6 to 7, 48 from 8 to 9 and 46 from 10 to 12.

Data Collection
Three tests were carried out, one of the tests was used to assess orientation attention, which measures the ability to concentrate, the power of achievement and the fatigue resistance and thus detects possible deficits in attention. The test lasts 10 minutes and consists of finding the 2 symbols referred to, selecting them from a set of randomly placed stimuli. Another test consists of measuring the ability to associate geometric shapes to symbols (for children from 5 to 7 years old) and numbers to patterns (for children from 8 to 16 years old) and to correctly memorise these associations. The last test applied is the only test that has eyemovement tracking and emotions recording. This consists of naming colours or shapes and colours as quickly as possible. As the previous test is divided by age groups, from 5 to 6 the quick naming of colours test (which contains stimuli of a single category) is applied and, from 7 to 16 the quick naming of shapes and colours test (which contains stimuli from 2 class). Of the 121 children who participated in this study, 14 (children with 6 years old) took the quick naming of colour test and, the rest (children between 7 and 12 years old) took the quick naming of shapes and colours test. The participants performed the 3 three tests in the following order: first, the test that measures attention capacity, then the test that measures memory capacity, and finally, the rapid naming test. Then, these tests are corrected to obtain the level of attention and memory. That said, the naming test takes place where the child is placed approximately 66cm from the computer, centred with the screen and the webcam. The test is explained to the child and an example is given. When the test starts, the place (row/column) where an error or omission occurred is registered.
When the proof ends, all the points, the fixations, the confidence of 8 emotions are registered on the platform. Then, with these data, some metrics are calculated, such as the duration of the test (which is necessary to calculate your general result), total points, total fixations, total regressions and the percentage of fixations. A regression is a horizontal eye movement (shown in Figure 2) between two fixations, which occurs from right to left. To these data is added the result of the attention and memory capacity tests, as well as the result of the rapid naming tests such as errors, omissions, total errors plus omissions and the overall result of this test. The calculation of metrics and the joining of these data is done through python scripts and later saved in an excel file. -Experiment -Integer: experiment identifier; -Person ID -Integer: person identifier; -Age -Integer: respondent's age; -Wisc -Integer: represent the percentile obtained in the memory test; -Cancellation -Integer: represent the percentile obtained in the attention test; -Rapid Naming -Integer: represent the percentile obtained in rapid naming test The average of emotions over each execution is calculated to divide respondents by the emotion with the highest average. That is, respondents were divided into three files, one for those who expressed more joy (during the test), another for surprise and another for sadness.These are made up of the experiment identifier, the execution identifier, the person identifier, the execution time and the value of the result of the tests.
To obtain results, it was chosen to see the correlation between the attributes, to see if there is any relationship between them. For this analysis, the SPSS program was used. As different tests were performed, according to age, it was decided to divide this analysis into the following three scenarios: -S12 -Only the data from the quick naming of colour.
-S13 -Only the data from the quick naming of shapes and colours.
Before calculating the correlation matrix, it is necessary to check if the attributes follow is normality. For this verification, there are 2 tests: Shapiro-Wilk, which is applied when n is less than 50, and Kolmogorov-Smirnov, which is applied when n is greater than 50. If the significance of the attribute is less than 0.05 then the attribute does not follow a normal distribution. with this check, you know which correlation to use. If all attributes follow a normal distribution then it applies to Pearson's, if one of the attributes does not follow a normal distribution then Spearman's ρ is applied.
After analysing the correlation matrix, graphs representing the mapping of points over time were generated. The average of each emotion expressed throughout the test was also analysed.

Results Discussion
In scenario S1, as n is equal to 14, the Shapiro-Wilk test was applied to verify the normality of the attributes. It was obtained that the attributes "Cancellation", "Time", "Errors", and "Omissions" do not follow a normal distribution, so the Spearman's ρ will apply. Through Spearman's ρ, it was obtained that, about the attributes referring to the tests of assessment of attention capacity and memory, the only attribute that has a significant correlation is the "Total E+O". In both cases, it is a negative correlation that is when the value of one of the variables rises the value of the other variable goes down. In this case, it means that children with a lower attention or memory capacity tend to make more mistakes in the quick naming test. These correlations are represented in Table 1 (S1).
There are other significant correlations, such as the correlation between the variable that corresponds to the assessment of the test and quick naming with the total number of regressions that the respondent gave. This is also a negative correlation, with a value of -0.818, and means that the worse the result in the quick naming test (smaller value) the greater the number of regressions. Another correlation, that is interesting to note is the correlation between the total number of fixations given in the rapid naming test with the total time because it is a positive correlation, that is, there is a tendency that the greater the number of fixations, the longer the test execution time.
For scenario S2, as n equals 107, the Kolmogorov-Smirnov test was applied to verify the normality of the attributes. The only attributes in this scenario that follow the normal distribution are "Cancellation" and " % de Regressions". That said, in this scenario, Spearman's ρ is also applied to calculate the correlations, these are represented in Table 1 (S2). In this scenario, more significant correlations are obtained, despite their low values. the most relevant are the correlations between the attribute that represents memory capacity and the other attributes. The correlations with the highest value are the positive correlation between the quick naming test result value (0.432) and the negative correlation with the attribute representing the total test execution time (-0.429). The correlation between the attribute that represents the memory capacity and the attributes "Points", "Fixations", "Regressions", "% of Regressions" is significant and negative, which shows a tendency that the greater the memory capacity, the lower the values of these attributes. There are also significant correlations, which did not exist in the previous scenario, between the attribute "E+O" and the rest attributes, except for the attribute "Cancellation".
Concerning scenario S3, the Kolmogorov-Smirnov test was also applied because n = 121. As in the previous scenario, the only attributes that follow a normal distribution are "Cancellation" and "% of Regressions". As in the other scenarios, Spearman's ρ is also applies for the calculation of correlations, the results of which are represented in Table 1 (S3) . In this scenario, all attributes have a significant correlation with the attribute that represents the memory test result. Two positive correlations are being the strongest (0.437) with the attribute that represents the quick naming test result. The remaining correlations are negative, being the strongest (-0.364) with the total time of execution of the fast naming test. The attribute that represents the result of the attention test has 2 positive correlations, the strongest being with the attribute "Rapid Naming" (0.307) and a significant negative correlation with the attribute "E+O".
Through the graphs of the points that occurred over time, it was possible to prove the result of the correlations, that is, the graph of a child with a memory deficit has a large number of points and fixations. Also, it is clear that if there is a deficit of attention, they are more dispersed. As for the expressed emotions, it appears that 23 out of 36 children who expressed, on average, greater emotion of happiness, have good memory capacity and, 10 of these 23 have good attentional capacity. Those who expressed, on average, a greater emotion of surprise tend to have a lower memory capacity (28 out of 50), although 22 out of 50 have a good memory capacity. Of the 50, only 18 have good memory capacity. Regarding those who express more sadness, 13 of the 28 have lower memory and attention capacity and, 6 have good memory and attention.

Conclusion and Future Work
In this study, different diagnostic tests for attention and memory deficits were carried out in several children aged 6 to 12 years. We also carried out a rapid naming test taken from the BANC ("Bateria de Avaliação Neuropsicológica de Coimbra"), coupled with the registration/recording of eyes movement and facial expressions for emotions detecting. With the data from the memory and attention diagnostic test, it was possible to better characterise the respondent and try to relate their profile with the result of the rapid naming tests.
It was possible to verify analytically that as the memory deficit increases, among others, the number of fixations, the number of regressions and the time taken to complete the test increases. However, there are no statistically significant values when we correlate these data with attention deficit. Through the analysis of eye movement, it was possible to observe that children with greater attention and memory capacity had a greater ability to read the 5 lines of geometric figures in an orderly manner (which form the above-mentioned language test), while children with deficit attention or memory were more likely to get lost, that is, they don't just focus on the line they are reading. Regarding emotions, this study shows us that children who expressed happiness during the test point to a better memory capacity, while children who expressed emotions of surprise or sadness have a propensity for memory deficit.
To conclude, data mining in healthcare is kind of complex because it involves a large number of variables that come from different sensors, which usually capture data at different time periods, which makes it more complicated to combine the data. With the collected data correlations were used, but there are other ways (like association rules) that, depending on the available data, can also bring relevant information.
As a prospect for future work, it would be interesting to explore the data collected by other sensors, such as an electroencephalogram. To apply new DM methods to the data sets and collect data from a larger population. It would also be interesting to search for understanding whether emotions occur as a result of running the test or whether an emotion is inherent to the child.