A multimethod approach to assessing children’s 21st century digital skills

Authors: Laure Lu Chen, Nirmala Rao

Please cite as: Lu Chen, L. & Rao, N. (2022): A multimethod approach to assessing children’s 21st century digital skills. In S. Kotilainen (Ed.), Methods in practice: Studying children and youth online (chapter 3). Retrieved DD Month YYYY, from https://core-evidence.eu/methods-toolkit/handbook-part1/handbook-multimethod-approach, doi: https://doi.org/10.21241/ssoar.83031.

Education is central to preparing children to master 21st century digital skills, and many countries have reformed the assessment of learning of important skills among school-aged children. This paper considers the assessment of 21st century digital skills leveraging practices from the Learning and Assessment for Digital Citizenship project¹ in Hong Kong.

Claro et al. (2012) first defined the abilities to solve problems of information, communication and knowledge in digital environments as 21st century ICT skills. Later, drawing on a systematic review of the literature, van Laar et al. (2017) broadened this conceptualization and coined the term 21st century digital skills. In essence, 21st century digital skills comprise: (1) capabilities to use digital technologies; (2) competencies to retrieve, process and manage information; (3) capacities for critical thinking, creativity, and collaboration; and (4) problem solving abilities. Sharing with the same conceptualization, Vuorikari et al. (2016) developed a DigComp Framework, specifying five domains: (1) information and data literacy, (2) communication and collaboration, (3) digital content creation, (4) safety, and (5) problem-solving. This paper adopts the DigComp Framework and describes 21st century digital skills as the competences required by individuals when using technologies to achieve particular goals.  

21st century digital skills can be measured by three methods: self-reports, performance assessments, and observational measures. Since self-reports are subject to social desirability bias, the “Learning and Assessment for Digital Citizenship” project employed the latter two methods for assessments as computer-based performance assessments and observational game-based assessment. Compared to a mono-method paradigm, a multimethod approach avoids common method variance (Campbell & Fiske, 1959). 

The project used a longitudinal design with three cohorts (Primary 3, Secondary 1, Secondary 3) of school-aged children and therefore developed a DigComp performance assessment for five grades of students: Primary 3, Primary 5, Secondary 1, Secondary 3 and Secondary 5. This computer-based performance assessment consisted of multiple-choice items and measured five domains of 21st century digital skills as: information and data literacy, communication and collaboration, digital content creation, safety, and problem-solving. Participants completed the assessment on PCs or tablets in school. This provided an authentic context of technological use that has face validity. That stated this assessment is a human-to-agent design and is thus unable to measure true collaboration in problem solving. To compensate for this limitation, the project added a task-based assessment involving human-to-human interaction developed by the ATC21S project (Griffin et al., 2015). As this assessment was designed for children above 11 years, only pairs of secondary school students participated by communicating through the online chat box to complete designated tasks. Both computer-based performance assessments bring new ways to represent and measure 21st century digital skills in daily settings and increase the time flexibility of implementation.

In addition to the DigComp assessment, primary school children participated in an observational game-based assessment on collaborative problem solving. As children are easily affected by the surrounding environment, it is essential to provide naturalistic contexts to measure their skills (Epstein et al., 2004). Hence, researchers utilise real-world scenarios to do so. Participants were placed in groups to play a game on the tablets and allowed to draft plans on paper. Researchers took field notes and video recorded their interactions. After the game ended, researchers collected participants’ paper notes and interviewed the participants. Although interpretation of observational assessment is subject to bias and resource intensity, it provides rich data on the cognitive and social development. 

Lessons learned

• This research project adapted items from existing instruments and developed new ones to assess the development of 21st century digital skills in students from Primary 3 to Secondary 3 in 2018/2019 and 2020/2021 school years. We provide five recommendations for forthcoming research. 

• First, the assessment of 21st century digital skills should be comprehensive and enable comparisons among a wide age range of students. 

• Second, longitudinal studies that enable identification of developmental trajectories of 21st century digital skills should be conducted. 

• Third, agility in the research process is important. For example, the tectonic shift to e-learning during COVID-19 pandemic led to changes in the planned measures and/or methods of data collection so that we were able to compare competencies before and during the COVID-19 pandemic. 

• Fourth, the differences between human-to-agent and human-to-human paradigms need to be given adequate attention in the design of studies and the interpretation of results therefrom. 

• Finally, attention needs to be accorded to differences in the abilities of collaborators in drawing conclusions from our findings. 

End Note: ¹ Details of this project can be found at https://www.ecitizen.hk/ 

Acknowledgement: The authors acknowledge the support given by the Research Grants Council of the HKSAR Government, #T44-707/16N, under the Theme-based Research Scheme.



Download the full handbook here: PDF.