Please cite as: Kotilainen, S., & Bhroin, N. N. (2022): Final words on evaluation and ethics. In S. Kotilainen (Ed.), Methods in practice: Studying children and youth online (chapter 14). Retrieved DD Month YYYY, from https://core-evidence.eu/methods-toolkit/handbook-part3/handbook-final-words, doi: https://doi.org/10.21241/ssoar.83031.
Evaluation of research is about discussing the soundness as robustness of the study, especially in the case of mixed methods. Evaluation should be in mind already in the starting phase of the research since selected ways of evaluation are depending on the whole process of the study starting from the aims and questions through methods to findings. Researchers can apply different kinds of tools and techniques for evaluation of the study. Moreover, evaluation includes research ethics. It is worth thinking about that as well from the beginning of the study, in the stage of making the research plan.
Methodological case studies in this publication have been reflective descriptions on multiple topics. In that sense, authors have evaluated the mixed methods implementations mostly from the perspective of their robustness including validities of making the study. For example, Maarit Jaakkola (chapter 5 in this volume) is focusing on the “didactics of a successful individual interview”. She is highlighting strategies of exemplification, visualization and playfulness for guaranteeing the successful interview process with young children. “Playful techniques” are suggested by Lamprini Chartofylaka, Pinelopi Troullinou and Antoine Delcroix (chapter 4) as well. Laure Lu Chen and Nirmala Rao (chapter 3) highlight, among several other things, the differences “in the abilities of collaborators in drawing conclusions from our findings”. This reminds us about the users of our studies and how important it is to make findings not only robust but understandable as well. Additionally, it leads to thinking about how to make good enough.
Heidi Hartikainen, Afsaneh Razi and Pamela Wisniewski (chapter 6) highlight a valid and robust process of the study as “code for relevancy and use an iterative process for selecting keywords to search”. They identified challenges in making research with children and youth on sensitive themes, in collecting and analyzing digital data.
Authors focus on mixed methods itself in guaranteeing good research as well. For example, Michaela Lebedikova, Michal Tkaczyk, Jana Blahosova, Steriani Elavsky and David Smahel (chapter 2) describe their mixed method study combining “the self-reported data with objective smartphone data logs and screenshot collections”. Mixed methods were applied in using an app in the study focusing youth usage of smartphones from the perspective of their well-being. Even digital technologies based methods face technical challenges, still they suggest going further into machine learning based solutions in collecting and analyzing data because of making more accurate, better anonymized and real-time classified studies.
Developing ethical protocols for digital research
The contributions in the section titled as “Ethical Aspects on studying Children and Youth as users Online” present protocols considered and developed by the authors to support the design and development of ethical research. All of the authors call for contextual sensitivity and flexible approaches. Kruakae Pothong and Sonia Livingstone (chapter 11) for example were mindful of the need to limit the duration of online consultation and to allow for more spontaneous and flexible approaches to group formation in interview contexts. They also turned off the chat function during interviews and only used break-out rooms when they had the resources to moderate these. Christian Ilbury (chapter 12) reminds us of researchers' duty of care to young participants, and reports on his use of the technical affordances of social media applications, such as ‘screenshot notification’ to remind research participants of the presence of the researcher. This, Ilbury argues, is particularly relevant in contexts where young participants might reveal potentially sensitive information. Ilbury also calls for a reflective approach to sampling and analysis, and submits that children and young people could be involved in reviewing data collected about them to consider whether or not it should be included for further analysis. Veronika Kalmus et al. (chapter 13) highlight the importance of the three ‘Cs’ of planning for the research context, namely ‘Communication’ about the research with relevant parties, considering how to appropriately secure informed ‘Consent’, and considering the differences between engaging in research in physical, hybrid and digital ‘Classroom’ contexts.
Taken together, these contributions highlight some of the ethical dilemmas which arise when engaging in digital research with children and young people. The authors share the lessons they have learned, and protocols they have developed, to support securing informed consent, and facilitating meaningful participation and engagement in a range of research contexts. The contributions reflect the need to plan for engagement with children and young people in digital environments, and to consider ways to ensure meaningful and informed participation that reflect the changing experiences and expectations of research participants. In particular, the importance of considering the socio-technical contexts of these environments, including chat functions, break-out rooms and the potential for affordances such as screenshot notifications are highlighted. The need to consider and further develop digital research protocols that take account of these factors are also emphasized.
The CO:RE Compass for Research Ethics is a resource base which is designed to support researchers and students who intend to engage in research with children and young people in digital and online contexts. The short articles contained in this section will be accessible from the research base. The compass also includes blog posts, webinars and FAQs that address these and similar issues, while also linking to further resources including research ethical guidelines, reports and articles.
The CO:RE Methods Toolkit is a resource for mixed methods in studying children and youth in digital environments. This case study collection is a part of the Toolkit which in turn, is a resource for researchers and students in building sustainable research settings and conducting meaningful research with children in digital environments for increasing soundness, goodness, robustness, and practicality in research.