Examining a Quantitative Study Through the Lens of Mixed Methodology

When the local news reports on a piece of research, it focuses chiefly on the result, but there are many processes behind that research result that will affect its value. One of these critical processes is the selection of an appropriate research methodology. The following is an examination of the research methodology behind, The effectiveness of computer and tablet assisted intervention in early childhood students’ understanding of numbers. An empirical study conducted in Greece, by S. Papadakis, M. Kalogiannis, and N. Zaranis (2018). This examination looks at how the researcher, the research, the researched, and the reader of that research might have been affected if the authors had been guided by the research methodology analyzed by Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework by Alicia O’Cathain (2015).

Before looking at the Greek study, it is important to understand Mixed Methods Research through the writing of Alicia O’Cathain. O’Cathain is Director of the Medical Care Research Unit at The University of Sheffield in Sheffield, United Kingdom. She is a prolific writer with published works ranging in dates from 1988 to 2019. She has written extensively on research methodology. In addition, she runs mixed methods workshops and was an Associate Editor of The Journal of Mixed Methods Research from 2007 to 2012. Her current work focuses on the development and evaluation of complex interventions for chronic conditions. (Sheffield, n.d.)

In Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework (O’Cathain, 2015), O’Cathain’s discusses the difficulties of finding common ground between qualitative research assessment and quantitative research assessment, the two components of Mixed methods research. Quantitative research looks for explanations or causes using tools such as surveys, questionnaires or equipment to collect numerical data, often in a controlled environment. Qualitative research looks for understanding, meaning, or descriptions in behaviour, perception or society. Qualitative research uses the researchers themselves, immersed in the natural setting of the subjects, as tools for collecting data through interviews and observation (“Qualitative or Quantitative Research?,” n.d.). After describing the challenges of developing consistent language and criteria acceptable to proponents of both methodologies, O’Cathain lays out her framework for assessment over eight domains: planning quality, design quality, data quality, interpretive rigor, inference transferability, reporting quality, synthesizability, and utility. When followed, this framework would guide a high quality study.

The second piece of research, The effectiveness of computer and tablet assisted intervention in early childhood students’ understanding of numbers. An empirical study conducted in Greece, was conducted by a team of researchers in the Department of Preschool Education at the University of Crete (Papadakis, Kalogiannakis, & Zaranis, 2018). Briefly, they sought to understand the effect of using computer and tablet games on young children’s understanding of numbers. They applied this quantitative study to 21 kindergarten classes, finding that tablet games yielded the best test scores followed by computer games, followed by the control group receiving traditional instruction. They also concluded that the gender of the children was not an issue.

The question is, how would changing the greek research from a quantitative study to a quality mixed methods study impact the research, the researcher, the researched, and the reader of the research. Since mixed methods research is the combination of qualitative and quantitative research methodologies, the difference for the greek study is the addition of a qualitative component.

Impacts for the research itself will depend on the path changes of the researchers. For the researchers, the switch to mixed methods means additional planning and changes to the research design. They would need to expand the foundational element to include literature that looks at the application of qualitative methodologies within a kindergarten setting as well as literature to aid in the determination of what forms of data analysis to use; perhaps there is a better tool than the standardized test for measuring results. They would need to determine what observers would be looking for and what interviewers would be asking about, and then adjust their rationale. The addition of observers means more adults will be needed and this will increase costs. If interviews are added, this will increase the amount of time in the field, especially if they are choosing to interview all 365 children. This will also affect the cost.

Interviews or observations may uncover a possible compounding variable. Children might reveal that knowing that the other groups are doing something novel with computers or laptops might cause them to feel negatively about the group they are in and therefore, not enjoy it as much. Perhaps observations would reveal that the children in the computer group need better mousing skills so they do not receive as much practise as the children in tablet group. Perhaps, there is instant verbal feedback in the digital options that provides an external motivator that the control group does not receive. Possibly, the digital groups are wearing earphones that minimize the distractions; observation could tell the researchers the amount of time the children were on task. If the interviews and observations are conducted at the beginning of the study, the plan may be subject to more changes; changes affect time and money. If they are conducted at the end or concurrently, this may affect data quality.

If time and money are a concern, the researchers may choose to reduce the size of their sample which may affect the inference transferability of the research. If the extended plan goes ahead, the quantitative data may not change, but the qualitative expansion will create a lot more data that needs to be analyzed and interpreted, adding the element of meta-inferencing. The addition of meta-inferencing should be a positive change for the research because it could help set the conditions under which the improved learning would occur. This could also improve inference transferability and utility.

For the researched, the children in the study, a switch to mixed methods would mean several changes. More adults in the room can be exciting for some children but anxiety causing for others. This could lead to missed days of school. Prolonged experience with these adults could form emotional attachments for some children, as well as some of the research team. Interviews and/or observations could reveal that the children have different feelings about various adults or the other children in their groups that affect their ability to fully engage in it. Children would likely be upset at not being in the tablet group since many have tablets at home and regard it as a toy. Children in the control group would experience this even more, having had no access to either the tablet or the computer games. This might affect their ideas around fairness. Also, tablet and/or control groups could be experiencing teasing outside of the class time because they are regarded as having lower status. Interviews and observations might serve to catch these feelings and find a proactive way to mitigate them.

The reader of the research may be wondering about the possible confounding variables and the impacts the research activities have on the children. An examination of these possibilities through the addition of the qualitative component at the outset might minimize or eliminate these concerns by addressing them before the quantitative component really begins. Then the reader can feel more confident in utilizing the research to develop an educational program that improves the mathematical learning in the classroom setting. If the reader is an application designer, this would help justify the time and expense of creating relevant and marketable products. If the reader is a teacher or administrator, this could help justify the purchase of tablets for kindergarten classrooms along with the time and support to use them as effectively.

In conclusion, if the costs in time and money are plausible, the greek study would benefit from the quality Mixed Methods approach advocated by Alicia’s O’Cathain. Because the Mixed Methods approach has two components instead of one, it would seem possible that researchers could fill all the gaps in their research but they must be careful of “going down the rabbit hole” because the costs and time required could become unmanageable. If a research team feels the need to cut something out of their plan to make it feasible, let it not be the consideration of the social emotional well-being of children.

References