Start With the 30,000-Foot View
Before diving deep, you need a map of the territory. The first step is to get a broad summary of the study's methodology and ask the AI to flag common areas of concern based on the research design. This helps you quickly orient yourself and decide where
to focus your critical attention. Generative AI can excel at identifying inconsistencies between a study's stated goals and its actual procedures. The Prompt: 'Act as a research methodologist. I will provide the methodology section of a published paper. Your task is to summarize the research design, procedures, and analytical strategy in five bullet points. Then, based only on this summary, list the top three potential methodological concerns a critical reviewer might raise about this type of study design.'
Interrogate the Sample Population
A study's conclusions are only as strong as the sample they're drawn from. A common flaw is a sample that doesn't accurately represent the population it claims to generalize to. Use AI to specifically probe for selection bias, inadequate sample size, or a mismatch between the sample and the research question. This forces a closer look at who was included and, just as importantly, who was left out. The Prompt: 'Analyze the 'Participants' and 'Sampling' sections of this study. Is the sample size justified? Evaluate the sampling method for potential selection bias. Based on the described sample, assess the generalizability of the findings to [insert the broader population the study claims to represent]. List any inconsistencies or weaknesses.'
Scrutinize the Control Group
In experimental research, a weak or poorly matched control group can invalidate the entire study. The control group must be comparable to the experimental group in every significant way except for the intervention being tested. An AI can act as a tireless devil's advocate, questioning the assumptions made about the control. It can flag potential confounding variables that may differ between groups. The Prompt: 'Critically evaluate the control group in this experimental design. Was the control group appropriate for the intervention being tested? Identify three potential confounding variables that could have influenced the outcomes, besides the intervention itself. Explain how these variables could present an alternative explanation for the results.'
Evaluate Statistical Methods
Statistical errors are common in published research, ranging from using the wrong test for the data to misinterpreting the results. While AI cannot replace a statistician, it can quickly spot mismatches between the type of data collected and the statistical analysis performed. This is a powerful first-pass filter for catching basic errors. The Prompt: 'The research question is [insert research question]. The data collected is [describe data, e.g., 'categorical survey responses and continuous test scores']. The authors used [insert statistical test, e.g., 'a series of t-tests']. Assess whether the chosen statistical analysis is the most appropriate method for this research question and data type. Suggest one alternative statistical approach if a potential weakness is identified.'
Uncover Unstated Limitations
Authors are expected to list their study's limitations, but they may not acknowledge every weakness. AI can compare the study's methodology against a vast database of research principles to suggest limitations the authors might have overlooked or downplayed. This helps provide a more balanced view of the study's true contribution. The Prompt: 'Read the methodology and discussion sections. The authors list [mention one or two stated limitations]. Based on the study's design, sampling, and measurements, what are two additional, unstated limitations that might affect the interpretation of the findings? Frame these as questions for a critical review.'
Check for Measurement-Conclusion Mismatches
A frequent issue is a study's conclusions overreaching what was actually measured. For instance, a study might measure a student's 'intention to use a new technology' but then conclude it has proven the factors that drive 'actual adoption'. An AI is adept at parsing language and can highlight these subtle but significant leaps in logic between the 'Methods' and 'Discussion' sections. The Prompt: 'Act as a critical logician. Compare the measurement instruments and variables described in the 'Methods' section with the main conclusions drawn in the 'Discussion' section. Identify any instances where the conclusion makes a claim about a concept that was not directly measured. List the specific measurement and the corresponding overreaching conclusion.'
