Cognitive Science of Medical Expertise
Objective
The goal of this multi-year research project was to synthesize cognitive science principles to reimagine the Maintenance of Certification (MOC) process for physicians. We aimed to transition the field from a model of "summative assessment" (one-time testing) to a "longitudinal learning" framework that supports the lifelong acquisition and retention of specialized medical knowledge.
Background
Medical expertise is not static. While physicians develop immense skill through training, cognitive science suggests that professional knowledge is susceptible to decay and bias over time. Physicians rely on a dual-process system: "System 1" (rapid, experience-based heuristics) and "System 2" (analytical deduction). While experience builds powerful pattern recognition, it can also lead to idiosyncratic knowledge gaps or overconfidence. Because human self-assessment is flawed, relying on personal perceptions is insufficient to maintain expertise. We explored how the "Testing Effect" (retrieval practice) can be leveraged as a pedagogical tool to prevent this decline.
My role and responsibilities
As a graduate student at the University of Pittsburgh, I:
• Conducted extensive literature reviews and evidence synthesis across cognitive psychology and medical education.
• Collaborated with a multi-disciplinary team to translate theoretical principles (e.g., spacing effects, metacognitive illusions) into actionable policy recommendations for medical boards.
• Led bi-weekly meetings with our multi-disciplinary team.
• Co-authored a comprehensive suite of five publications and a white paper presented for the American Board of Medical Specialties (ABMS).
• Led design and creation of overall model to represent our empirical framework.
• Collaborated with a multi-disciplinary team to translate theoretical principles (e.g., spacing effects, metacognitive illusions) into actionable policy recommendations for medical boards.
• Led bi-weekly meetings with our multi-disciplinary team.
• Co-authored a comprehensive suite of five publications and a white paper presented for the American Board of Medical Specialties (ABMS).
• Led design and creation of overall model to represent our empirical framework.
Methods and Approach
We used a theory-driven synthesis approach to evaluate how longitudinal assessment frameworks could replace traditional high-stakes testing.
1. Dual-Process Analysis: Examining how System 1 and System 2 reasoning interact during physician diagnosis and treatment.
2. Moderator Review: Evaluating the technical variables of testing—such as timing, frequency, and feedback format—that optimize the retention of complex information.
3. Metacognitive Evaluation: Analyzing the strengths and weaknesses of physician self-assessment and the role of heuristic-driven overconfidence.
4. Motivational Framework: Applying expectancy-value theory to understand the perceived costs and benefits of continuous professional assessment.
2. Moderator Review: Evaluating the technical variables of testing—such as timing, frequency, and feedback format—that optimize the retention of complex information.
3. Metacognitive Evaluation: Analyzing the strengths and weaknesses of physician self-assessment and the role of heuristic-driven overconfidence.
4. Motivational Framework: Applying expectancy-value theory to understand the perceived costs and benefits of continuous professional assessment.
Figure 1. Synthesis of topics influencing quality of knowledge and expertise. This framework illustrates the transition from high-stakes, decennial testing to a continuous retrieval-based system. By shifting the focus to frequent, low-stakes "testing as learning," the system leverages the testing effect and spaced repetition to maintain skill levels above the threshold of clinical competence.
Findings
Our synthesis provided converging evidence that physicians further out from training tend to perform worse on knowledge tests unless they engage in deliberate maintenance. We found that the act of retrieval (testing) is a powerful learning event that strengthens memory better than passive restudying. Furthermore, we identified that while feedback is critical, its effectiveness depends on the physician’s mindset and the perceived value of the assessment. Our research successfully provided a cognitive foundation for the "Longitudinal Assessment" shift now being adopted by the 24 Member Boards of the American Board of Medical Specialties.
Significance
This work has direct implications for the future of professional certification and patient safety. By providing a cognitive blueprint for longitudinal assessment, we helped shift the national conversation from "punitive testing" to "supported learning." This research ensures that the systems used to certify physicians are architected based on how the human brain actually retains expertise, ultimately fostering a more resilient and current medical workforce.
Collaborators
Scott Fraundorf (co-principal investigator)
Benjamin Rottman (co-principal investigator, academic advisor)
Timothy Nokes-Malach (investigator)
Associated Publications
Rottman, B.M.†, Caddick, Z.A.†, Nokes-Malach, T.J., Fraundorf, S.H. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: I. Reimagining Maintenance of Certification to Promote Lifelong Learning. Cognitive Research: Principles and Implications, 8(46). doi:10.1186/s41235-023-00496-9
Caddick, Z.A.†, Fraundorf, S.H.†, Rottman, B.M.†, Nokes-Malach, T.J. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: II. Acquiring, Maintaining, and Updating Cognitive Skills. Cognitive Research: Principles and Implications, 8(47). doi:10.1186/s41235-023-00497-8
Fraundorf, S.H., Caddick, Z.A., Nokes-Malach, T.J., Rottman, B.M. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: III. Strengths and Weaknesses of Self-Assessment. Cognitive Research: Principles and Implications, 8(58). doi:10.1186/s41235-023-00511-z
Fraundorf, S.H., Caddick, Z.A., Nokes-Malach, T.J., Rottman, B.M. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: IV. Best Practices and Open Questions in Using Testing to Enhance Learning and Retention. Cognitive Research: Principles and Implications, 8(53). doi:10.1186/s41235-023-00508-8
Nokes-Malach, T.J., Fraundorf, S.H., Caddick, Z.A., Rottman, B.M. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: V. Using a motivational framework to understand the benefits and costs of testing. Cognitive Research: Principles & Implications, 8(64). doi:10.1186/s41235-023-00518-6
Fraundorf, S.H., Caddick, Z., Rottman, B., Nokes-Malach, T., Swanson, D.B., Bazemore, A., O’Neill, T., Lipner, R.S. (2022). Conceptual foundations for designing continuing certification assessments for physicians. American Board of Medical Specialties.
Note: † denotes shared first authorship.
Caddick, Z.A.†, Fraundorf, S.H.†, Rottman, B.M.†, Nokes-Malach, T.J. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: II. Acquiring, Maintaining, and Updating Cognitive Skills. Cognitive Research: Principles and Implications, 8(47). doi:10.1186/s41235-023-00497-8
Fraundorf, S.H., Caddick, Z.A., Nokes-Malach, T.J., Rottman, B.M. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: III. Strengths and Weaknesses of Self-Assessment. Cognitive Research: Principles and Implications, 8(58). doi:10.1186/s41235-023-00511-z
Fraundorf, S.H., Caddick, Z.A., Nokes-Malach, T.J., Rottman, B.M. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: IV. Best Practices and Open Questions in Using Testing to Enhance Learning and Retention. Cognitive Research: Principles and Implications, 8(53). doi:10.1186/s41235-023-00508-8
Nokes-Malach, T.J., Fraundorf, S.H., Caddick, Z.A., Rottman, B.M. (2023). Cognitive Perspectives on Maintaining Physicians’ Medical Expertise: V. Using a motivational framework to understand the benefits and costs of testing. Cognitive Research: Principles & Implications, 8(64). doi:10.1186/s41235-023-00518-6
Fraundorf, S.H., Caddick, Z., Rottman, B., Nokes-Malach, T., Swanson, D.B., Bazemore, A., O’Neill, T., Lipner, R.S. (2022). Conceptual foundations for designing continuing certification assessments for physicians. American Board of Medical Specialties.
Note: † denotes shared first authorship.
Tools and Software Used
Google Suite (GDrive, shared documents), Excel, various scientific search engines (e.g., Google Scholar, PubMed, PsycInfo).
Resources
1 A consolidated PDF of all 5 articles can be downloaded here.
2 Alternatively, you can download each paper individually here: one, two, three, four, five.