SlideShare a Scribd company logo
1 of 25
Download to read offline
Blueprinting Project:
    Update and Next Steps
              Elaine Rodeck, Ph.D.
      Examination Strategist, Blueprint Lead
            Medical Council of Canada
    Annual General Meeting, September 30, 2012




1
Overview
    1.   Blueprinting Project - Purpose, Timeline
    2.   Delphi Process
    3.   National Survey
    4.   Next Steps
    5.   Q&A




2
Historical Perspective
    • 2009
      – Medical Council of Canada
        (MCC) Objectives Committee
        revised objectives using
        CanMEDS as the framework
    • October 2011
      – Report of the Assessment Review Task Force (ARTF)
        approved by Council



                                                            3
3
ARTF Recommendation 2
    • The content of the MCC assessment processes shall
      be expanded by:
       – Defining the knowledge and behaviours in the CanMEDS
         roles that demonstrates competence of a physician about to
         enter independent practice
       – Reviewing the adequacy of the content and skills coverage
         on the blueprints for all MCC examinations
       – Revising the examination blueprints and reporting systems
       – Determining whether any core competencies cannot be
         tested by the MCC examinations

                                                                 4
4
MCC Validity Research Agenda




                                   5
5
Why do a blueprint exercise?
     The exercise is designed:
      …to define and validate the critical
         competencies that the MCC examinations
         should sample
      in order:
       …to assure Medical Regulatory Authorities
         (MRAs) and stakeholders that a physician
         has the requisite knowledge, skills and
         professional behaviours to enter
         supervised/unsupervised clinical practice
                                                     6
6
Who is participating in the process?
    •   Blueprinting Core Project Team
    •   Governance Board
    •   Steering Group
    •   Process Participants
        – Subject matter experts (SMEs) for the Delphi questionnaire and
          follow-up review by SMEs
        – National Survey participants – a cross section of health providers,
          teachers, learners, and members of the public
        – SMEs blueprint panelists
        – MCC’s Central Examination Committee


                                                                                7
7
How is this being done?
      • Phase 1
        – January 2012 to fall 2013
        – Information gathering
           •   Delphi Group/Follow-up with SMEs
           •   Incidence and Prevalence paper
           •   National Survey
           •   Pilot Project on resident supervision
           •   Medical Education Assessment Advisory Committee paper
        – Establish recommendations for the new
          MCCQE test specifications

                                                                       8
8
Blueprinting Project…
     – Phase I
         • Approved blueprints (test specifications) for the revisions to the MCCQE Parts I and II
         • Identification of requirements not measurable in formats such as MCCQE Part I or Part II

     – Phase II - operational transition
         •    Gap analysis of the existing item banks
         •    Content and form development, infrastructure, workflow design
         •    Pilot testing
         •    Communication Plan for all stakeholders
         •    Develop process to consider new types of assessment

     – Phase III - the launch
         • New examinations and content
         • Monitoring for ongoing content renewal.

             Consultation with stakeholders throughout the process is key to ensuring outcomes
                                   are the result of a collaborative process!


                                                                                                 9
9
What have we done?
      • Incidence and Prevalence paper
      • Delphi process of CanMEDS roles and
        MCC objectives
      • Follow-up review by experts who have
        content knowledge



                                               10
10
Blueprinting Project:
        Delphi Process




11
The Delphi Process…
     •   Tool for consensus measurement
     •   Determines the extent of agreement amongst a group of experts*
         •   Anonymity
               – to prevent dominance of the process by one or a few experts
         •   Iteration
               – several rounds are conducted to allow individuals to change opinions
         •   Controlled feedback
               – provide feedback on individual and group responses
         •   Statistical group response
               – synthesis and analysis of the group response

         *Jones, J & Hunter D (1995) Consensus methods for medical and health services research.
             BMJ 311: 376-80.


                                                                                             12
12
The Delphi Process…
     The Panelists
     • 22 panelists…pan-Canadian medical school
       representation including Quebec, medical disciplines,
       registrar representation, gender, etc.

     The Questionnaire
     • Medical expert objectives – Over 200 statements
     • Non-medical expert objectives – including…professional,
       communicator, scholar, manager, collaborator, advocate
     • Results Review – subsequent review by SMEs

                                                                 13
13
The Delphi Process…
     Rate objectives for entry
     into unsupervised practice.
     1.Unnecessary
     2.Not Important
     3.Slightly Important
     4.Important
     5.Very Important
     6.Essential
     7.Cannot Rate

                                   14
14
Example - Delphi Process Question
     What is your opinion?                                  1.         Unnecessary
                                                            2.         Not Important
     Glucose Abnormal, Serum/Diabetes                       3.         Slightly Important
     Mellitus/Polydipsia                                    4.         Important

     Diabetes mellitus                                      5.
                                                            6.
                                                                       Very Important
                                                                       Essential
     Rationale: Diabetes mellitus is an increasingly        7.         Cannot Rate
     common multi-system disease associated with
     a relative or absolute impairment of insulin                25%              25%   25%      25%

     secretion together with varying degrees of
     peripheral resistance to the action of insulin.

     Key Objectives: Given a patient with diabetes
     mellitus, the candidate will diagnose the cause,
     severity and complications, and will initiate an
     appropriate management plan. Particularly
     important are early detection of the disease,
     and recognition of medical emergencies such
     as acute hypoglycemia, diabetic ketoacidosis,
     and hyperosmolar nonketotic coma.                  0%             0%   0%
                                                                                                15
15                                                      1        2     3     4     5        6        7
Example - Delphi Process Question
     What is your opinion?                                 1.         Unnecessary
                                                           2.         Not Important
     Hair and Nail Complaints                              3.         Slightly Important
     Nail complaints                                       4.         Important
                                                           5.         Very Important
     Rationale: Nail disorders are common                  6.         Essential
     conditions. Although in themselves nail               7.         Cannot Rate
     changes may be innocuous, they may indicate
     underlying disease.                                        25%              25%   25%      25%




     Key Objectives: Given a patient with nail
     abnormalities, the candidate will diagnose the
     cause (local or systemic), severity, and
     complications, and will initiate an appropriate
     management plan.




                                                       0%             0%   0%
                                                                                               16
16                                                     1        2     3     4     5        6        7
Example - Delphi Process Question
     What is your opinion?                                  1.         Unnecessary
                                                            2.         Not Important
     Abdominal Pain                                         3.         Slightly Important
     Abdominal pain, Acute                                  4.         Important
                                                            5.         Very Important
     Rationale: Acute abdominal pain is a common            6.         Essential
     complaint in adults, leading to frequent               7.         Cannot Rate
     physician visits both in the Emergency
     Department and office setting. Acute                        25%              25%   25%      25%

     abdominal pain may result from serious intra-
     abdominal, intrathoracic, or retroperitoneal
     processes.

     Key Objectives: Given a patient with acute
     abdominal pain, the candidate will diagnose
     the cause, severity, and complications, and will
     initiate an appropriate management plan. In
     particular, the candidate will identify those
     patients requiring emergency medical or
     surgical treatment.                                0%             0%   0%
                                                                                                17
17                                                      1        2     3     4     5        6        7
Example - Delphi Process Question
     What is your opinion?                                  1.Unnecessary
                                                            2.Not Important
     Hair and Nail Complaints                               3.Slightly Important
     Alopecia                                               4.Important
                                                            5.Very Important
     Rationale: Alopecia may be physiological or            6.Essential
     due to local scalp disease or underlying               7.Cannot Rate
     systemic disease and can result in
     psychological distress.                                    25%                25%   25%   25%




     Key Objectives: Given a patient with alopecia,
     the candidate will diagnose the cause, severity,
     and complications, and will initiate an
     appropriate management plan.




                                                        0%            0%    0%
                                                                                              18
18                                                      1       2     3     4       5     6        7
Example - Delphi Process Question
     What is your opinion?                                1.         Unnecessary
                                                          2.         Not Important
     Population Health                                    3.         Slightly Important
     Assessing and measuring health                       4.         Important
                                                          5.         Very Important
     status at the Population Level                       6.         Essential
     Rationale: Knowing the health status of the          7.         Cannot Rate
     population allows for better planning and
                                                               25%              25%   25%      25%
     evaluation of health programs and tailoring
     interventions to meet patient/community
     needs. Physicians are also active participants
     in disease surveillance programs, encouraging
     them to address health needs in the population
     and not merely health demands.
     Key Objectives: Describe the health status of
     a defined population. Measure and record the
     factors that affect the health status of a
     population with respect to the principles of
     causation.                                       0%             0%   0%
                                                                                              19
19                                                    1        2     3     4     5        6        7
Blueprinting Project:
       National Survey




20
National Survey
       • Sampling plan in process – fall 2012
         – Physicians, Residents, Students, Program
           Directors, Recently certified CFPC/RCPSC,
           Nurses, Pharmacists, MRAs, public members
       • Survey timing – early February 2013
       • SME panel review – May 2013
       • CEC review – June 2013

                                                       21
21
Blueprinting Project:
         Next Steps




22
Next Steps
     • 2012 Fall
        – Develop national survey questions
        – CEC to approve national survey
     • 2013
        – February – National survey
        – March /April – Analyze data, prepare for SME panel meetings
        – May – SME panel meetings (two)
        – June – CEC to review draft blueprints
        – July – Stakeholder invitation to comment
        – August/September – Revise blueprints to include stakeholder
          comments
        – AGM – Present revised blueprints to AGM for consultation
        – Late fall – Blueprints to be revised and approved by CEC
        – Late 2013/early 2014 – Executive Board to approve blueprints

                                                                         23
23
Blueprinting Project:
            Q&A




24
Questions, Comments & Discussion
             Elaine Rodeck
            ERodeck@mcc.ca

               Thank you!


                                        25
25

More Related Content

Viewers also liked

Yayasan MENDAKI Seminar for Parents
Yayasan MENDAKI Seminar for ParentsYayasan MENDAKI Seminar for Parents
Yayasan MENDAKI Seminar for ParentsJimmy Keng
 
Rethinking On-Site Clinics
Rethinking On-Site ClinicsRethinking On-Site Clinics
Rethinking On-Site Clinicsweatrust
 
Medical education 2013
Medical education 2013Medical education 2013
Medical education 2013tbf413
 
Medical education in india-current status
Medical education in india-current statusMedical education in india-current status
Medical education in india-current statusMitasha Singh
 
Developing short answer questions (sa qs)
Developing short answer questions (sa qs)Developing short answer questions (sa qs)
Developing short answer questions (sa qs)Javed Iqbal
 
Short Answer Presentation
Short Answer PresentationShort Answer Presentation
Short Answer PresentationMiss l
 
Tips For Constructing Objective Written Exam Questions
Tips For Constructing Objective Written Exam QuestionsTips For Constructing Objective Written Exam Questions
Tips For Constructing Objective Written Exam QuestionsSoha Rashed
 
Concept of integrated education
Concept of integrated educationConcept of integrated education
Concept of integrated educationAmruta_Apte
 
Integrative teaching strategy
Integrative teaching strategyIntegrative teaching strategy
Integrative teaching strategyJerwin Patiga
 
Class 10 Cbse Social Science Sample Paper Model 1
Class 10 Cbse Social Science Sample Paper Model 1Class 10 Cbse Social Science Sample Paper Model 1
Class 10 Cbse Social Science Sample Paper Model 1Sunaina Rawat
 
Blueprint of exam questions
Blueprint of exam questions Blueprint of exam questions
Blueprint of exam questions Dr.Shahid Hassan
 
Assessment of Learning - Multiple Choice Test
Assessment of Learning - Multiple Choice TestAssessment of Learning - Multiple Choice Test
Assessment of Learning - Multiple Choice TestXiTian Miran
 
Blueprint in education
Blueprint in educationBlueprint in education
Blueprint in educationKiran Kushwaha
 
MCQ test item analysis
MCQ test item analysisMCQ test item analysis
MCQ test item analysisSoha Rashed
 
Validity, reliability & practicality
Validity, reliability & practicalityValidity, reliability & practicality
Validity, reliability & practicalitySamcruz5
 

Viewers also liked (19)

Yayasan MENDAKI Seminar for Parents
Yayasan MENDAKI Seminar for ParentsYayasan MENDAKI Seminar for Parents
Yayasan MENDAKI Seminar for Parents
 
Rethinking On-Site Clinics
Rethinking On-Site ClinicsRethinking On-Site Clinics
Rethinking On-Site Clinics
 
Medical education in India
Medical education in IndiaMedical education in India
Medical education in India
 
Medical education 2013
Medical education 2013Medical education 2013
Medical education 2013
 
Medical education in india-current status
Medical education in india-current statusMedical education in india-current status
Medical education in india-current status
 
Developing short answer questions (sa qs)
Developing short answer questions (sa qs)Developing short answer questions (sa qs)
Developing short answer questions (sa qs)
 
Short Answer Presentation
Short Answer PresentationShort Answer Presentation
Short Answer Presentation
 
Tips For Constructing Objective Written Exam Questions
Tips For Constructing Objective Written Exam QuestionsTips For Constructing Objective Written Exam Questions
Tips For Constructing Objective Written Exam Questions
 
Concept of integrated education
Concept of integrated educationConcept of integrated education
Concept of integrated education
 
Integrative teaching strategy
Integrative teaching strategyIntegrative teaching strategy
Integrative teaching strategy
 
Integrated teaching
Integrated teachingIntegrated teaching
Integrated teaching
 
Class 10 Cbse Social Science Sample Paper Model 1
Class 10 Cbse Social Science Sample Paper Model 1Class 10 Cbse Social Science Sample Paper Model 1
Class 10 Cbse Social Science Sample Paper Model 1
 
Blueprint ppt
Blueprint pptBlueprint ppt
Blueprint ppt
 
Thematic teaching
Thematic teachingThematic teaching
Thematic teaching
 
Blueprint of exam questions
Blueprint of exam questions Blueprint of exam questions
Blueprint of exam questions
 
Assessment of Learning - Multiple Choice Test
Assessment of Learning - Multiple Choice TestAssessment of Learning - Multiple Choice Test
Assessment of Learning - Multiple Choice Test
 
Blueprint in education
Blueprint in educationBlueprint in education
Blueprint in education
 
MCQ test item analysis
MCQ test item analysisMCQ test item analysis
MCQ test item analysis
 
Validity, reliability & practicality
Validity, reliability & practicalityValidity, reliability & practicality
Validity, reliability & practicality
 

Similar to Blueprinting 2012

Fit for Purpose: Developing Business Cases for New Services in Research Libr...
 Fit for Purpose: Developing Business Cases for New Services in Research Libr... Fit for Purpose: Developing Business Cases for New Services in Research Libr...
Fit for Purpose: Developing Business Cases for New Services in Research Libr...Mike Furlough
 
ESI Supplemental 3 Slides, Fit for Purpose
ESI Supplemental 3 Slides, Fit for PurposeESI Supplemental 3 Slides, Fit for Purpose
ESI Supplemental 3 Slides, Fit for PurposeDuraSpace
 
Critical Path Management
Critical Path ManagementCritical Path Management
Critical Path ManagementThierryFerreira
 
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...Daniel McLinden
 
Charleston conference 2011 business cases for new service development in rese...
Charleston conference 2011 business cases for new service development in rese...Charleston conference 2011 business cases for new service development in rese...
Charleston conference 2011 business cases for new service development in rese...cfhunter
 
Senior leadership lean_transformation
Senior leadership lean_transformationSenior leadership lean_transformation
Senior leadership lean_transformationMario Charlin
 
VC M&E Module 11 - Evaluations and Value Chain Projects
VC M&E Module 11 - Evaluations and Value Chain ProjectsVC M&E Module 11 - Evaluations and Value Chain Projects
VC M&E Module 11 - Evaluations and Value Chain ProjectsCARE Economic Development Unit
 
BEST PRACTICE: Identification, Documentation, and Confirmation
BEST PRACTICE: Identification, Documentation, and ConfirmationBEST PRACTICE: Identification, Documentation, and Confirmation
BEST PRACTICE: Identification, Documentation, and Confirmationzorengubalane
 
Launch & Grow a Successful Simulation Program
Launch & Grow a Successful Simulation ProgramLaunch & Grow a Successful Simulation Program
Launch & Grow a Successful Simulation ProgramSIMUL8 Corporation
 
MiPCT Webinar 03/21/2012
MiPCT Webinar 03/21/2012MiPCT Webinar 03/21/2012
MiPCT Webinar 03/21/2012mednetone
 
Quality Gurus (TOTAL QUALITY MANGEMENT SYSTEM) Food Technology
Quality Gurus (TOTAL QUALITY MANGEMENT SYSTEM) Food TechnologyQuality Gurus (TOTAL QUALITY MANGEMENT SYSTEM) Food Technology
Quality Gurus (TOTAL QUALITY MANGEMENT SYSTEM) Food TechnologyZargiya
 
How to improve your time to market by moving to Agile with good governance (K...
How to improve your time to market by moving to Agile with good governance (K...How to improve your time to market by moving to Agile with good governance (K...
How to improve your time to market by moving to Agile with good governance (K...APMG-International Showcase UK
 
Multi Media Lab Training
Multi Media Lab TrainingMulti Media Lab Training
Multi Media Lab TrainingDIv CHAS
 
CMC Project Management
CMC Project ManagementCMC Project Management
CMC Project ManagementXinsuit
 

Similar to Blueprinting 2012 (20)

Fit for Purpose: Developing Business Cases for New Services in Research Libr...
 Fit for Purpose: Developing Business Cases for New Services in Research Libr... Fit for Purpose: Developing Business Cases for New Services in Research Libr...
Fit for Purpose: Developing Business Cases for New Services in Research Libr...
 
ESI Supplemental 3 Slides, Fit for Purpose
ESI Supplemental 3 Slides, Fit for PurposeESI Supplemental 3 Slides, Fit for Purpose
ESI Supplemental 3 Slides, Fit for Purpose
 
Critical Path Management
Critical Path ManagementCritical Path Management
Critical Path Management
 
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
 
model-of-human-occ-and-prac-dev-LLL event-London Region-heasman.pdf
model-of-human-occ-and-prac-dev-LLL event-London Region-heasman.pdfmodel-of-human-occ-and-prac-dev-LLL event-London Region-heasman.pdf
model-of-human-occ-and-prac-dev-LLL event-London Region-heasman.pdf
 
Project Outcomes
Project OutcomesProject Outcomes
Project Outcomes
 
Charleston conference 2011 business cases for new service development in rese...
Charleston conference 2011 business cases for new service development in rese...Charleston conference 2011 business cases for new service development in rese...
Charleston conference 2011 business cases for new service development in rese...
 
Senior leadership lean_transformation
Senior leadership lean_transformationSenior leadership lean_transformation
Senior leadership lean_transformation
 
Ch01
Ch01Ch01
Ch01
 
VC M&E Module 11 - Evaluations and Value Chain Projects
VC M&E Module 11 - Evaluations and Value Chain ProjectsVC M&E Module 11 - Evaluations and Value Chain Projects
VC M&E Module 11 - Evaluations and Value Chain Projects
 
BEST PRACTICE: Identification, Documentation, and Confirmation
BEST PRACTICE: Identification, Documentation, and ConfirmationBEST PRACTICE: Identification, Documentation, and Confirmation
BEST PRACTICE: Identification, Documentation, and Confirmation
 
Launch & Grow a Successful Simulation Program
Launch & Grow a Successful Simulation ProgramLaunch & Grow a Successful Simulation Program
Launch & Grow a Successful Simulation Program
 
MiPCT Webinar 03/21/2012
MiPCT Webinar 03/21/2012MiPCT Webinar 03/21/2012
MiPCT Webinar 03/21/2012
 
Quality Gurus (TOTAL QUALITY MANGEMENT SYSTEM) Food Technology
Quality Gurus (TOTAL QUALITY MANGEMENT SYSTEM) Food TechnologyQuality Gurus (TOTAL QUALITY MANGEMENT SYSTEM) Food Technology
Quality Gurus (TOTAL QUALITY MANGEMENT SYSTEM) Food Technology
 
Response from Taskforce
Response from TaskforceResponse from Taskforce
Response from Taskforce
 
How to improve your time to market by moving to Agile with good governance (K...
How to improve your time to market by moving to Agile with good governance (K...How to improve your time to market by moving to Agile with good governance (K...
How to improve your time to market by moving to Agile with good governance (K...
 
10 things about_evaluation_final_with_hyperlinks
10 things about_evaluation_final_with_hyperlinks10 things about_evaluation_final_with_hyperlinks
10 things about_evaluation_final_with_hyperlinks
 
VC M&E Module 4 - Select Performance Indicators
VC M&E Module 4 - Select Performance IndicatorsVC M&E Module 4 - Select Performance Indicators
VC M&E Module 4 - Select Performance Indicators
 
Multi Media Lab Training
Multi Media Lab TrainingMulti Media Lab Training
Multi Media Lab Training
 
CMC Project Management
CMC Project ManagementCMC Project Management
CMC Project Management
 

More from MedCouncilCan

Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
Peering through the Looking Glass: Towards a Programmatic View of the Qualify...Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
Peering through the Looking Glass: Towards a Programmatic View of the Qualify...MedCouncilCan
 
A journey towards programmatic assessment
A journey towards programmatic assessmentA journey towards programmatic assessment
A journey towards programmatic assessmentMedCouncilCan
 
Pushing the Boundaries of Medical Licensing
Pushing the Boundaries of Medical Licensing Pushing the Boundaries of Medical Licensing
Pushing the Boundaries of Medical Licensing MedCouncilCan
 
Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...MedCouncilCan
 
Séance d'orientation pour les candidats de l'examen de la CNE 2015
Séance d'orientation pour les candidats de l'examen de la CNE 2015Séance d'orientation pour les candidats de l'examen de la CNE 2015
Séance d'orientation pour les candidats de l'examen de la CNE 2015MedCouncilCan
 
2015 NAC candidate orientation presentation
2015 NAC candidate orientation presentation2015 NAC candidate orientation presentation
2015 NAC candidate orientation presentationMedCouncilCan
 
Surveying the landscape: An overview of tools for direct observation and asse...
Surveying the landscape: An overview of tools for direct observation and asse...Surveying the landscape: An overview of tools for direct observation and asse...
Surveying the landscape: An overview of tools for direct observation and asse...MedCouncilCan
 
Moving Towards Programmatic Assessment
Moving Towards Programmatic AssessmentMoving Towards Programmatic Assessment
Moving Towards Programmatic AssessmentMedCouncilCan
 
The New Blueprint: challenging the comfort zone
The New Blueprint: challenging the comfort zoneThe New Blueprint: challenging the comfort zone
The New Blueprint: challenging the comfort zoneMedCouncilCan
 
Séance d'orientation pour les candidats de l'examen de la CNE de 2014
Séance d'orientation pour les candidats de l'examen de la CNE de 2014Séance d'orientation pour les candidats de l'examen de la CNE de 2014
Séance d'orientation pour les candidats de l'examen de la CNE de 2014MedCouncilCan
 
2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentationMedCouncilCan
 
NAC PRA update - 2014 Ottawa Conference
NAC PRA update - 2014 Ottawa ConferenceNAC PRA update - 2014 Ottawa Conference
NAC PRA update - 2014 Ottawa ConferenceMedCouncilCan
 
Blueprinting update - 2014 Ottawa Conference
Blueprinting update - 2014 Ottawa ConferenceBlueprinting update - 2014 Ottawa Conference
Blueprinting update - 2014 Ottawa ConferenceMedCouncilCan
 
Orientation à l’intention des candidats au format ECOS (2014)
Orientation à l’intention des candidats au format ECOS (2014)Orientation à l’intention des candidats au format ECOS (2014)
Orientation à l’intention des candidats au format ECOS (2014)MedCouncilCan
 
2014 Candidate Orientation Presentation - Certification Examination in Family...
2014 Candidate Orientation Presentation - Certification Examination in Family...2014 Candidate Orientation Presentation - Certification Examination in Family...
2014 Candidate Orientation Presentation - Certification Examination in Family...MedCouncilCan
 
Présentation d'orientation de 2014 - EACMC, partie II
Présentation d'orientation de 2014 - EACMC, partie IIPrésentation d'orientation de 2014 - EACMC, partie II
Présentation d'orientation de 2014 - EACMC, partie IIMedCouncilCan
 
Présentation d'orientation - Examen de la CNE 2014
Présentation d'orientation - Examen de la CNE 2014Présentation d'orientation - Examen de la CNE 2014
Présentation d'orientation - Examen de la CNE 2014MedCouncilCan
 
2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation 2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation MedCouncilCan
 
Pre-Exam Orientation for Candidates taking the Certification Examination in F...
Pre-Exam Orientation for Candidates taking the Certification Examination in F...Pre-Exam Orientation for Candidates taking the Certification Examination in F...
Pre-Exam Orientation for Candidates taking the Certification Examination in F...MedCouncilCan
 

More from MedCouncilCan (20)

Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
Peering through the Looking Glass: Towards a Programmatic View of the Qualify...Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
 
A journey towards programmatic assessment
A journey towards programmatic assessmentA journey towards programmatic assessment
A journey towards programmatic assessment
 
Pushing the Boundaries of Medical Licensing
Pushing the Boundaries of Medical Licensing Pushing the Boundaries of Medical Licensing
Pushing the Boundaries of Medical Licensing
 
Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...
 
Séance d'orientation pour les candidats de l'examen de la CNE 2015
Séance d'orientation pour les candidats de l'examen de la CNE 2015Séance d'orientation pour les candidats de l'examen de la CNE 2015
Séance d'orientation pour les candidats de l'examen de la CNE 2015
 
2015 NAC candidate orientation presentation
2015 NAC candidate orientation presentation2015 NAC candidate orientation presentation
2015 NAC candidate orientation presentation
 
Surveying the landscape: An overview of tools for direct observation and asse...
Surveying the landscape: An overview of tools for direct observation and asse...Surveying the landscape: An overview of tools for direct observation and asse...
Surveying the landscape: An overview of tools for direct observation and asse...
 
Moving Towards Programmatic Assessment
Moving Towards Programmatic AssessmentMoving Towards Programmatic Assessment
Moving Towards Programmatic Assessment
 
The New Blueprint: challenging the comfort zone
The New Blueprint: challenging the comfort zoneThe New Blueprint: challenging the comfort zone
The New Blueprint: challenging the comfort zone
 
The
The The
The
 
Séance d'orientation pour les candidats de l'examen de la CNE de 2014
Séance d'orientation pour les candidats de l'examen de la CNE de 2014Séance d'orientation pour les candidats de l'examen de la CNE de 2014
Séance d'orientation pour les candidats de l'examen de la CNE de 2014
 
2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation
 
NAC PRA update - 2014 Ottawa Conference
NAC PRA update - 2014 Ottawa ConferenceNAC PRA update - 2014 Ottawa Conference
NAC PRA update - 2014 Ottawa Conference
 
Blueprinting update - 2014 Ottawa Conference
Blueprinting update - 2014 Ottawa ConferenceBlueprinting update - 2014 Ottawa Conference
Blueprinting update - 2014 Ottawa Conference
 
Orientation à l’intention des candidats au format ECOS (2014)
Orientation à l’intention des candidats au format ECOS (2014)Orientation à l’intention des candidats au format ECOS (2014)
Orientation à l’intention des candidats au format ECOS (2014)
 
2014 Candidate Orientation Presentation - Certification Examination in Family...
2014 Candidate Orientation Presentation - Certification Examination in Family...2014 Candidate Orientation Presentation - Certification Examination in Family...
2014 Candidate Orientation Presentation - Certification Examination in Family...
 
Présentation d'orientation de 2014 - EACMC, partie II
Présentation d'orientation de 2014 - EACMC, partie IIPrésentation d'orientation de 2014 - EACMC, partie II
Présentation d'orientation de 2014 - EACMC, partie II
 
Présentation d'orientation - Examen de la CNE 2014
Présentation d'orientation - Examen de la CNE 2014Présentation d'orientation - Examen de la CNE 2014
Présentation d'orientation - Examen de la CNE 2014
 
2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation 2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation
 
Pre-Exam Orientation for Candidates taking the Certification Examination in F...
Pre-Exam Orientation for Candidates taking the Certification Examination in F...Pre-Exam Orientation for Candidates taking the Certification Examination in F...
Pre-Exam Orientation for Candidates taking the Certification Examination in F...
 

Blueprinting 2012

  • 1. Blueprinting Project: Update and Next Steps Elaine Rodeck, Ph.D. Examination Strategist, Blueprint Lead Medical Council of Canada Annual General Meeting, September 30, 2012 1
  • 2. Overview 1. Blueprinting Project - Purpose, Timeline 2. Delphi Process 3. National Survey 4. Next Steps 5. Q&A 2
  • 3. Historical Perspective • 2009 – Medical Council of Canada (MCC) Objectives Committee revised objectives using CanMEDS as the framework • October 2011 – Report of the Assessment Review Task Force (ARTF) approved by Council 3 3
  • 4. ARTF Recommendation 2 • The content of the MCC assessment processes shall be expanded by: – Defining the knowledge and behaviours in the CanMEDS roles that demonstrates competence of a physician about to enter independent practice – Reviewing the adequacy of the content and skills coverage on the blueprints for all MCC examinations – Revising the examination blueprints and reporting systems – Determining whether any core competencies cannot be tested by the MCC examinations 4 4
  • 6. Why do a blueprint exercise? The exercise is designed: …to define and validate the critical competencies that the MCC examinations should sample in order: …to assure Medical Regulatory Authorities (MRAs) and stakeholders that a physician has the requisite knowledge, skills and professional behaviours to enter supervised/unsupervised clinical practice 6 6
  • 7. Who is participating in the process? • Blueprinting Core Project Team • Governance Board • Steering Group • Process Participants – Subject matter experts (SMEs) for the Delphi questionnaire and follow-up review by SMEs – National Survey participants – a cross section of health providers, teachers, learners, and members of the public – SMEs blueprint panelists – MCC’s Central Examination Committee 7 7
  • 8. How is this being done? • Phase 1 – January 2012 to fall 2013 – Information gathering • Delphi Group/Follow-up with SMEs • Incidence and Prevalence paper • National Survey • Pilot Project on resident supervision • Medical Education Assessment Advisory Committee paper – Establish recommendations for the new MCCQE test specifications 8 8
  • 9. Blueprinting Project… – Phase I • Approved blueprints (test specifications) for the revisions to the MCCQE Parts I and II • Identification of requirements not measurable in formats such as MCCQE Part I or Part II – Phase II - operational transition • Gap analysis of the existing item banks • Content and form development, infrastructure, workflow design • Pilot testing • Communication Plan for all stakeholders • Develop process to consider new types of assessment – Phase III - the launch • New examinations and content • Monitoring for ongoing content renewal. Consultation with stakeholders throughout the process is key to ensuring outcomes are the result of a collaborative process! 9 9
  • 10. What have we done? • Incidence and Prevalence paper • Delphi process of CanMEDS roles and MCC objectives • Follow-up review by experts who have content knowledge 10 10
  • 11. Blueprinting Project: Delphi Process 11
  • 12. The Delphi Process… • Tool for consensus measurement • Determines the extent of agreement amongst a group of experts* • Anonymity – to prevent dominance of the process by one or a few experts • Iteration – several rounds are conducted to allow individuals to change opinions • Controlled feedback – provide feedback on individual and group responses • Statistical group response – synthesis and analysis of the group response *Jones, J & Hunter D (1995) Consensus methods for medical and health services research. BMJ 311: 376-80. 12 12
  • 13. The Delphi Process… The Panelists • 22 panelists…pan-Canadian medical school representation including Quebec, medical disciplines, registrar representation, gender, etc. The Questionnaire • Medical expert objectives – Over 200 statements • Non-medical expert objectives – including…professional, communicator, scholar, manager, collaborator, advocate • Results Review – subsequent review by SMEs 13 13
  • 14. The Delphi Process… Rate objectives for entry into unsupervised practice. 1.Unnecessary 2.Not Important 3.Slightly Important 4.Important 5.Very Important 6.Essential 7.Cannot Rate 14 14
  • 15. Example - Delphi Process Question What is your opinion? 1. Unnecessary 2. Not Important Glucose Abnormal, Serum/Diabetes 3. Slightly Important Mellitus/Polydipsia 4. Important Diabetes mellitus 5. 6. Very Important Essential Rationale: Diabetes mellitus is an increasingly 7. Cannot Rate common multi-system disease associated with a relative or absolute impairment of insulin 25% 25% 25% 25% secretion together with varying degrees of peripheral resistance to the action of insulin. Key Objectives: Given a patient with diabetes mellitus, the candidate will diagnose the cause, severity and complications, and will initiate an appropriate management plan. Particularly important are early detection of the disease, and recognition of medical emergencies such as acute hypoglycemia, diabetic ketoacidosis, and hyperosmolar nonketotic coma. 0% 0% 0% 15 15 1 2 3 4 5 6 7
  • 16. Example - Delphi Process Question What is your opinion? 1. Unnecessary 2. Not Important Hair and Nail Complaints 3. Slightly Important Nail complaints 4. Important 5. Very Important Rationale: Nail disorders are common 6. Essential conditions. Although in themselves nail 7. Cannot Rate changes may be innocuous, they may indicate underlying disease. 25% 25% 25% 25% Key Objectives: Given a patient with nail abnormalities, the candidate will diagnose the cause (local or systemic), severity, and complications, and will initiate an appropriate management plan. 0% 0% 0% 16 16 1 2 3 4 5 6 7
  • 17. Example - Delphi Process Question What is your opinion? 1. Unnecessary 2. Not Important Abdominal Pain 3. Slightly Important Abdominal pain, Acute 4. Important 5. Very Important Rationale: Acute abdominal pain is a common 6. Essential complaint in adults, leading to frequent 7. Cannot Rate physician visits both in the Emergency Department and office setting. Acute 25% 25% 25% 25% abdominal pain may result from serious intra- abdominal, intrathoracic, or retroperitoneal processes. Key Objectives: Given a patient with acute abdominal pain, the candidate will diagnose the cause, severity, and complications, and will initiate an appropriate management plan. In particular, the candidate will identify those patients requiring emergency medical or surgical treatment. 0% 0% 0% 17 17 1 2 3 4 5 6 7
  • 18. Example - Delphi Process Question What is your opinion? 1.Unnecessary 2.Not Important Hair and Nail Complaints 3.Slightly Important Alopecia 4.Important 5.Very Important Rationale: Alopecia may be physiological or 6.Essential due to local scalp disease or underlying 7.Cannot Rate systemic disease and can result in psychological distress. 25% 25% 25% 25% Key Objectives: Given a patient with alopecia, the candidate will diagnose the cause, severity, and complications, and will initiate an appropriate management plan. 0% 0% 0% 18 18 1 2 3 4 5 6 7
  • 19. Example - Delphi Process Question What is your opinion? 1. Unnecessary 2. Not Important Population Health 3. Slightly Important Assessing and measuring health 4. Important 5. Very Important status at the Population Level 6. Essential Rationale: Knowing the health status of the 7. Cannot Rate population allows for better planning and 25% 25% 25% 25% evaluation of health programs and tailoring interventions to meet patient/community needs. Physicians are also active participants in disease surveillance programs, encouraging them to address health needs in the population and not merely health demands. Key Objectives: Describe the health status of a defined population. Measure and record the factors that affect the health status of a population with respect to the principles of causation. 0% 0% 0% 19 19 1 2 3 4 5 6 7
  • 20. Blueprinting Project: National Survey 20
  • 21. National Survey • Sampling plan in process – fall 2012 – Physicians, Residents, Students, Program Directors, Recently certified CFPC/RCPSC, Nurses, Pharmacists, MRAs, public members • Survey timing – early February 2013 • SME panel review – May 2013 • CEC review – June 2013 21 21
  • 22. Blueprinting Project: Next Steps 22
  • 23. Next Steps • 2012 Fall – Develop national survey questions – CEC to approve national survey • 2013 – February – National survey – March /April – Analyze data, prepare for SME panel meetings – May – SME panel meetings (two) – June – CEC to review draft blueprints – July – Stakeholder invitation to comment – August/September – Revise blueprints to include stakeholder comments – AGM – Present revised blueprints to AGM for consultation – Late fall – Blueprints to be revised and approved by CEC – Late 2013/early 2014 – Executive Board to approve blueprints 23 23
  • 25. Questions, Comments & Discussion Elaine Rodeck ERodeck@mcc.ca Thank you! 25 25

Editor's Notes

  1. Overall impacts that will result from BP include: 1. Potentially how the exams will be organized (i.e., Part I and II, and EE), 2. How content will be developed by the MCC test committees (i.e. role of MCC test committees in content development may change from question development to review). However, physician expertise will continue to be critical to process of content development. 3. Looking beyond the project at competencies that were described through the project by stakeholders as very important but cannot be assessed adequately through a point in time assessment
  2. Phase III will involve more than the launch of the new exams, it will involve: scoring the new exams, monitoring content (i.e., questions) to ensure continuous supple of new items, and updating test specifications for changes in expectations of candidates at entry into supervised and unsupervised practice.
  3. Sample clicker responses – used for all questions
  4. High rating of importance - Mean 5.8, std dev 0.54, range 4-6 Range across all questions – 3.2-6.0 Questions selected from each end of range.
  5. Low rating of importance - Mean 3.5, std dev 0.93, range 1-5
  6. High rating of importance - Mean 5.8, std dev 0.67, range 4-6
  7. Low rating of importance - Mean 3.5, std dev 0.93, range 1-5
  8. Low rating of importance - Mean 3.77, std dev 0.88, range 2-5
  9. Beyond national survey and including reference that Council will have the opportunity to review the draft test specifications at the 2013 AGM. Consult members of medical community.