SlideShare a Scribd company logo
1 of 47
Download to read offline
Statistical Analysis for Educational
Outcomes Measurement in CME
Jason J. Olivieri, MPH
Director, Outcomes Services
Imedex, LLC
(404) 319-9782
j.olivieri@imedex.com
www.assessCME.wordpress.com
Statistical Questions in CME
1. Was there an educational effect?
2. If so, how big was the effect?
3. How does this effect compare with other CME
activities?
Was there an educational effect?
Statistical tests of significance
• Determine whether any observed differences (e.g., pre- vs. post) are
due to random chance.
• Observed differences not attributed to random chance are
considered “statistically significant” and reflect a true difference
between groups that could be associated with participation in this
educational activity.
• Statistical significance is reported as a “P value”. A P value ≤ .05 is
considered statistically significant. Generally speaking, a P value ≤
.05 means that there is a 5% chance or less that the result of a
particular comparison is due to random chance.
Statistical tests of significance:
Choosing the right test
• Define variable type (typically either categorical or
ordinal)
• Is the comparison group data (e.g., pre vs. post) paired
or unpaired?
• What is the sample size?
Statistical tests of significance
• Categorical variables (e.g., knowledge “test” question)
– Unpaired data (comparison groups not matched)
• Chi square (samples ≥ 5)
• Fisher’s exact test (samples < 5)
– Paired data (matched comparison groups)
• McNemar’s test
Knowledge Change
Which of the following is NOT a complication associated with bisphosphonate therapy?
19%
10%
6%
17%
48%
8%
4%
2% 1%
85%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Arm pain Renal dysfunction Osteonecrosis of the jaw Atypical fx of the femur Peripheral neuropathy
(correct)
Pre-activity (n = 111) Post-activity (n = 157)
37% increase
Competency Change (case vignette)
Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung
(EGFR+)
51%
12%
19%
12%
6%
75%
6%
3%
9%
6%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Erlotinib alone
(evidence-based, preferred)
Erlotinib + bevacizumab Carboplatin-paclitaxel &
bevacizumab
Carboplatin-pemetrexed &
bevacizumab
Full house: erlotinib +
carboplatin-paclitaxel &
bevacizumab
Control (n = 65) Participants (n = 65)
24% increase
Statistical tests of significance
• Ordinal variables (e.g., rating scale)
– Unpaired data (comparison groups not matched)
• Mann-Whitney U
– Pair data (matched comparison groups)
• Wilcoxon signed-rank test
5.9
6
6.5
5.8
5.6
4.8
6.3
4.4
4.3
4.7
3.2
3.8
2.9
5.5
0 1 2 3 4 5 6 7
Achieving hemostatis with Coblation for adenotonsillectomy
Using Coblation for adenoidectomy
Using Coblation for complete tonsillectomy
Using Coblation for partial tonsillectomy
Using a microdebrider for adenoidectomy
Using a microdebrider for intracapsular tonsillectomy
Selecting an appropriate surgical technique for
adenotonsillectomy for specific indications in patients
Confidence in performing seven clinical tasks in adenotonsillectomy
(1 = not at all confident, 7 = extremely confident)
Pre (n = 57) Post (n = 49)
Knowledge (self-efficacy) Change
Competency Change
Using bevacizumab-based combo therapy for non-squamous NSCLC
32%
9%
41%
14%
5%
14%
18%
23%
32%
14%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1 = Never Not very often Sometimes Often 5 = Always
Current vs. planned use of bevacizumab-based combo therapy in nonsquamous NSCLC
Current (mean = 2.5, n = 44) Planned (mean = 3.1, n = 39)
Overall 26% increase in
planned vs current use
Once you know which statistical test to use,
where do you go?
www.VassarStats.net
Statistical test of significance:
Categorical variable example
•Participants in a CME activity were administered a paper-based pre-
and post-activity survey which contained five knowledge “test” questions
based on the CME content
•Survey participant responses were anonymous (i.e., pre/post not
matched)
•You want to determine if the proportion of respondents answering the
question correctly pre- vs. post-activity is significantly different
Knowledge Change
Which of the following is NOT a complication associated with bisphosphonate therapy?
19%
10%
6%
17%
48%
8%
4%
2% 1%
85%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Arm pain Renal dysfunction Osteonecrosis of the jaw Atypical fx of the femur Peripheral neuropathy
(correct)
Pre-activity (n = 111) Post-activity (n = 157)
37% increase
Statistical tests of significance
• Categorical variables (e.g., knowledge “test” question)
– Unpaired data (comparison groups not matched)
• Chi square (samples ≥ 5)
• Fisher’s exact test (samples < 5)
– Paired data (matched comparison groups)
• McNemar’s test
Calculating significance for a categorical variable
• Determine # of correct / incorrect answers for each
group (e.g., pre- and post-activity)
– Pre-activity: .48*111 = 53 correct / 58 incorrect
– Post-activity: .85*157 = 133 correct / 25 incorrect
• Plug these values into online calculator
(www.vassarstats.net)
– Click “frequency data”
– Click “Chi-Square, Cramer’s V and Lambda”
Statistical Analysis for Educational Outcomes Measurement in CME
Statistical Analysis for Educational Outcomes Measurement in CME
Knowledge Change
Which of the following is NOT a complication associated with bisphosphonate therapy?
19%
10%
6%
17%
48%
8%
4%
2% 1%
85%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Arm pain Renal dysfunction Osteonecrosis of the jaw Atypical fx of the femur Peripheral neuropathy
(correct)
Pre-activity (n = 111) Post-activity (n = 157)
37% increase
The proportion of respondents
answering this question correctly
pre- vs. post activity was not likely
due to random chance, P < . 0001
Statistical test of significance:
Ordinal variable example
•Participants in a CME activity were asked via ARS to rate their pre- and
post-activity use of five clinical practice strategies tied to the CME
content
•Survey participant responses were anonymous (i.e., pre/post not
matched)
•You want to determine whether the difference in rating pre- vs. post-
activity is significantly different
Competency Change
Using bevacizumab-based combo therapy for non-squamous NSCLC
32%
9%
41%
14%
5%
14%
18%
23%
32%
14%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1 = Never Not very often Sometimes Often 5 = Always
Current vs. planned use of bevacizumab-based combo therapy in nonsquamous NSCLC
Current (mean = 2.5, n = 44) Planned (mean = 3.1, n = 39)
Overall 26% increase in
planned vs current use
Statistical tests of significance
• Ordinal variables (e.g., rating scale)
– Unpaired data (comparison groups not matched)
• Mann-Whitney U
– Pair data (matched comparison groups)
• Wilcoxon signed-rank test
Calculating significance for an ordinal variable
• Go to www.vassarstats.net
– Click “ordinal data”
– Click “Mann-Whitney U test”
• Enter in the total number of pre-activity (“sample A”) and post-
activity (“sample B”) respondents
• Copy and paste pre- and post-activity responses into “sample A”
and “sample B” boxes
• Click “Import data to data cells”
• Click “Calculate from Raw Data”
Statistical Analysis for Educational Outcomes Measurement in CME
Statistical Analysis for Educational Outcomes Measurement in CME
Statistical Analysis for Educational Outcomes Measurement in CME
Statistical Analysis for Educational Outcomes Measurement in CME
Competency Change
Using bevacizumab-based combo therapy for non-squamous NSCLC
32%
9%
41%
14%
5%
14%
18%
23%
32%
14%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1 = Never Not very often Sometimes Often 5 = Always
Current vs. planned use of bevacizumab-based combo therapy in nonsquamous NSCLC
Current (mean = 2.5, n = 44) Planned (mean = 3.1, n = 39)
Overall 26% increase in
planned vs current use
The difference between pre- and
post-activity rating was statistically
significant, P = . 02
That’s it! You’re now able to
answer whether there was an
educational effect.
Now onto quantifying the
magnitude of such effect…
CME
activity
EOM plan
Pre- vs. post-activity assessment
via ARS or paper survey
Data 8 case vignette or clinical
practice strategy questions
Didactic presentation followed
by case-based discussion
Example paper survey question (clinical practice strategy)
Using bevacizumab-based combo therapy for non-squamous NSCLC
CME
activity
EOM plan
Pre- vs. post-activity assessment
via ARS or paper survey
Data 8 case vignette questions or
clinical practice strategies
Didactic presentation followed
by case-based discussion
How do we summarize
this data?
And how do we then
compare this result to
results of other activities?
What is effect size?
•Quantifies the magnitude of effect (maximum expected range: -3 to +3)
•Difference in means (e.g., pre-test and post-test) divided by the square
root of the pooled-group variances (Cohen’s d)
•Enables the comparison of CME effects across activities on a common
dimensionless scale
•Calculated from comparison data (e.g., pre/post, post/control) linked
directly to CME content
−Knowledge questions
−Case vignettes
−Self-reported frequency of use of key clinical practice strategies
How is it calculated?
Calculating effect size
• Can be done using only MS Excel® and
free, online resources
• Approach dependent upon variable type:
– ordinal (e.g., clinical practice strategy)
– categorical (e.g., case vignette)
Example paper survey question (clinical practice strategy)
Using bevacizumab-based combo therapy for non-squamous NSCLC
Calculating effect size for an ordinal variable
(e.g., clinical practice strategy)
• Calculate average and standard deviation for each group
(e.g., pre- and post-activity)
– Pre-activity: mean (SD) = 2.5 (1.3)
– Post-activity: mean (SD) = 3.1 (1.2)
• Plug these values into an online calculator
(http://www.uccs.edu/~lbecker/)
Statistical Analysis for Educational Outcomes Measurement in CME
Clinical practice strategy (CPS) use rating (1= never, 5 = always)
Pre-test Post-test Effect size
CPS #1
Standard
deviation
CPS #1
Standard
deviation
Cohen’s d
2.5 1.3 3.1 1.2 .48
*d = .2 (small effect), d = .5 (medium effect), d = .8 (large effect)
How is effect size interpreted? Ordinal variable example
How big was the educational effect?
Expressed in standard deviation units: The average
score of a post-test respondent was .48 standard
deviations above the average score of a pre-test
respondent
Effect sizes are proportional (.48 is twice as much
effect as .24)
How does the effect compare to other activities?
Cohen (1988): .2 = small, .5 = medium, .8 = large
Wolf (1986): .25 = educationally significant, .50 = clinically significant
Example ARS question (case vignette)
Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung
(EGFR+)
Calculating effect size for a categorical variable
(e.g., case vignette)
• Determine # of correct / incorrect answers for each
group (e.g., pre- and post-activity)
– Pre-activity: .51*65 = 33 correct / 32 incorrect
– Post-activity: .75*65 = 49 correct / 26 incorrect
• Plug these values into online calculator
(www.vassarstats.net)
– Click “frequency data”
– Click “Chi-Square, Cramer’s V and Lambda”
Statistical Analysis for Educational Outcomes Measurement in CME
Calculating effect size for a categorical
variable, continued
• Visit www.lyonsmorris.com/ma1/index.cfm
• Select “Correlation coefficient (r) to Effect Size”
• Enter Cramer’s V (.1474)
• Enter total number of pre-activity and post-activity
respondents
Statistical Analysis for Educational Outcomes Measurement in CME
How effective was your live CME in 2013?
10 live CME
activities
8 ARS questions
per activity
80 slides 10 effect sizes
One summary
effect size for
live CME
Statistical Analysis for Educational Outcomes Measurement in CME
Overall EIS (by format)
June 2010 – Sept 2013
n=9
1. Cohen. J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.
2. Mansouri & Lockyer. J Contin Educ Health Prof 2007;27:6-15.
3. Drexel et al. Int J Chron Obstruct Pulmon Dis 2011; 6: 297–307.
4. Casebeer et al. BMC Med Educ 2010;10: 42.
4Casebeer et al 2010. Knowledge
effect size (eLearning) = .82
3Drexel et al 2011.
Competence effect size = .85
2Mansouri & Lockyer 2007.
Knowledge effect size = .6
1Cohen J 1988. Small effect = .2, Medium effect = .5, Large effect = .8
Questions?
Resource:
www.assessCME.wordpress.com

More Related Content

What's hot

Statistics for Librarians, Session 2: Descriptive statistics
Statistics for Librarians, Session 2: Descriptive statisticsStatistics for Librarians, Session 2: Descriptive statistics
Statistics for Librarians, Session 2: Descriptive statisticsUniversity of North Texas
 
Factor analysis (fa)
Factor analysis (fa)Factor analysis (fa)
Factor analysis (fa)Rajdeep Raut
 
Factor Analysis (Marketing Research)
Factor Analysis (Marketing Research)Factor Analysis (Marketing Research)
Factor Analysis (Marketing Research)Mohammad Saif Alam
 
Factor Analysis in Research
Factor Analysis in ResearchFactor Analysis in Research
Factor Analysis in ResearchQasim Raza
 
Unit 4 editing and coding (2)
Unit 4 editing and coding (2)Unit 4 editing and coding (2)
Unit 4 editing and coding (2)kalailakshmi
 
Factor analysis using spss 2005
Factor analysis using spss 2005Factor analysis using spss 2005
Factor analysis using spss 2005jamescupello
 
Marketing Research-Factor Analysis
Marketing Research-Factor AnalysisMarketing Research-Factor Analysis
Marketing Research-Factor AnalysisArun Gupta
 
Multivariate data analysis regression, cluster and factor analysis on spss
Multivariate data analysis   regression, cluster and factor analysis on spssMultivariate data analysis   regression, cluster and factor analysis on spss
Multivariate data analysis regression, cluster and factor analysis on spssAditya Banerjee
 
Factor analysis
Factor analysisFactor analysis
Factor analysissaba khan
 
Correlation Research Design
Correlation Research DesignCorrelation Research Design
Correlation Research DesignSu Qee
 
Basics of Educational Statistics (T-test)
Basics of Educational Statistics (T-test)Basics of Educational Statistics (T-test)
Basics of Educational Statistics (T-test)HennaAnsari
 

What's hot (20)

Statistics for Librarians, Session 2: Descriptive statistics
Statistics for Librarians, Session 2: Descriptive statisticsStatistics for Librarians, Session 2: Descriptive statistics
Statistics for Librarians, Session 2: Descriptive statistics
 
Factor analysis (fa)
Factor analysis (fa)Factor analysis (fa)
Factor analysis (fa)
 
Factor Analysis (Marketing Research)
Factor Analysis (Marketing Research)Factor Analysis (Marketing Research)
Factor Analysis (Marketing Research)
 
Factor Analysis in Research
Factor Analysis in ResearchFactor Analysis in Research
Factor Analysis in Research
 
Unit 4 editing and coding (2)
Unit 4 editing and coding (2)Unit 4 editing and coding (2)
Unit 4 editing and coding (2)
 
Quantitative Data analysis
Quantitative Data analysisQuantitative Data analysis
Quantitative Data analysis
 
Factor analysis (1)
Factor analysis (1)Factor analysis (1)
Factor analysis (1)
 
Factor analysis
Factor analysisFactor analysis
Factor analysis
 
Factor analysis
Factor analysisFactor analysis
Factor analysis
 
Factor analysis using spss 2005
Factor analysis using spss 2005Factor analysis using spss 2005
Factor analysis using spss 2005
 
Statistics
StatisticsStatistics
Statistics
 
Marketing Research-Factor Analysis
Marketing Research-Factor AnalysisMarketing Research-Factor Analysis
Marketing Research-Factor Analysis
 
Factor analysis
Factor analysisFactor analysis
Factor analysis
 
Statistical analysis in SPSS_
Statistical analysis in SPSS_ Statistical analysis in SPSS_
Statistical analysis in SPSS_
 
Priya
PriyaPriya
Priya
 
Multivariate data analysis regression, cluster and factor analysis on spss
Multivariate data analysis   regression, cluster and factor analysis on spssMultivariate data analysis   regression, cluster and factor analysis on spss
Multivariate data analysis regression, cluster and factor analysis on spss
 
Factor analysis
Factor analysisFactor analysis
Factor analysis
 
Correlation Research Design
Correlation Research DesignCorrelation Research Design
Correlation Research Design
 
Factor analysis
Factor analysisFactor analysis
Factor analysis
 
Basics of Educational Statistics (T-test)
Basics of Educational Statistics (T-test)Basics of Educational Statistics (T-test)
Basics of Educational Statistics (T-test)
 

Viewers also liked

Measurement and uncertainty
Measurement and uncertainty Measurement and uncertainty
Measurement and uncertainty IB School
 
Measurement and uncertainties
Measurement and uncertaintiesMeasurement and uncertainties
Measurement and uncertaintieskleybf
 
Introduction to Mechanical Measurement
Introduction to Mechanical MeasurementIntroduction to Mechanical Measurement
Introduction to Mechanical MeasurementNaman Dave
 
Measurement errors, Statistical Analysis, Uncertainty
Measurement errors, Statistical Analysis, UncertaintyMeasurement errors, Statistical Analysis, Uncertainty
Measurement errors, Statistical Analysis, UncertaintyDr Naim R Kidwai
 
Presentation on measurements & measurement systems
Presentation on measurements & measurement systemsPresentation on measurements & measurement systems
Presentation on measurements & measurement systemsROBI Axiata Company Limited
 
Basic Statistical Concepts and Methods
Basic Statistical Concepts and MethodsBasic Statistical Concepts and Methods
Basic Statistical Concepts and MethodsAhmed-Refat Refat
 
Role of Statistics in Scientific Research
Role of Statistics in Scientific ResearchRole of Statistics in Scientific Research
Role of Statistics in Scientific ResearchVaruna Harshana
 

Viewers also liked (10)

Measurement and uncertainty
Measurement and uncertainty Measurement and uncertainty
Measurement and uncertainty
 
Measurement uncertainty
Measurement uncertaintyMeasurement uncertainty
Measurement uncertainty
 
Measurement and uncertainties
Measurement and uncertaintiesMeasurement and uncertainties
Measurement and uncertainties
 
Errors in measurement
Errors in measurementErrors in measurement
Errors in measurement
 
Introduction to Mechanical Measurement
Introduction to Mechanical MeasurementIntroduction to Mechanical Measurement
Introduction to Mechanical Measurement
 
Measurement errors, Statistical Analysis, Uncertainty
Measurement errors, Statistical Analysis, UncertaintyMeasurement errors, Statistical Analysis, Uncertainty
Measurement errors, Statistical Analysis, Uncertainty
 
Presentation on measurements & measurement systems
Presentation on measurements & measurement systemsPresentation on measurements & measurement systems
Presentation on measurements & measurement systems
 
Basic Statistical Concepts and Methods
Basic Statistical Concepts and MethodsBasic Statistical Concepts and Methods
Basic Statistical Concepts and Methods
 
Statistical Analysis
Statistical AnalysisStatistical Analysis
Statistical Analysis
 
Role of Statistics in Scientific Research
Role of Statistics in Scientific ResearchRole of Statistics in Scientific Research
Role of Statistics in Scientific Research
 

Similar to Statistical Analysis for Educational Outcomes Measurement in CME

Introduction to Data Management in Human Ecology
Introduction to Data Management in Human EcologyIntroduction to Data Management in Human Ecology
Introduction to Data Management in Human EcologyKern Rocke
 
Quantitative analysis
Quantitative analysisQuantitative analysis
Quantitative analysisRajesh Mishra
 
Quantitative_analysis.ppt
Quantitative_analysis.pptQuantitative_analysis.ppt
Quantitative_analysis.pptmousaderhem1
 
Multivariate Analysis and Visualization of Proteomic Data
Multivariate Analysis and Visualization of Proteomic DataMultivariate Analysis and Visualization of Proteomic Data
Multivariate Analysis and Visualization of Proteomic DataUC Davis
 
Week 2 measures of disease occurence
Week 2  measures of disease occurenceWeek 2  measures of disease occurence
Week 2 measures of disease occurenceHamdi Alhakimi
 
Network meta-analysis & models for inconsistency
Network meta-analysis & models for inconsistencyNetwork meta-analysis & models for inconsistency
Network meta-analysis & models for inconsistencycheweb1
 
Some nonparametric statistic for categorical &amp; ordinal data
Some nonparametric statistic for categorical &amp; ordinal dataSome nonparametric statistic for categorical &amp; ordinal data
Some nonparametric statistic for categorical &amp; ordinal dataProfessor Desmond Ayim-Aboagye
 
Ipac 2014
Ipac 2014Ipac 2014
Ipac 2014cwhms
 
Quick introduction to critical appraisal of quantitative research
Quick introduction to critical appraisal of quantitative researchQuick introduction to critical appraisal of quantitative research
Quick introduction to critical appraisal of quantitative researchAlan Fricker
 
Introduction to biostatistics
Introduction to biostatisticsIntroduction to biostatistics
Introduction to biostatisticsINAAMUL HAQ
 
Statistical issues in subgroup analyses
Statistical issues in subgroup analysesStatistical issues in subgroup analyses
Statistical issues in subgroup analysesFrancois MAIGNEN
 
Nonparametric tests assignment
Nonparametric tests assignmentNonparametric tests assignment
Nonparametric tests assignmentROOHASHAHID1
 
Non parametric study; Statistical approach for med student
Non parametric study; Statistical approach for med student Non parametric study; Statistical approach for med student
Non parametric study; Statistical approach for med student Dr. Rupendra Bharti
 
Cochrane Collaboration
Cochrane CollaborationCochrane Collaboration
Cochrane CollaborationNinian Peckitt
 
Sample Size Calculations for Impact Evaluations
Sample Size Calculations for Impact EvaluationsSample Size Calculations for Impact Evaluations
Sample Size Calculations for Impact EvaluationsMarcos Vera
 

Similar to Statistical Analysis for Educational Outcomes Measurement in CME (20)

Introduction to Data Management in Human Ecology
Introduction to Data Management in Human EcologyIntroduction to Data Management in Human Ecology
Introduction to Data Management in Human Ecology
 
Quantitative analysis
Quantitative analysisQuantitative analysis
Quantitative analysis
 
Quantitative_analysis.ppt
Quantitative_analysis.pptQuantitative_analysis.ppt
Quantitative_analysis.ppt
 
Multivariate Analysis and Visualization of Proteomic Data
Multivariate Analysis and Visualization of Proteomic DataMultivariate Analysis and Visualization of Proteomic Data
Multivariate Analysis and Visualization of Proteomic Data
 
Ins and outs of clinical trials
Ins and outs of clinical trialsIns and outs of clinical trials
Ins and outs of clinical trials
 
Week 2 measures of disease occurence
Week 2  measures of disease occurenceWeek 2  measures of disease occurence
Week 2 measures of disease occurence
 
Metaanalysis copy
Metaanalysis    copyMetaanalysis    copy
Metaanalysis copy
 
Network meta-analysis & models for inconsistency
Network meta-analysis & models for inconsistencyNetwork meta-analysis & models for inconsistency
Network meta-analysis & models for inconsistency
 
Some nonparametric statistic for categorical &amp; ordinal data
Some nonparametric statistic for categorical &amp; ordinal dataSome nonparametric statistic for categorical &amp; ordinal data
Some nonparametric statistic for categorical &amp; ordinal data
 
Biostatistics ii
Biostatistics iiBiostatistics ii
Biostatistics ii
 
Ipac 2014
Ipac 2014Ipac 2014
Ipac 2014
 
Quick introduction to critical appraisal of quantitative research
Quick introduction to critical appraisal of quantitative researchQuick introduction to critical appraisal of quantitative research
Quick introduction to critical appraisal of quantitative research
 
Causal Inference Opening Workshop - Smooth Extensions to BART for Heterogeneo...
Causal Inference Opening Workshop - Smooth Extensions to BART for Heterogeneo...Causal Inference Opening Workshop - Smooth Extensions to BART for Heterogeneo...
Causal Inference Opening Workshop - Smooth Extensions to BART for Heterogeneo...
 
Introduction to biostatistics
Introduction to biostatisticsIntroduction to biostatistics
Introduction to biostatistics
 
Statistical issues in subgroup analyses
Statistical issues in subgroup analysesStatistical issues in subgroup analyses
Statistical issues in subgroup analyses
 
Nonparametric tests assignment
Nonparametric tests assignmentNonparametric tests assignment
Nonparametric tests assignment
 
Non parametric study; Statistical approach for med student
Non parametric study; Statistical approach for med student Non parametric study; Statistical approach for med student
Non parametric study; Statistical approach for med student
 
Cochrane Collaboration
Cochrane CollaborationCochrane Collaboration
Cochrane Collaboration
 
Sample Size Calculations for Impact Evaluations
Sample Size Calculations for Impact EvaluationsSample Size Calculations for Impact Evaluations
Sample Size Calculations for Impact Evaluations
 
Statistics.pptx
Statistics.pptxStatistics.pptx
Statistics.pptx
 

More from D. Warnick Consulting

How Did All These People Get Here? Looking Beyond Level 1 Outcomes in Live Ev...
How Did All These People Get Here? Looking Beyond Level 1 Outcomes in Live Ev...How Did All These People Get Here? Looking Beyond Level 1 Outcomes in Live Ev...
How Did All These People Get Here? Looking Beyond Level 1 Outcomes in Live Ev...D. Warnick Consulting
 
Moore than Outcomes Levels (Guerrero & Abdolrasulnia)
Moore than Outcomes Levels (Guerrero & Abdolrasulnia)Moore than Outcomes Levels (Guerrero & Abdolrasulnia)
Moore than Outcomes Levels (Guerrero & Abdolrasulnia)D. Warnick Consulting
 
Activity Planning: Improving the Process to Align with Accreditation Criteria...
Activity Planning: Improving the Process to Align with Accreditation Criteria...Activity Planning: Improving the Process to Align with Accreditation Criteria...
Activity Planning: Improving the Process to Align with Accreditation Criteria...D. Warnick Consulting
 
Implications, Applications and Approaches to Complying with the Sunshine (Dwo...
Implications, Applications and Approaches to Complying with the Sunshine (Dwo...Implications, Applications and Approaches to Complying with the Sunshine (Dwo...
Implications, Applications and Approaches to Complying with the Sunshine (Dwo...D. Warnick Consulting
 
Qualitative Research and CEHP (Turell & Howson)
Qualitative Research and CEHP (Turell & Howson)Qualitative Research and CEHP (Turell & Howson)
Qualitative Research and CEHP (Turell & Howson)D. Warnick Consulting
 
Sneak Peek at the 2014 Benchmarking Survey (Guerrero & Dzenko)
Sneak Peek at the 2014 Benchmarking Survey (Guerrero & Dzenko)Sneak Peek at the 2014 Benchmarking Survey (Guerrero & Dzenko)
Sneak Peek at the 2014 Benchmarking Survey (Guerrero & Dzenko)D. Warnick Consulting
 
Marketing to Today’s Clinicians (Laudenslager)
Marketing to Today’s Clinicians (Laudenslager)Marketing to Today’s Clinicians (Laudenslager)
Marketing to Today’s Clinicians (Laudenslager)D. Warnick Consulting
 
Supporter/Provider Communication: Optimal vs Minimal
Supporter/Provider Communication: Optimal vs Minimal Supporter/Provider Communication: Optimal vs Minimal
Supporter/Provider Communication: Optimal vs Minimal D. Warnick Consulting
 
Criterion 3 (compliance and non compliance)
Criterion 3 (compliance and non compliance)Criterion 3 (compliance and non compliance)
Criterion 3 (compliance and non compliance)D. Warnick Consulting
 
ACCME Criterion 2 (Examples of Non-Compliance)
ACCME Criterion 2 (Examples of Non-Compliance)ACCME Criterion 2 (Examples of Non-Compliance)
ACCME Criterion 2 (Examples of Non-Compliance)D. Warnick Consulting
 
ACCME Criterion 2 (Examples of Compliance)
ACCME Criterion 2 (Examples of Compliance)ACCME Criterion 2 (Examples of Compliance)
ACCME Criterion 2 (Examples of Compliance)D. Warnick Consulting
 
Utilizing Social Media for Learner Engagement
Utilizing Social Media for Learner EngagementUtilizing Social Media for Learner Engagement
Utilizing Social Media for Learner EngagementD. Warnick Consulting
 
Social Media and the CME Professional
Social Media and the CME ProfessionalSocial Media and the CME Professional
Social Media and the CME ProfessionalD. Warnick Consulting
 
CME Saves Lives: A Pep Talk For The CME Community
CME Saves Lives: A Pep Talk For The CME CommunityCME Saves Lives: A Pep Talk For The CME Community
CME Saves Lives: A Pep Talk For The CME CommunityD. Warnick Consulting
 

More from D. Warnick Consulting (16)

How Did All These People Get Here? Looking Beyond Level 1 Outcomes in Live Ev...
How Did All These People Get Here? Looking Beyond Level 1 Outcomes in Live Ev...How Did All These People Get Here? Looking Beyond Level 1 Outcomes in Live Ev...
How Did All These People Get Here? Looking Beyond Level 1 Outcomes in Live Ev...
 
Moore than Outcomes Levels (Guerrero & Abdolrasulnia)
Moore than Outcomes Levels (Guerrero & Abdolrasulnia)Moore than Outcomes Levels (Guerrero & Abdolrasulnia)
Moore than Outcomes Levels (Guerrero & Abdolrasulnia)
 
Activity Planning: Improving the Process to Align with Accreditation Criteria...
Activity Planning: Improving the Process to Align with Accreditation Criteria...Activity Planning: Improving the Process to Align with Accreditation Criteria...
Activity Planning: Improving the Process to Align with Accreditation Criteria...
 
Implications, Applications and Approaches to Complying with the Sunshine (Dwo...
Implications, Applications and Approaches to Complying with the Sunshine (Dwo...Implications, Applications and Approaches to Complying with the Sunshine (Dwo...
Implications, Applications and Approaches to Complying with the Sunshine (Dwo...
 
Qualitative Research and CEHP (Turell & Howson)
Qualitative Research and CEHP (Turell & Howson)Qualitative Research and CEHP (Turell & Howson)
Qualitative Research and CEHP (Turell & Howson)
 
Sneak Peek at the 2014 Benchmarking Survey (Guerrero & Dzenko)
Sneak Peek at the 2014 Benchmarking Survey (Guerrero & Dzenko)Sneak Peek at the 2014 Benchmarking Survey (Guerrero & Dzenko)
Sneak Peek at the 2014 Benchmarking Survey (Guerrero & Dzenko)
 
Marketing to Today’s Clinicians (Laudenslager)
Marketing to Today’s Clinicians (Laudenslager)Marketing to Today’s Clinicians (Laudenslager)
Marketing to Today’s Clinicians (Laudenslager)
 
Supporter/Provider Communication: Optimal vs Minimal
Supporter/Provider Communication: Optimal vs Minimal Supporter/Provider Communication: Optimal vs Minimal
Supporter/Provider Communication: Optimal vs Minimal
 
Criterion 3 (compliance and non compliance)
Criterion 3 (compliance and non compliance)Criterion 3 (compliance and non compliance)
Criterion 3 (compliance and non compliance)
 
ACCME Criterion 2 (Examples of Non-Compliance)
ACCME Criterion 2 (Examples of Non-Compliance)ACCME Criterion 2 (Examples of Non-Compliance)
ACCME Criterion 2 (Examples of Non-Compliance)
 
ACCME Criterion 2 (Examples of Compliance)
ACCME Criterion 2 (Examples of Compliance)ACCME Criterion 2 (Examples of Compliance)
ACCME Criterion 2 (Examples of Compliance)
 
Social Media and CME
Social Media and CMESocial Media and CME
Social Media and CME
 
Utilizing Social Media for Learner Engagement
Utilizing Social Media for Learner EngagementUtilizing Social Media for Learner Engagement
Utilizing Social Media for Learner Engagement
 
Social Media and the CME Professional
Social Media and the CME ProfessionalSocial Media and the CME Professional
Social Media and the CME Professional
 
CME Saves Lives: A Pep Talk For The CME Community
CME Saves Lives: A Pep Talk For The CME CommunityCME Saves Lives: A Pep Talk For The CME Community
CME Saves Lives: A Pep Talk For The CME Community
 
Anatomy of a tweet
Anatomy of a tweetAnatomy of a tweet
Anatomy of a tweet
 

Recently uploaded

SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSyedNadeemGillANi
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesCeline George
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...CaraSkikne1
 
Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya
 
A gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceA gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceApostolos Syropoulos
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17Celine George
 
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...M56BOOKSTORE PRODUCT/SERVICE
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxraviapr7
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.EnglishCEIPdeSigeiro
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapitolTechU
 
How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17Celine George
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.raviapr7
 
ARTICULAR DISC OF TEMPOROMANDIBULAR JOINT
ARTICULAR DISC OF TEMPOROMANDIBULAR JOINTARTICULAR DISC OF TEMPOROMANDIBULAR JOINT
ARTICULAR DISC OF TEMPOROMANDIBULAR JOINTDR. SNEHA NAIR
 
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptxSlides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptxCapitolTechU
 
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...Dr. Asif Anas
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17Celine George
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17Celine George
 

Recently uploaded (20)

SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 Sales
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...
 
Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....
 
A gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceA gentle introduction to Artificial Intelligence
A gentle introduction to Artificial Intelligence
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17
 
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptx
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptx
 
How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.
 
ARTICULAR DISC OF TEMPOROMANDIBULAR JOINT
ARTICULAR DISC OF TEMPOROMANDIBULAR JOINTARTICULAR DISC OF TEMPOROMANDIBULAR JOINT
ARTICULAR DISC OF TEMPOROMANDIBULAR JOINT
 
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptxSlides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
 
March 2024 Directors Meeting, Division of Student Affairs and Academic Support
March 2024 Directors Meeting, Division of Student Affairs and Academic SupportMarch 2024 Directors Meeting, Division of Student Affairs and Academic Support
March 2024 Directors Meeting, Division of Student Affairs and Academic Support
 
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17
 

Statistical Analysis for Educational Outcomes Measurement in CME

  • 1. Statistical Analysis for Educational Outcomes Measurement in CME Jason J. Olivieri, MPH Director, Outcomes Services Imedex, LLC (404) 319-9782 j.olivieri@imedex.com www.assessCME.wordpress.com
  • 2. Statistical Questions in CME 1. Was there an educational effect? 2. If so, how big was the effect? 3. How does this effect compare with other CME activities?
  • 3. Was there an educational effect? Statistical tests of significance • Determine whether any observed differences (e.g., pre- vs. post) are due to random chance. • Observed differences not attributed to random chance are considered “statistically significant” and reflect a true difference between groups that could be associated with participation in this educational activity. • Statistical significance is reported as a “P value”. A P value ≤ .05 is considered statistically significant. Generally speaking, a P value ≤ .05 means that there is a 5% chance or less that the result of a particular comparison is due to random chance.
  • 4. Statistical tests of significance: Choosing the right test • Define variable type (typically either categorical or ordinal) • Is the comparison group data (e.g., pre vs. post) paired or unpaired? • What is the sample size?
  • 5. Statistical tests of significance • Categorical variables (e.g., knowledge “test” question) – Unpaired data (comparison groups not matched) • Chi square (samples ≥ 5) • Fisher’s exact test (samples < 5) – Paired data (matched comparison groups) • McNemar’s test
  • 6. Knowledge Change Which of the following is NOT a complication associated with bisphosphonate therapy? 19% 10% 6% 17% 48% 8% 4% 2% 1% 85% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Arm pain Renal dysfunction Osteonecrosis of the jaw Atypical fx of the femur Peripheral neuropathy (correct) Pre-activity (n = 111) Post-activity (n = 157) 37% increase
  • 7. Competency Change (case vignette) Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung (EGFR+) 51% 12% 19% 12% 6% 75% 6% 3% 9% 6% 0% 10% 20% 30% 40% 50% 60% 70% 80% Erlotinib alone (evidence-based, preferred) Erlotinib + bevacizumab Carboplatin-paclitaxel & bevacizumab Carboplatin-pemetrexed & bevacizumab Full house: erlotinib + carboplatin-paclitaxel & bevacizumab Control (n = 65) Participants (n = 65) 24% increase
  • 8. Statistical tests of significance • Ordinal variables (e.g., rating scale) – Unpaired data (comparison groups not matched) • Mann-Whitney U – Pair data (matched comparison groups) • Wilcoxon signed-rank test
  • 9. 5.9 6 6.5 5.8 5.6 4.8 6.3 4.4 4.3 4.7 3.2 3.8 2.9 5.5 0 1 2 3 4 5 6 7 Achieving hemostatis with Coblation for adenotonsillectomy Using Coblation for adenoidectomy Using Coblation for complete tonsillectomy Using Coblation for partial tonsillectomy Using a microdebrider for adenoidectomy Using a microdebrider for intracapsular tonsillectomy Selecting an appropriate surgical technique for adenotonsillectomy for specific indications in patients Confidence in performing seven clinical tasks in adenotonsillectomy (1 = not at all confident, 7 = extremely confident) Pre (n = 57) Post (n = 49) Knowledge (self-efficacy) Change
  • 10. Competency Change Using bevacizumab-based combo therapy for non-squamous NSCLC 32% 9% 41% 14% 5% 14% 18% 23% 32% 14% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 1 = Never Not very often Sometimes Often 5 = Always Current vs. planned use of bevacizumab-based combo therapy in nonsquamous NSCLC Current (mean = 2.5, n = 44) Planned (mean = 3.1, n = 39) Overall 26% increase in planned vs current use
  • 11. Once you know which statistical test to use, where do you go? www.VassarStats.net
  • 12. Statistical test of significance: Categorical variable example •Participants in a CME activity were administered a paper-based pre- and post-activity survey which contained five knowledge “test” questions based on the CME content •Survey participant responses were anonymous (i.e., pre/post not matched) •You want to determine if the proportion of respondents answering the question correctly pre- vs. post-activity is significantly different
  • 13. Knowledge Change Which of the following is NOT a complication associated with bisphosphonate therapy? 19% 10% 6% 17% 48% 8% 4% 2% 1% 85% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Arm pain Renal dysfunction Osteonecrosis of the jaw Atypical fx of the femur Peripheral neuropathy (correct) Pre-activity (n = 111) Post-activity (n = 157) 37% increase
  • 14. Statistical tests of significance • Categorical variables (e.g., knowledge “test” question) – Unpaired data (comparison groups not matched) • Chi square (samples ≥ 5) • Fisher’s exact test (samples < 5) – Paired data (matched comparison groups) • McNemar’s test
  • 15. Calculating significance for a categorical variable • Determine # of correct / incorrect answers for each group (e.g., pre- and post-activity) – Pre-activity: .48*111 = 53 correct / 58 incorrect – Post-activity: .85*157 = 133 correct / 25 incorrect • Plug these values into online calculator (www.vassarstats.net) – Click “frequency data” – Click “Chi-Square, Cramer’s V and Lambda”
  • 18. Knowledge Change Which of the following is NOT a complication associated with bisphosphonate therapy? 19% 10% 6% 17% 48% 8% 4% 2% 1% 85% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Arm pain Renal dysfunction Osteonecrosis of the jaw Atypical fx of the femur Peripheral neuropathy (correct) Pre-activity (n = 111) Post-activity (n = 157) 37% increase The proportion of respondents answering this question correctly pre- vs. post activity was not likely due to random chance, P < . 0001
  • 19. Statistical test of significance: Ordinal variable example •Participants in a CME activity were asked via ARS to rate their pre- and post-activity use of five clinical practice strategies tied to the CME content •Survey participant responses were anonymous (i.e., pre/post not matched) •You want to determine whether the difference in rating pre- vs. post- activity is significantly different
  • 20. Competency Change Using bevacizumab-based combo therapy for non-squamous NSCLC 32% 9% 41% 14% 5% 14% 18% 23% 32% 14% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 1 = Never Not very often Sometimes Often 5 = Always Current vs. planned use of bevacizumab-based combo therapy in nonsquamous NSCLC Current (mean = 2.5, n = 44) Planned (mean = 3.1, n = 39) Overall 26% increase in planned vs current use
  • 21. Statistical tests of significance • Ordinal variables (e.g., rating scale) – Unpaired data (comparison groups not matched) • Mann-Whitney U – Pair data (matched comparison groups) • Wilcoxon signed-rank test
  • 22. Calculating significance for an ordinal variable • Go to www.vassarstats.net – Click “ordinal data” – Click “Mann-Whitney U test” • Enter in the total number of pre-activity (“sample A”) and post- activity (“sample B”) respondents • Copy and paste pre- and post-activity responses into “sample A” and “sample B” boxes • Click “Import data to data cells” • Click “Calculate from Raw Data”
  • 27. Competency Change Using bevacizumab-based combo therapy for non-squamous NSCLC 32% 9% 41% 14% 5% 14% 18% 23% 32% 14% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 1 = Never Not very often Sometimes Often 5 = Always Current vs. planned use of bevacizumab-based combo therapy in nonsquamous NSCLC Current (mean = 2.5, n = 44) Planned (mean = 3.1, n = 39) Overall 26% increase in planned vs current use The difference between pre- and post-activity rating was statistically significant, P = . 02
  • 28. That’s it! You’re now able to answer whether there was an educational effect. Now onto quantifying the magnitude of such effect…
  • 29. CME activity EOM plan Pre- vs. post-activity assessment via ARS or paper survey Data 8 case vignette or clinical practice strategy questions Didactic presentation followed by case-based discussion
  • 30. Example paper survey question (clinical practice strategy) Using bevacizumab-based combo therapy for non-squamous NSCLC
  • 31. CME activity EOM plan Pre- vs. post-activity assessment via ARS or paper survey Data 8 case vignette questions or clinical practice strategies Didactic presentation followed by case-based discussion How do we summarize this data? And how do we then compare this result to results of other activities?
  • 32. What is effect size? •Quantifies the magnitude of effect (maximum expected range: -3 to +3) •Difference in means (e.g., pre-test and post-test) divided by the square root of the pooled-group variances (Cohen’s d) •Enables the comparison of CME effects across activities on a common dimensionless scale •Calculated from comparison data (e.g., pre/post, post/control) linked directly to CME content −Knowledge questions −Case vignettes −Self-reported frequency of use of key clinical practice strategies
  • 33. How is it calculated?
  • 34. Calculating effect size • Can be done using only MS Excel® and free, online resources • Approach dependent upon variable type: – ordinal (e.g., clinical practice strategy) – categorical (e.g., case vignette)
  • 35. Example paper survey question (clinical practice strategy) Using bevacizumab-based combo therapy for non-squamous NSCLC
  • 36. Calculating effect size for an ordinal variable (e.g., clinical practice strategy) • Calculate average and standard deviation for each group (e.g., pre- and post-activity) – Pre-activity: mean (SD) = 2.5 (1.3) – Post-activity: mean (SD) = 3.1 (1.2) • Plug these values into an online calculator (http://www.uccs.edu/~lbecker/)
  • 38. Clinical practice strategy (CPS) use rating (1= never, 5 = always) Pre-test Post-test Effect size CPS #1 Standard deviation CPS #1 Standard deviation Cohen’s d 2.5 1.3 3.1 1.2 .48 *d = .2 (small effect), d = .5 (medium effect), d = .8 (large effect) How is effect size interpreted? Ordinal variable example How big was the educational effect? Expressed in standard deviation units: The average score of a post-test respondent was .48 standard deviations above the average score of a pre-test respondent Effect sizes are proportional (.48 is twice as much effect as .24) How does the effect compare to other activities? Cohen (1988): .2 = small, .5 = medium, .8 = large Wolf (1986): .25 = educationally significant, .50 = clinically significant
  • 39. Example ARS question (case vignette) Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung (EGFR+)
  • 40. Calculating effect size for a categorical variable (e.g., case vignette) • Determine # of correct / incorrect answers for each group (e.g., pre- and post-activity) – Pre-activity: .51*65 = 33 correct / 32 incorrect – Post-activity: .75*65 = 49 correct / 26 incorrect • Plug these values into online calculator (www.vassarstats.net) – Click “frequency data” – Click “Chi-Square, Cramer’s V and Lambda”
  • 42. Calculating effect size for a categorical variable, continued • Visit www.lyonsmorris.com/ma1/index.cfm • Select “Correlation coefficient (r) to Effect Size” • Enter Cramer’s V (.1474) • Enter total number of pre-activity and post-activity respondents
  • 44. How effective was your live CME in 2013? 10 live CME activities 8 ARS questions per activity 80 slides 10 effect sizes One summary effect size for live CME
  • 46. Overall EIS (by format) June 2010 – Sept 2013 n=9 1. Cohen. J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates. 2. Mansouri & Lockyer. J Contin Educ Health Prof 2007;27:6-15. 3. Drexel et al. Int J Chron Obstruct Pulmon Dis 2011; 6: 297–307. 4. Casebeer et al. BMC Med Educ 2010;10: 42. 4Casebeer et al 2010. Knowledge effect size (eLearning) = .82 3Drexel et al 2011. Competence effect size = .85 2Mansouri & Lockyer 2007. Knowledge effect size = .6 1Cohen J 1988. Small effect = .2, Medium effect = .5, Large effect = .8