Medical Knowledge Self-Assessment Program Performance InterpretationGuidelines with Norm Tables MKSAP 16Copyright 2013 American College of Physicians

MKSAP 16Performance Interpretation Guidelines with Norm TablesSection I. How Do You Compare to Your Peers? . 6Section II. Participant Groups . 9Section III. What You Need to Know About Your Scores . 9A. Introduction. 9B. Content Validity . 10C. Reliability (Rel) . 11D. Standard Error of Measurement (SEM) . 12Section IV. Norm Tables . 13Section V. Answer Key and Percentage Answering Correctly . 16Part ACardiovascular Medicine . 16Dermatology . 18Gastroenterology and Hepatology. 19Hematology and Oncology . 21Neurology . 23Rheumatology. 25Part BEndocrinology and Metabolism . 27General Internal Medicine . 28Infectious Disease . 30Nephrology . 32Pulmonary and Critical Care Medicine . 34Section VI. Self-Scoring Instructions . 36Page 1

Principal StaffSenior Vice President, Medical EducationEditor-in-ChiefPatrick C. Alguire, MD, FACPVice President, Medical EducationD. Theresa Kanya, MBADirector, Self-Assessment ProgramsSean McKinneyManaging EditorMargaret WellsDeputy Editor-in-ChiefPhilip A. Masters, MD, FACPSenior Medical Associate for Content DevelopmentCynthia D. Smith, MD, FACPProgram AdministratorValerie DangovetskySenior Staff EditorsKatie IdellBecky Krumm, ELSEllen McDonald, PhD, ELSStaff EditorMegan ZborowskiProduction Administrator/EditorRandy Hendrickson, MS, MPPAssistant EditorsLinnea DonnarummaJohn HaefeleAcknowledgmentsMKSAP 16 was developed by the American College of Physicians.Reporting is being done by ACT, Inc., Iowa City, Iowa.Page 2

Continuing Medical EducationThe American College of Physicians is accredited by the AccreditationCouncil for Continuing Medical Education (ACCME) to provide continuingmedical education for physicians.The American College of Physicians designates this enduring material fora maximum of 174 AMA PRA Category 1 Credits . Physicians shouldclaim only the credit commensurate with the extent of their participation inthe activity.The American Medical Association has determined that physicians notlicensed in the United States who participate in this CME activity areeligible for AMA PRA Category 1 Credit .Credit is available from July 31, 2012, to July 31, 2015, for the Part Abooks, and from December 31, 2012, to December 31, 2015 for the Part Bbooks.Learning ObjectivesThe learning objectives of the Medical Knowledge Self-AssessmentProgram are to: Close gaps between actual care in your practice and preferredstandards of care, based on best evidence Diagnose disease states that are less common and sometimesoverlooked and confusing Improve management of comorbid conditions that can complicatepatient care Determine when to refer patients for surgery or care bysubspecialists Pass the ABIM certification examination Pass the ABIM maintenance of certification examinationPage 3

Target Audience General internists and primary care physicians Subspecialists who need to remain up-to-date in internal medicine Residents preparing for the certifying examination in internalmedicine Physicians participating for maintenance of certification in internalmedicine (recertification)Educational DisclaimerThe editors and publisher of Medical Knowledge Self-AssessmentProgram 16 recognize that the development of new material offers manyopportunities for error. Despite our best efforts, some errors may persist inprint. Drug dosage schedules are, we believe, accurate and in accordancewith current standards. Readers are advised, however, to ensure that therecommended dosages in MKSAP 16 concur with the informationprovided in the product information material. This is especially important incases of new, infrequently used, or highly toxic drugs. Application of theinformation in MKSAP 16 remains the professional responsibility of thepractitioner.The primary purpose of MKSAP 16 is educational. Information presented,as well as publications, technologies, products, and/or services discussed,is intended to inform subscribers about the knowledge, techniques, andexperiences of the contributors. A diversity of professional opinion exists,and the views of the contributors are their own and not those of the ACP.Inclusion of any material in the program does not constitute endorsementor recommendation by the ACP. The ACP does not warrant the safety,reliability, accuracy, completeness, or usefulness of and disclaims any andall liability for damages and claims that may result from the use ofinformation,publications,technologies,discussed in this program.Page 4products,and/orservices

Publisher’s InformationCopyright 2013 American College of Physicians, Inc. All rights reserved.This publication is protected by copyright. No part of this publication maybe reproduced, stored in a retrieval system, or transmitted in any form orby any means, electronic or mechanical, including photocopy, without theexpress consent of the ACP. MKSAP 16 is for individual use only. Onlyone account per subscription will be permitted for the purpose of earningCME credits and MOC points/credits and for other authorized uses ofMKSAP 16.Important copyright information from the ACP: Unauthorized reproductionof this publication is unlawful. The ACP prohibits reproduction of thispublication or any of its parts in any form either for individual use or fordistribution.The ACP will consider granting an individual permission to reproduce onlylimited portions of this publication for his or her own exclusive use. Sendrequests in writing to MKSAP Permissions, American College ofPhysicians, 190 N. Independence Mall West, Philadelphia, PA 191061572.Printed in the United States of America.For order information in the US or Canada call 800-523-1546, extension2600. All other countries call 215-351-2600. Fax inquiries to 215-3512799, or email to [email protected] 5

SECTION I. HOW DO YOU COMPARE TO YOUR PEERS?Welcome to the new Medical Knowledge Self-Assessment Program(MKSAP) Performance Interpretation Guidelines with Norm Tables. TheNORM TABLES were based on the results of pre-publication test-takerswho did not have access to the text, test answers, and critiques. The goal ofthis approach was to record test scores that realistically reflectedphysicians’ knowledge.This supplement will help you interpret your scores on the multiple-choicetests in the 16th edition of MKSAP and help you plan an individualizedprogram of continuing education. MKSAP 16 can also help you identifystrengths or weaknesses in specific subspecialty areas of internal medicine,enhance your general knowledge, keep you abreast of new developments,and prepare you for future examinations.MKSAP 16 was composed of three program components, referred to asParts A, B, and C. The first two components, Parts A and B, included Text,Bibliographies, and Multiple-Choice Questions for 11 internal medicinesubspecialty books. Part A was composed of six subspecialty test sections,and Part B was composed of five subspecialty test sections. Parts A and Balso included the Answers, Critiques, Educational Objectives, andBibliographies for Multiple-Choice Questions. Part C is this supplement,Performance Interpretation Guidelines with Norm Tables.You may also score your own tests by using the answer keys provided inSection V of this supplement and the Self-Scoring Instructions provided inSection VI.Page 6

To achieve the maximum educational benefit from Parts A and B ofMKSAP 16 you should rely on the materials listed below:1. Text and Multiple-Choice Questions2. Answers, Critiques, and Bibliographies3. Your Individual Score Reports, after submitting tests for CME credit4. This supplement, Performance Interpretation Guidelines with NormTablesPage 7

This supplement contains five additional sections.SECTION II.PARTICIPANT GROUPS.This section describesparticipants who submitted their tests for the analysis of data andnorm groups.SECTION III.SCORES.WHAT YOU NEED TO KNOW ABOUT YOURThis section describes the steps taken to ensure thevalidity of the test questions and the scores. It also describes howthe accuracy of the scores was evaluated and why this information isimportant.SECTION IV. NORM TABLES. This section contains norm data forparticipants and instructions for using the norm tables.SECTION V. ANSWER KEY AND PERCENTAGE ANSWERINGCORRECTLY.This section contains an answer key for eachsubspecialty section and defines the percentage of participantsanswering each test question correctly.SECTION VI. SELF-SCORING INSTRUCTIONS. This informationwill help you compute your percent correct score for eachsubspecialty section.Page 8

SECTION II. PARTICIPANT GROUPSBefore you interpret the statistics provided in Section III and use the NORMTABLES, consider which participants' scores were used to develop thedata. Groups were defined on the basis of information provided byparticipants in online question modules. These pre-publication participants,recruited through emails sent to ACP members and notices posted on theACP Web site, were asked to use their existing knowledge to answerquestions, without accessing clinical reference material. As part of a pretest screening process, physicians were asked to identify their primarysubspecialty and were excluded from accessing tests related to theirprimary area of expertise.SECTION III. WHAT YOU NEED TO KNOW ABOUT YOUR SCORESA. INTRODUCTIONParticipants in MKSAP 16 may note that their scores vary greatly from theNORM TABLES scores. There may be a variety of reasons for thisdiscrepancy. One, of course, is that some participants are significantly moreknowledgeable or less knowledgeable than other participants.Other factors that could contribute to score variance: Participants who read the text and soon thereafter answer the questionswill do better on the test than someone who answers the questionswithout having read the text. Those who read the test questions just before studying the text and thenanswer the questions may also score higher. Some participants take the test as a "closed book" exercise; others donot. Some participants seek consultation from content experts; others attendeducational group meetings before completing the tests. Your own scores may be influenced by such factors.Page 9

B. CONTENT VALIDITYThe validity of the scores relied on the expertise of distinguished cliniciansand educators who selected the content presented in each text section.Content in the text emphasized recent significant medical developments.Test questions reflected primarily the material presented in each textsection. After the questions were written, they passed through a series ofeditorial procedures designed to ascertain clarity, conciseness, andfreedom from technical flaws. Even after the tests were administered,questions with unusual statistical performance were reviewed again bycontent experts. Questions in these tests that were determined to beflawed, too ambiguous, outdated, erroneous, or too subspecialized havebeen excluded from scoring. These questions are not counted toward CMEcredit. The excluded questions include: Hematology and Oncology: question 114 Rheumatology: question 39 General Internal Medicine: question 7 Infectious Disease: question 97 Pulmonary and Critical Care Medicine: question 65Page 10

C. RELIABILITY (REL)It is important that test scores provide reliable estimates of ability. Thereliability (REL) coefficient was a measure of internal consistency thatprovided an estimate of the accuracy or stability of the scores. Anexamination was reliable to the extent that, if the ACP were to re-examineparticipants with a comparable form of the test, scores would be about thesame on the second test as on the first. The reliability coefficient may rangefrom 0.00 to 1.00.When scores are used to provide educational feedback for the assessmentof strengths and weaknesses, as they are for MKSAP 16, it isrecommended that the reliability be .70 or higher. The table below showsthe reliability for each subspecialty section.SUBSPECIALTY SECTIONRELCardiovascular Medicine.87Dermatology.80Gastroenterology and Hepatology.78Hematology and y and Metabolism.82General Internal Medicine.85Infectious Disease.83Nephrology.84Pulmonary and Critical Care Medicine.85Page 11

D. STANDARD ERROR OF MEASUREMENT (SEM)Ideally, the ACP would like to create tests that provide perfectly accuratemeasurements of participants' knowledge. In practice, however, scores arenever completely free of error; they are estimates of what might be called"true ability." The standard error of measurement (SEM) is an index thatindicates how much error there is in test scores.Using the SEM, it is possible to define a score interval that indicates how fara test score is likely to stray from a participant's true ability. The interval isfound by adding the SEM to a score and subtracting it from that score.Observed scores are expected to fall within one SEM of the true ability ofparticipants two times out of three, or approximately 67% of the time. Aparticipant whose true ability was 70 would usually receive a score between65 and 75 (70 - 5 and 70 5) in a test. The table below shows the SEM foreach subspecialty test.SUBSPECIALTY SECTIONSEMCardiovascular Medicine4.65Dermatology3.33Gastroenterology and Hepatology4.22Hematology and logy and Metabolism3.89General Internal Medicine5.29Infectious Disease4.58Nephrology4.55Pulmonary and Critical Care Medicine4.31Page 12

SECTION IV. NORM TABLESThe NORM TABLES are useful for determining your standing relative tothat of participants who completed tests before the program was published.Norm data are provided for each group of participants. The first columncontains PERCENT CORRECT SCORES. The remaining columns containPERCENTILE RANKS. The percentile rank indicates the percentage of thegroup that scores lower than the corresponding percent correct score.To use the NORM TABLES, first choose a percent correct score from eitheryour Individual Performance Report or your own hand score computations.Then select the table for the norm group with which you would like tocompare yourself. In the first column, locate your PERCENT CORRECTSCORE. Then read across the table to find your PERCENTILE RANK.For example, suppose that your score was 83% or 84% correct inCardiovascular Medicine. At the percent correct scores of 83-84 on theNORM TABLE, read across the page to the Cardiovascular Medicinecolumn to find your percentile rank of 97.3. Your percentile rank means that97.3 of the group scored lower than you in Cardiovascular Medicine.For each norm table, the N under the subspecialty section indicates theNumber of participants who completed tests for that subspecialty before theprogram was published.Page 13

NORM TABLES FOR MKSAP 16CardiovascularPercent CorrectMedicineScore(N 333)99 and above97-9895-9693-9491-9299.489-90Dermatology(N 246)Gastroenterology and Hematology andHepatologyOncology(N 325)(N 263)Neurology(N .90.627-2825-2623-2421-22Below 140.90.5

NORM TABLES FOR MKSAP 16 (continued)EndocrinologyandPercent Correct Rheumatology MetabolismScore(N 298)(N 170)99 and sease(N 185)99.799.5Nephrology(N 179)Pulmonary andCritical CareMedicine(N icine(N . 151.91.3

SECTION V. ANSWER KEY AND PERCENTAGE ANSWERINGCORRECTLYThis section contains an answer key for each subspecialty section and thepercentage of pre-publication test-takers who answered each testquestion correctly. As noted previously, subspecialists were excluded fromtaking tests in their primary areas of expertise to ensure a more realisticmeasurement of the general knowledge of all participants.CARDIOVASCULAR umns continued on next 677787980818283848586878889909192Page 970711442483156464739(Columns continued on next page)

CARDIOVASCULAR MEDICINE 1112113114115116117118119120Page 61556471

7172Page 69552995829478827087839138617263

GASTROENTEROLOGY AND 426985614354(Columns continued on next 990919293Page 39694768(Columns continued on next page)


HEMATOLOGY AND AACDBDA(Columns continued on next 667378893Page 9743795765041322466758965717570—731167(Columns continued on next page)

HEMATOLOGY AND ONCOLOGY 5136137138139140141142143144Page 603179

BACEBBAD(Columns continued on next 640733675Page 2498680854368(Columns continued on next page)


59647494—556256326956(Columns continued on next 990919293Page 8368506265(Columns continued on next page)


ENDOCRINOLOGY AND 263646566676869707172737475767778798081828384Page 3

GENERAL INTERNAL 5757696246890909864(Columns continued on next 117118119120121122123124125126127128129Page 6189739460(Columns continued on next page)

GENERAL INTERNAL MEDICINE 161162163164165166167168Page 918289566591587585573994265034693189746378

INFECTIOUS 45687450(Columns continued on next 596979899Page 57462647037—8068(Columns continued on next page)

INFECTIOUS DISEASE NNUMBER100101102103104105106107108Page 50

BBDDDECBC(Columns continued on next 18788044Page 66278247497164(Columns continued on next page)

NEPHROLOGY NNUMBER100101102103104105106107108Page 14

PULMONARY AND CRITICAL CARE 79563729765(Columns continued on next 596979899Page 646872117887806713(Columns continued on next page)


SECTION VI. SELF-SCORING INSTRUCTIONSCompute your percent correct score as follows:Step 1. Give yourself a point for each correct response to each questionin the test in which you are working.Step 2. Divide the number of correct responses in Step 1 by the numberof “accepted” questions for the test in which you are working. Thenumber of “accepted” questions in each test is listed in the tableat the bottom of this page.Step 3. Multiply the result by 100.The final result will be your total percent score, which you may compare tothe Norm Tables under Section IV.ExampleInfectious DiseaseStep 1: Record your points.105Step 2: Divide the result by 107, which isthe total number of “accepted” questions.105 107Step 3: Multiply result by 100.981 x 100Total percent correct scoreSubspecialty SectionCardiovascular MedicineDermatologyGastroenterology and HepatologyHematology and OncologyNeurologyRheumatologyEndocrinology and MetabolismGeneral Internal MedicineInfectious DiseaseNephrologyPulmonary and Critical Care Medicine98%Number ge 36Number ofExcludedQuestions00010101101Number ofAcceptedQuestions1207296143969584167107108107

questions with unusual statistical performance were reviewed again by content experts. Questions in these tests that were determined to be flawed, too ambiguous, outdated, erroneous, or too subspecialized have been excluded from scoring. These questions are counted toward CME not