Transcription
Morbidity and Mortality Weekly ReportWeekly / Vol. 63 / No. 20May 23, 2014Using Online Reviews by Restaurant Patrons to Identify Unreported Cases ofFoodborne Illness — New York City, 2012–2013Cassandra Harrison, MSPH1,2, Mohip Jorder, MS3, Henri Stern3, Faina Stavinsky, MS1, Vasudha Reddy, MPH1, Heather Hanson, MPH1,HaeNa Waechter, MPH1, Luther Lowe4, Luis Gravano, PhD3, Sharon Balter, MD1 (Author affiliations at end of text)While investigating an outbreak of gastrointestinal diseaseassociated with a restaurant, the New York City Department ofHealth and Mental Hygiene (DOHMH) noted that patrons hadreported illnesses on the business review website Yelp (http://www.yelp.com) that had not been reported to DOHMH. Toexplore the potential of using Yelp to identify unreported outbreaks, DOHMH worked with Columbia University and Yelpon a pilot project to prospectively identify restaurant reviewson Yelp that referred to foodborne illness. During July 1,2012–March 31, 2013, approximately 294,000 Yelp restaurantreviews were analyzed by a software program developed forthe project. The program identified 893 reviews that requiredfurther evaluation by a foodborne disease epidemiologist. Ofthe 893 reviews, 499 (56%) described an event consistent withfoodborne illness (e.g., patrons reported diarrhea or vomitingafter their meal), and 468 of those described an illness within4 weeks of the review or did not provide a period. Only 3% ofthe illnesses referred to in the 468 reviews had also been reporteddirectly to DOHMH via telephone and online systems duringthe same period. Closer examination determined that 129 ofthe 468 reviews required further investigation, resulting in telephone interviews with 27 reviewers. From those 27 interviews,three previously unreported restaurant-related outbreaks linkedto 16 illnesses met DOHMH outbreak investigation criteria;environmental investigation of the three restaurants identifiedmultiple food-handling violations. The results suggest thatonline restaurant reviews might help to identify unreportedoutbreaks of foodborne illness and restaurants with deficiencies in food handling. However, investigating reports of illnessin this manner might require considerable time and resources.data publicly available on the website but in an XML format, andtext classification programs were trained to automatically analyzereviews. For this pilot project, a narrow set of criteria were chosen toidentify those reviews with a high likelihood of describing foodborneillness. Reviews were assessed retrospectively, using the followingcriteria: 1) presence of the keywords “sick,” “vomit,” “diarrhea,” or“food poisoning” in contexts denoting foodborne illness; 2) two ormore persons reported ill; and 3) an incubation period 10 hours.Ten hours was chosen because most foodborne illnesses are notcaused by toxins but rather by organisms with an incubation periodof 10 hours (1). Data mining software was used to train the textclassification programs (2). A foodborne disease epidemiologistmanually examined output results to determine whether reviewsselected by text classification met the criteria for inclusion, and programs with the highest accuracy rate were incorporated into the finalsoftware used for the pilot project to analyze reviews prospectively.The software program downloaded weekly data and providedthe date of the restaurant review, a link to the review, the fullINSIDE446 Rabies Death Attributed to Exposure in CentralAmerica with Symptom Onset in a U.S. DetentionFacility — Texas, 2013450 Notes from the Field: Coccidioides immitis Identifiedin Soil Outside of Its Known Range —Washington, 2013451 Notes from the Field: Trichinellosis Caused byConsumption of Wild Boar Meat — Illinois, 2013452 Announcement453 QuickStatsProject ProtocolBeginning in April 2012, Yelp provided DOHMH with a privatedata feed of New York City restaurant reviews. The feed providedContinuing Education examination available athttp://www.cdc.gov/mmwr/cme/conted info.html#weekly.U.S. Department of Health and Human ServicesCenters for Disease Control and Prevention
Morbidity and Mortality Weekly Reportreview text, establishment name, establishment address, andscores for each of three outbreak criteria (i.e., keywords, numberof persons ill, and incubation period), plus an average of the threecriteria. Scores for individual criteria ranged from 0 to 1, with ascore closer to 1 indicating the review likely met the score criteria.Reviews submitted to Yelp during July 1, 2012–March 31,2013 were analyzed. All reviews with an average review scoreof 0.5 were evaluated by a foodborne disease epidemiologist(Figure). Because the average review score was calculated byaveraging the individual criteria scores, reviews could receivean average score of 0.5 without meeting all individual criteria.Reviews with an average review score of 0.5 were evaluatedfor the following three criteria: 1) consistent with foodborneillness occurring after a meal, rather than an alternative explanation for the illness keyword; 2) meal date within 4 weeks ofreview (or no meal date provided); 3) two or more persons illor a single person with symptoms of scombroid poisoning orsevere neurologic illness. Reviews that met all three of thesecriteria were then investigated further by DOHMH. In addition, reviews were investigated further if manual checkingidentified multiple reviews within 1 week that described recentfoodborne illness at the same restaurant.To identify previously reported complaints of foodborneillness, reviews were compared with complaints reported toDOHMH by telephone or online at 311, New York City’s nonemergency information service that can be used by the public toreport suspected foodborne illness (3). Yelp reviews categorizedas indicating recent or potentially recent illness were comparedwith complaints from the previous 4 weeks in the 311 database.To follow up with reviewers, DOHMH created a Yelp accountto send private messages to reviewers’ Yelp accounts. Reviewersneeded to log in at Yelp to view their messages.For reviews not requiring further investigation and not foundin the 311 database, DOHMH sent messages advising reviewersof the availability of 311 reporting. For reviews requiring furtherinvestigation, DOHMH sent messages requesting telephoneinterviews. Reviewers consenting to interviews were askedto provide details about the restaurant visit, meal date, foodsconsumed during the meal, party size, illness symptoms, and ahistory of foods consumed in the 3 days before symptom onset.Review-Based FindingsDuring July 1, 2012–March 31, 2013, the software systemscreened approximately 294,000 reviews and identified 893with an average score of 0.5, indicating possible foodborneillness (Figure). Of these reviews, 499 (56%) described an eventconsistent with foodborne illness, as determined by the manualchecking of a foodborne epidemiologist. This equated to anaverage of 23 reviews evaluated by a foodborne epidemiologist each week, with an average of 13 reviews categorized asconsistent with foodborne illness. The remaining 394 (44%)reviews contained keywords but did not suggest foodborneillness (e.g., “I didn’t get sick at all after my meal”).The MMWR series of publications is published by the Center for Surveillance, Epidemiology, and Laboratory Services, Centers for Disease Control and Prevention (CDC),U.S. Department of Health and Human Services, Atlanta, GA 30329-4027.Suggested citation: [Author names; first three, then et al., if more than six.] [Report title]. MMWR 2014;63:[inclusive page numbers].Centers for Disease Control and PreventionThomas R. Frieden, MD, MPH, DirectorHarold W. Jaffe, MD, MA, Associate Director for ScienceJoanne Cono, MD, ScM, Director, Office of Science QualityChesley L. Richards, MD, MPH, Deputy Director for Public Health Scientific ServicesMichael F. Iademarco, MD, MPH, Director, Center for Surveillance, Epidemiology, and Laboratory ServicesMMWR Editorial and Production Staff (Weekly)Charlotte K. Kent, PhD, MPH, Acting Editor-in-ChiefJohn S. Moran, MD, MPH, EditorTeresa F. Rutledge, Managing EditorDouglas W. Weatherwax, Lead Technical Writer-EditorDonald G. Meadows, MA, Jude C. Rutledge, Writer-EditorsMMWR Editorial BoardMartha F. Boyd, Lead Visual Information SpecialistMaureen A. Leahy, Julia C. Martinroe,Stephen R. Spriggs, Terraye M. StarrVisual Information SpecialistsQuang M. Doan, MBA, Phyllis H. KingInformation Technology SpecialistsWilliam L. Roper, MD, MPH, Chapel Hill, NC, ChairmanMatthew L. Boulton, MD, MPH, Ann Arbor, MITimothy F. Jones, MD, Nashville, TNVirginia A. Caine, MD, Indianapolis, INRima F. Khabbaz, MD, Atlanta, GABarbara A. Ellis, PhD, MS, Atlanta, GADennis G. Maki, MD, Madison, WIJonathan E. Fielding, MD, MPH, MBA, Los Angeles, CAPatricia Quinlisk, MD, MPH, Des Moines, IADavid W. Fleming, MD, Seattle, WAPatrick L. Remington, MD, MPH, Madison, WIWilliam E. Halperin, MD, DrPH, MPH, Newark, NJWilliam Schaffner, MD, Nashville, TNKing K. Holmes, MD, PhD, Seattle, WA442MMWR / May 23, 2014 / Vol. 63 / No. 20
Morbidity and Mortality Weekly ReportFIGURE. Results of investigation of online reviews by restaurant patrons that referred to possible foodborne illness — pilot project, New York City,July 1, 2012–March 31, 2013 294,000reviews were downloadedfrom Yelp weeklyand scored by a customsoftware program forlikelihood of foodborneillness893reviews received a score 0.5and were then evaluated by afoodborne epidemiologist499reviews were consistent withfoodborne illness; these werethen assessed to determinewhether illness was recent468reviews either describedillness within 4 weeks of thereview or provided no period;these were then assessed fornumber of ill persons129reviewers described two ormore ill persons or one personwith scombroid poisoning orsevere neurologic illness;DOHMH sent messages toreviewers requesting phoneinterviews27reviewers wereinterviewed byDOHMHreviews with a score 0.5 were notinvestigated further394reviews were not consistentwith foodborne illness andwere not investigatedfurther31reviews described illness 4weeks before the reviewand were not investigatedfurther339reviews described one personill without scombroid poisoningor severe neurologic illness;reviews were not investigatedfurther, but DOHMH sentmessage to the reviewersregarding 311 reporting102reviewers did notaccept theinterviewer’s request3interview results met DOHMHoutbreak criteria, andenvironmental investigationswere conductedAbbreviation: DOHMH Department of Health and Mental Hygiene.MMWR / May 23, 2014 / Vol. 63 / No. 20443
Morbidity and Mortality Weekly ReportOf the 499 reviews describing an event consistent with foodborne illness, 468 (94%) indicated recent or potentially recentillness. Of these 468 reviews, only 15 (3%) were also reportedto 311 during the same period. A total of 339 reviews thatindicated only one person became ill and had no scombroidpoisoning or severe neurologic symptoms were excluded, leaving 129 reviews that required further investigation (Figure). Ofthe 129, a total of 27 (21%) reviewers completed a telephoneinterview inquiring about meals and illnesses. The median timefrom review date to DOHMH contact to schedule a telephoneinterview was 8 days. The interviews provided information on27 restaurants, and 24 restaurants were identified as potentiallocations of recent exposure because the meal dates were within4 weeks of the interview.From the 27 interviews, DOHMH determined whether thecomplaints warranted an outbreak investigation by considering the following criteria: 1) more than one person becameill, 2) no other common meals were suspected, 3) ill personslived in different households, and 4) the cases had similaronset periods (indicating a likely foodborne cause rather thanperson-to-person transmission). For scombroid poisoning orneurologic symptoms, DOHMH considered whether symptoms and onset were consistent with scombrotoxin, ciguateratoxin, or botulism poisoning.Three outbreaks meeting DOHMH outbreak investigation criteria were identified, accounting for 16 illnesses notpreviously reported to DOHMH. Interviews with reviewersidentified likely food items associated with illness at each of thethree restaurants: house salad, shrimp and lobster cannelloni,and macaroni and cheese spring rolls (Table). The reviews ofthe three restaurants had been posted on Yelp 2–5 days afterthe meals. Environmental investigations were conducted at twoof the three restaurants during the week after the interviews; aroutine DOHMH inspection had already been conducted atthe other restaurant 2 days after the meal. The two investigations and the routine inspection identified multiple violationsat each of the outbreak restaurants (Table). Investigators wereunable to obtain laboratory data that might have identifiedthe infectious agents.DiscussionIn a New York City DOHMH pilot project, of 468 recent orpotentially recent online foodborne illness complaints posted onYelp and reviewed by foodborne epidemiologists, three previously unreported restaurant outbreaks were identified. Becausefoodborne cases have a common exposure, a restaurant patronreview-based system can identify small, point-source outbreaksthat are not easily found by systems reviewing large sources ofdata, such as syndromic surveillance of emergency departmentvisits (4), Google Flu Trends (5), and analysis of Twitter datafor influenza and other public health trends (6–8). Most importantly, foodborne epidemiologists can confirm reports becauseYelp offers a way to follow-up with reviewers for interview.In this project, only 15 (3%) of the 468 recent or potentiallyrecent illnesses identified on Yelp were also reported directlyto New York City’s nonemergency 311 service, suggesting thatknowledge about 311 reporting is limited. Of further note,after messages regarding the availability of 311 were sent to 290reviewers who did not meet the project criteria, 32 responded,of whom 25 (78%) said they were unaware of the 311 system orwould keep 311 in mind for the future. The 311 service receivesapproximately 3,000 food poisoning complaints each year, andfrom that number, about 1% are identified as outbreak-related(DOHMH, unpublished data, 2014).As social media usage continues to grow among U.S. adults(9), health departments might consider additional surveillanceTABLE. Unreported outbreaks of foodborne illness identified by investigation of online restaurant patron reviews — pilot project, New York City,July 1, 2012–March 31, 2013OutbreakMonth of mealLikely foodvehicleOutbreak ADecember 2012House salad7/9Environmental investigationand food preparation reviewconducted in response tointerview with reviewerCross-contamination in refrigeratorBare-hand contact with ready-to-eat foodImproperly sanitized work surfacesNo washing of ready-to-eat vegetablesOutbreak BJanuary 2013Shrimp andlobstercannelloni3/5Routine inspectionconducted 2 days aftermealImproper cold food storageImproper thawing proceduresFood contact surface not maintained properlyFood dispensing utensils stored improperlyMouse activity presentLive roaches presentOutbreak CMarch 2013Macaroni andcheese springrolls6/6Environmental investigationand food preparation reviewconducted in response tointerview with reviewerBare-hand contact with ready-to-eat foodCold storage temperatures not taken duringcold holding of pre-prepared food444No. of persons ill/No. in reviewer’s partyMMWR / May 23, 2014 / Vol. 63 / No. 20Public health actionEnvironmental findings
Morbidity and Mortality Weekly ReportWhat is already known on this topic?Health departments rely on the public to report restaurant-relatedfoodborne illness directly to them, yet many outbreaks gounreported. A large amount of publicly reported informationabout foodborne illness is available on restaurant review websites.What is added by this report?During a 9-month period, approximately 294,000 reviews ofNew York City restaurants posted on Yelp.com were screened bysoftware programs for possible cases of foodborne illness. Thesoftware flagged 893 reviews for evaluation by an epidemiologist, resulting in the identification of 468 reviews that wereconsistent with recent or potentially recent foodborne illness.Only 15 (3%) of these reviews described events that had beenreported to the health department. After further evaluation ofreviews and interviews with 27 reviewers, three previouslyunreported restaurant-related outbreaks were identified.What are the implications for public health practice?Review websites might be a valuable source of data in thepublic health setting. Restaurant patron reviews can helpidentify small, point-source outbreaks of foodborne illnessbecause cases have a known common exposure. Such reviewsmight be particularly useful if the website offers a way to reachreviewers for follow-up interviews.methods to capture illness reports from those more likelyto post a restaurant review online than to contact a healthdepartment. By incorporating website review data into publichealth surveillance programs, health departments might findadditional illnesses and improve detection of foodborne disease outbreaks in the community. Similar programs could bedeveloped to identify other public health hazards that reviewersmight describe, such as vermin in food establishments.The findings in this report are subject to at least four limitations.First, to increase the likelihood of identifying true foodborne illness, a narrow focus was chosen for the individual criteria usedto score reviews. Therefore, it is possible that some foodborneillnesses were not picked up by the screening software because oflow average review scores (e.g., because of illnesses resulting fromtoxins with short incubation periods). Second, personal contactinformation for reviewers was unavailable, requiring reviewersto check their Yelp accounts and provide a telephone number toparticipate, which extended the time from review to interviewand might have affected the response rate. Third, investigatorswere not able to identify any of the infectious agents in theoutbreaks. Finally, the system required substantial resources; inaddition to programming expertise, staff members were neededto read reviews, send e-mails, interview reviewers, and performfollow-up inspections.Additional work using social media might improve healthdepartment abilities to use the Internet for disease detection.Working with the Chicago Department of Public Health, theSmart Chicago Collaborative recently developed a system tocontact those who post foodborne illness complaints either onits website or on Twitter.* For health departments looking for analternative to analyzing review data weekly, creating an illnessreporting vehicle such as the Utah Department of Health’s “I GotSick” website (10) could be a more practical solution, althoughit might be less widely used than a review website such as Yelp.Review websites could assist by offering a link to the reviewer’s localhealth department’s reporting system at the time of review posting.DOHMH plans to continue to refine this project. To shortenthe time from review to investigation, Yelp will provide dailyinstead of weekly review feeds, and, to increase sensitivity,the project will be expanded to include additional reviewwebsites. To improve response rates, DOHMH will offer alink to an electronic survey. Finally, DOHMH is exploringthe possibility of linking multiple complaints pertaining tothe same restaurant, using data from different review websitesand DOHMH databases.* Available at https://foodborne.smartchicagoapps.org.1New York City Department of Health and Mental Hygiene; 2CDC/CSTEApplied Epidemiology Fellow; 3Columbia University; 4Yelp (Correspondingauthor: Vasudha Reddy, [email protected], 347-396-2676)References1. Scallan E, Hoekstra RM, Angulo FJ, et al. Foodborne illness acquired inthe United States—major pathogens. Emerg Infect Dis 2011;17:7–15.2. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH. TheWEKA data mining software: an update. ACM SIGKDD Explorations2009;11:10–8.3. The City of New York. NYC 311. New York, NY: The City of New York;2014. Available at http://www.nyc.gov/311.4. CDC. Three years of emergency department gastrointestinal syndromicsurveillance in New York City: what have we found? In: Syndromic surveillance:reports from a national conference, 2004. MMWR 2005;54(Suppl):175–80.5. Carneiro H, Mylonakis E. Google Trends: a web-based tool for real-timesurveillance of disease outbreaks. Clin Infect Dis 2009;49:1557–64.6. Culotta A. Towards detecting influenza epidemics by analyzing Twitter messages.In: Proceedings of the First Workshop on Social Media Analytics, July 25, 2010;Washington, DC. New York, NY: Association for Computing Machinery; 2010.Available at http://snap.stanford.edu/soma2010/papers/soma2010 16.pdf.7. Paul M, Dredze M. You are what you tweet: analyzing Twitter for publichealth. In: Proceedings of the Fifth International AAAI Conference on Weblogsand Social Media, July 17–21, 2011; Barcelona, Spain. Palo Alto, CA: AAAIPress; 2011:265–72.8. Sadilek A, Brennan S, Kautz H, Silenzio V. nEmesis: which restaurantsshould you avoid today? In: Proceedings of the First AAAI Conference onHuman Computation and Crowdsourcing, November 6–9, 2013; PalmSprings, California. Available at per/viewFile/7475/7413.9. Fox S, Rainie L, Pew Research Center. Pew Internet research project: the Webat 25 in the U.S.; 2014. Available at 5-in-the-u-s.10. Utah Department of Health. Report a foodborne illness. Salt Lake City,UT: State of Utah, Utah Department of Health; 2012. Available at http://igotsick.health.utah.gov.MMWR / May 23, 2014 / Vol. 63 / No. 20445
text classification programs were trained to automatically analyze reviews. For this pilot project, a narrow set of criteria were chosen to identify those reviews with a high likelihood of describing foodborne illness. Reviews were assessed retrospectively, using the following criteria: 1) presence of the keywords "sick," "vomit," "diarrhea," or "food poisoning" in contexts .
![Restaurant Management Software Crack For Windows [Latest]](/img/41/restaurant-management-software-crack-download-for-windows-latest.jpg)