M. Craigie
Reliability of health information on the Internet: An examination of experts’ ratings
Craigie, M.; Loader, B.; Burrows, R.; Muncer, S.J.
Authors
B. Loader
R. Burrows
S.J. Muncer
Abstract
Background: The use of medical experts in rating the content of health-related sites on the Internet has flourished in recent years. In this research, it has been common practice to use a single medical expert to rate the content of the Web sites. In many cases, the expert has rated the Internet health information as poor, and even potentially dangerous. However, one problem with this approach is that there is no guarantee that other medical experts will rate the sites in a similar manner. Objectives: The aim was to assess the reliability of medical experts' judgments of threads in an Internet newsgroup related to a common disease. A secondary aim was to show the limitations of commonly-used statistics for measuring reliability (eg, kappa). Method: The participants in this study were 5 medical doctors, who worked in a specialist unit dedicated to the treatment of the disease. They each rated the information contained in newsgroup threads using a 6-point scale designed by the experts themselves. Their ratings were analyzed for reliability using a number of statistics: Cohen's kappa, gamma, Kendall's W, and Cronbach's alpha. Results: Reliability was absent for ratings of questions, and low for ratings of responses. The various measures of reliability used gave conflicting results. No measure produced high reliability. Conclusions: The medical experts showed a low agreement when rating the postings from the newsgroup. Hence, it is important to test inter-rater reliability in research assessing the accuracy and quality of health-related information on the Internet. A discussion of the different measures of agreement that could be used reveals that the choice of statistic can be problematic. It is therefore important to consider the assumptions underlying a measure of reliability before using it. Often, more than one measure will be needed for "triangulation" purposes.
Citation
Craigie, M., Loader, B., Burrows, R., & Muncer, S. (2002). Reliability of health information on the Internet: An examination of experts’ ratings. Journal of Medical Internet Research, 4(1), https://doi.org/10.2196/jmir.4.1.e2
Journal Article Type | Article |
---|---|
Publication Date | 2002-01 |
Deposit Date | Apr 3, 2007 |
Journal | Journal of Medical Internet Research |
Publisher | JMIR Publications |
Peer Reviewed | Peer Reviewed |
Volume | 4 |
Issue | 1 |
DOI | https://doi.org/10.2196/jmir.4.1.e2 |
Keywords | Newsgroup, Internet, Rating information, Reliability, reproducibility of results, Statistics, Quality control. |
Publisher URL | http://www.jmir.org/2002/1/e2 |
You might also like
Meta-analysis and power: Some suggestions for the use of power in research synthesis
(2003)
Journal Article
Power dressing and meta-analysis: Incorporating power analysis into meta-analysis
(2002)
Journal Article
Predicting length of stay in hospital after brain injury
(2002)
Journal Article
Downloadable Citations
About Durham Research Online (DRO)
Administrator e-mail: dro.admin@durham.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search