Trying to Measure the Quality of Health Information on the Internet: Is It Time to Move On?

AMOL DESHPANDE, MD, MBA,
Consultant, Comprehensive Pain Program,
University Health Network and University of Toronto;
ALEJANDRO R. JADAD, MD, DPhil, FRCPC, FCAHS,
Chief Innovator and Founder, Centre for Global eHealth Innovation,
Canada Research Chair in eHealth Innovation,
Rose Family Chair in Supportive Care,
Professor, Departments of Public Health Sciences;
Health Policy, Management and Evaluation; and Anesthesia,
University Health Network and University of Toronto,
Toronto, Canada

Address reprint requests to Dr. A. Deshpande, Comprehensive Pain Program, Room 811-4-FP, Toronto Western Hospital, 399 Bathurst Street, Toronto, ON M5T 2S8; E-mail: amol.d@rogers.com

---

Be careful about reading health books. You may die of a misprint. — Mark Twain

CHRONICLING THE JOURNEY

The Internet has forever changed the way we access and obtain health information. In 2006, the Pew Internet & American Life Project estimated that almost 80% of Internet users had searched for online health information, with medical diseases, health problems, medical treatment, and nutrition being the most popular topics¹. Yet, for almost 2 decades, ever since Tim Berners-Lee introduced the World Wide Web to the public, there has continued to be an underlying concern about the quality of health information available online.

The natural assumption is to believe that there exists a link between the quality of information on the Internet and harm. However, a systematic review attempting to evaluate the number and characteristics of reported cases of harm in the peer-review literature determined that for a variety of reasons, there was little evidence to support this notion². Nonetheless, considerable resources continue to be spent on developing and disseminating quality assessment tools to evaluate online health information.

Five broad categories could be used to classify assessment instruments to evaluate the quality of online health information: codes of conduct (e.g., American Medical Association), quality labels [e.g., Health On the Net Foundation (HON)], user guides (e.g., DISCERN), filters (e.g., intute.ac.uk), and third-party certification [e.g., Utilization Review Accreditation Commission (URAC)]³. It has even been suggested that the nature of the site domain (i.e., .gov or .edu) could be associated with higher quality rankings⁴. However, others have questioned this assumption⁵. In fact, the debate as to what determines quality is so tenuous that the sheer number of instruments to accomplish this task has exploded from 58 to over 270 in less than a decade^6-8. Measurement of the quality of health information on the Internet, some have argued, might be a futile pursuit after all⁹.

In this issue of The Journal, Thompson and Graydon report on yet another attempt to validate and propose an assessment instrument designed for health professionals to specifically evaluate online information for methotrexate use in arthritis¹⁰. This tool, the Medication Website Assessment Tool (MWAT), included well-known criteria such as completeness and accuracy, format, reliability, and readability of the information. The categories were adapted from the US Department of Health and Human Services Steering Committee's "action plan" for evaluating and improving the usefulness of written medical information for printed documents. The authors tested the MWAT by evaluating 24 Web pages from 23 websites identified through the search engine Google. They identified 8 websites with content and format that received more than 75% of the total possible score. Most sites lost content points for lack of information on storage, lactation, and overdose, while formatting deficiencies included lack of bold text and highlighting. Reliability of the MWAT was determined based on concordance rates between the 2 authors, while external validity was assessed by comparing scores on the MWAT to those derived from the Healthcare Website Assessment Tool (HWAT 3.0)¹¹. Comparison of the 2 instruments yielded a similar ranking for 7 of the top 8 websites. The authors concluded that the MWAT could be valuable but called for further evaluation.

In the end, the MWAT, similar to its predecessors, suffers from several limitations, which, in addition to those mentioned by the authors, include uncertain levels of usability, reliability, and validity. The value of the instrument, at this time, remains unclear.

IS IT TIME TO SET A NEW DIRECTION?

Will we ever develop an ideal tool that allows individuals to assess the quality of health information? What are the determinants of this quality? Is it possible to assess or measure quality? Even if possible, is the formal assessment of quality even necessary? Does it even matter? Implicit answers to these questions might already be emerging along the Web's evolutionary path.

During the first decade of the 21st century, we have witnessed the birth of Web 2.0, heralded by a set of powerful tools designed to enhance creativity, information sharing, collaboration, and functionality of the Web, and to transfer power to the end user¹². This objective is in marked contrast to the original Web, Web 1.0, which was characterized by a hierarchical structure (ruled by Webmasters) offering static websites broadcast and distributed mostly through hypertext links.

The use of Web 2.0 technology to manage health information has given rise to the term "Medicine 2.0" (or Health 2.0), which reflects the use of a specific set of web tools (blogs, Podcasts, tagging, search, Wikis, etc.) by health professionals, researchers, and the public, to generate content, to personalize health services, to collaborate, and to promote multidirectional, interactive health educational experiences¹³. One of the overlying premises of this new era is that users determine the quality of content through a collective "bottom-up" approach (rather than "top-down") that reflects their needs, knowledge, and real-life experiences.

A key characteristic of Medicine 2.0 that enables this user-driven process is apomediation, the ability to disintermediate the creation, evaluation, and exchange of information, while still retaining an "expert source" to help guide an individual's online experience. The "expert source" known as an apomediary could be a person, a group, or even a technological tool that helps point users to information sources deemed to be relevant, credible, and trustworthy¹⁴. For example, Internet users could provide ratings or recommendations based on their own experiences to judge the quality and relevance of health information. Analogous to the peer-review process, aggregation of ratings from many individuals (a form of crowdsourcing) allows "good" information to be highlighted prominently, while "not so good" information gets pushed to the bottom. Alternatively, collaborative filtering systems, such as those used by Amazon.com, provide users with relevant information based on their own experiences and those of others with similar profiles. For health-related applications this could include matching patients or health providers that share the same disease process, treatment, or practice populations.

The interplay of users to collaborate and deal with information overload has already been proven successful in other areas outside the health space. For example, Wikipedia not only allows users to submit content on various topics, but also provides the capability for users to edit the content of others. Although there is the potential for misuse, Wikipedia, which relies on anonymous, unpaid volunteers, seems to be as accurate in covering scientific information as the Encyclopedia Britannica¹⁵. The ability of large numbers of individuals to manage huge volumes of online information has not been limited to Wikipedia. A study of the unmoderated Breast Cancer Mailing List noted that of 4600 postings by survivors and their loved ones, only 10 were found to be misleading or false. Of these 10, seven were corrected within 5 hours of the original posting¹⁶. The ability of multitudes to generate accurate information from diverse data sets has been well documented elsewhere and is not unique to Web 2.0¹⁷. However, as the Web continues to evolve, we will likely gain new insights as to how this happens along with a better understanding of how to handle health information from any source.

EXPLORING UNCHARTED WATERS

Since its inception in 1990 until the present day, the health system has grappled with how to manage potential harm associated with information available on the Internet. Research in this area, for the most part, continues to assume that techniques used to evaluate paper-based information can automatically be applied to online resources, ignoring the added complexity created by the multiple media formats, players, and channels that are brought together by the Internet. Studies continue to presume that the measurement of quality is possible and could contribute to "safeguard the public." Researchers, for their part, continue to view the role of health professionals as "providers and protectors," able to control, or in some way "regulate" the types and amount of information that patients, their loved ones, or the public should or could access. This contrasts sharply with what is happening today in the real world. Groups, mainly led by patients, are now beginning to take matters into their own hands to address problems that the health system has continued to ignore. Instead of conforming to the traditional asymmetrical offline patient-physician relationship, the public is embracing the tenets of Web 2.0, opening new horizons for a level playing field and improved health services.

The time has likely come to end our Byzantine discussions about whether and how to measure the quality of online health information. The public has moved on. It is time to join them in what promises to be an exciting voyage of human fellowship, with new discoveries and exciting ways to achieve optimal levels of health.

REFERENCES

Search PubMed for:

1. Fox S. Online health search 2006. Pew Internet & American Life Project, October 29, 2006. Washington, DC. [Internet. Accessed October 23, 2008.] Available from: http://www.pewInternet.org/pdfs/PIP_Online_Health_2006.pdf

2. Crocco AG, Villasis-Keever M, Jadad AR. Analysis of cases of harm associated with use of health information on the Internet. JAMA 2002;287:2869-71. [MEDLINE]

3. Wilson P, Risk A. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the Internet [review]. BMJ 2002;324:598-602. [MEDLINE]

4. Ansani NT, Vogt M, Henderson BA, et al. Quality of arthritis information on the Internet. Am J Health Syst Pharm 2005;62:1184-9. [MEDLINE]

5. Kunst H, Groot D, Latthe PM, Latthe M, Khan KS. Accuracy of information on apparently credible websites: survey of five common health topics. BMJ 2002;324:581-2. [MEDLINE]

6. Jadad AR, Gagliardi A. Rating health information on the Internet: navigating to knowledge or to Babel? JAMA 1998;279:611-4. [MEDLINE]

7. Gagliardi A, Jadad AR. Examination of instruments used to rate quality of health information on the Internet: chronicle of a voyage with an unclear destination. BMJ 2002;324:569-73. [MEDLINE]

8. Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F. Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? Int J Med Inform 2005;74:13-9. [MEDLINE]

9. The invention of talk. BMJ 2002;324 Mar 9:0/j.

10. Thompson AE, Graydon SL. Patient oriented methotrexate information sites on the Internet: A review of completeness, accuracy, format, reliability, credibility, and readability. J Rheumatol 2009;36:41-9.

11. Lewiecki EM, Rudolph LA, Kiebzak GM, Chavez JR, Thorpe BM. Assessment of osteoporosis-website quality. Osteoporos Int 2006;17:741-52. [MEDLINE]

12. Web 2.0. Wikipedia, the free encyclopedia. [Internet. Accessed October 23, 2008.] Available from: http://en.wikipedia.org/w/index.php?title=Web_2.0&oldid=247168362

13. Hughes B, Joshi I, Wareham J. Health 2.0 and medicine 2.0: tensions and controversies in the field. J Med Internet Res 2008;10:e23.[MEDLINE]

14. Eysenbach G. Medicine 2.0: social networking, collaboration, participation, apomediation, and openness. J Med Internet Res 2008;10:e22.[MEDLINE]

15. Giles J. Wikipedia rival calls in the experts. Nature 2006;443:493. [MEDLINE]

16. Esquivel A, Meric-Bernstam F, Bernstam EV. Accuracy and self correction of information received from an Internet breast cancer list: content analysis. BMJ 2006;332:939-42. [MEDLINE]

17. Suroweicki J. Wisdom of the crowds. New York: Anchor Books; 2005.