The past few years have witnessed a dramatic expansion in the amount
of publicly available medical information on the Internet. Such information,
often targetting consumers, varies greatly in format (discussion groups,
websites), in sponsorship, and in quality. Concerned with the potential
for misinformation, a number of groups are investigating quality control
methods, to help consumers find appropriate and valid medical information
online (see references). There are two basic
approaches to the problem, which are often combined: specification of a
"code of conduct" and evaluation of medical content.
The evaluation of medical content is a much thornier problem, often addressed by "rating" schemes, which attempt to rate and make public the quality of medical information sources. Apart from subjective or expert opinion, there are efforts at developing objective criteria for evaluating the content of medical information on the Internet (for example, see Mitretek reference).
Nevertheless, the problem of evaluating medical information on the Internet
needs to be addressed. One solution, to be described here, makes use of
the Internet's capacity for interactivity to allow multi-participant evaluation
of medical resources.
Rating health information on the Internet navigating to knowledge or
Jadad A, Gagliardi A. JAMA. (1998); 279:611-4.
The Internet as a Source for Current Patient Information.
McClung HJ, et al. Pediatrics. 1998;101: E2. Abstract
The quality of information on women's health on the Internet.
Wootton J. J Womens Health. 1997;6:575-81. Abstract
Medical information on the Internet: a study of an electronic bulletin
Culver JD, et al. J Gen Intern Med. 1997;12:466-70. Abstract
Posted August 24, 1998
Author: Michael Jacobson, MD, MPH (for information about author, see Journal Club on the Web, "about" page).
Subject: quality of health information on the Web
Date: Tue, 25 Aug 1998 15:51:01 -0500
From: Myrlah Olson <email@example.com>
Organization: Minnesota Department of Health
This is an interesting topic. To prepare for a recent presentation about health information on the Web, I reviewed quite a lot of sites offering information about immunization and vaccine preventable disease. In addition to reliable sites offering credible information (HHS's Healthfinder, American Academy of Pediatrics, CDC, Minnesota Department of Health) there were many offering information that ranged from self-serving and therefore skewed (the Advil site) to blatant misinformation (anti-vaccine sites). I also looked at several review sites for health information and the HONcode site. I was left with several impressions.
The issues we are dealing with in attempting to regulate the accuracy
of health information are not new. As long as there has been health
information, there has been misinformation. In Britain three centuries
ago, it was decided that an educated populace was best able to deal with
a public health crisis of staggering proportions-outbreaks of bubonic and
pneumonic plague. Bills of mortality were published (with trepidation)
that informed the public how many people were dying in what
parts of London. At the same time, a plethora of misinformation was disseminated about causes of plague and methods of preventing transmission. As long as we believe an informed public is a healthier public, we will have misinformation.
The other impression I had from reviewing Web health resources is that, although people have to some extent learned to distinguish reliable information from misinformation, the Web changes the rules. As a general rule, I believe the public has developed visual cues that they use to decide whether printed information they read is credible. Does the brochure or flyer they are seeing look professional; that is, is it attractive, printed on good paper, and designed and written clearly. This is the reason our organizations spend money on good communications departments and printing budgets. On the Web, those same rules don't work to discern quality information. Very nearly everything looks good. Some of the anti-vaccine sites I reviewed had the clearest design and writing, the most intuitive navigation throughout the site, and the best graphics; certainly better than some of the government sites with minimal budgets. We can no longer (if we ever could) judge health information by its appearance and yet that is how most Web users determine quality. (“How users read on the Web” by Jakob Nielson).
These observations lead to no answers, of course, but it has convinced me that any effort to educate consumers of health information on the Web must include an emphasis on criteria to replace the visual cues that no longer work. JAMA's list (authorship, attribution, disclosure, currency) is a good place to start although there are other good ones. They aren't difficult concepts to communicate.
Sorry for the long ramble, but it's an interesting subject. I look forward
to hearing more discussion about a registry. My first reaction is
that it would add one more layer between me and what I want to reach. We
Web users have also become very impatient. Although I used to wait 2 or
3 days for a literature search from the library, a ten second delay at
Medline now seems interminable.
I think the "problem" of how cheap and easy it is to look professional on the Web is a major one. It is one reason why some sort of quality control is necessary on the Internet, even though nothing remotely approaching it is available for consumers browsing the "health" section of their neighborhood bookstores.
As for the registry adding another layer and delay -- if implemented properly it shouldn't have to add any time at all to your searches and browsing (depending on how the registry is implemented and how you decide to use it). -- mj
Subject: Feedback on Web Medicine Quality Control
Date: Tue, 25 Aug 1998 09:43:08 -0700
From: Beverley Kane <firstname.lastname@example.org>
Overall the document is well-worded, judicious, and realistic. I especially like the paragraph which starts "Codes of conduct are valuable guidelines and should assure a minimum of accountability..."
I believe that ultimately there are 3 sources of valid on-line medical information for the average, intelligent consumer:
1. An agency that is known personally, such as one's own doctor or clinic.
2. A state or national agency such as the Board of Public Health, the AMA, the State Medical Society, the Americian Heart Assn, etc.
3. Folksy newsgroup discussions where suggestions are low tech, low risk: like, "I use Celestial Seasonings Tummy Mint Tea for upset stomach. Works every time."
Other than that, caveat emptor!
Beverley Kane, MD
Chief Medical Officer
Philips Medical Systems
The following comments are in response to a message I posted to the
Internet Working Group mailing list. -- mj
Subject: Re: Quality of medical information on the Internet
Date: Wed, 26 Aug 1998
From: email@example.com (Gary Malet)
To: "AMIA Internet Working Group" <firstname.lastname@example.org>
>In a nutshell, I believe that centralized "rating" of medical websites
will not be successful because
>of both consumer skepticism and the difficulty of impartially judging medical content.
I am surprised that you believe that it is difficult to impartially judge medical content. Isn't the "impartial evaluation of medical content" called medical science?
I imagine that both clinicians and patients usually seek information from medical knowledge databases that are scientifically-verifiable.
They are looking for therapy and management of conditions that is experienced, evidence-based, and built on hierarchies of excellence such as consensus views of specialists and practititioners.
>As a possible solution, I suggest a medical website registry which
>collect, collate and publicize the ratings of various organizations
>along with open viewer feedback.
As a medical practitioner, who is attempting to meet the challenge of
keeping up on diverse guidelines of professional
medicine, why would I go to a registry that combined sites and comments from faith healers and homeopaths?
If I was a homeopath why would I care about randomized clinical trials
of pharmaceutical products?
>In order to be listed with the
>registry, medical internet resources would have to comply with a basic
>code of conduct (such as the HON code).
Unfortunately, the HON code does not acknowledge the hierarchy of knowledge in medical science. Perhaps medical practitioners can take hold of the evaluation of web content. I believe that they have an appropriate role to communicate their experience and study of what works. Clinicians seem to share a common view of effective medical therapy. I am not sure I see the difficulty in developing consensus quality rankings for health and medicine sites on the web with effectiveness, evidence, and excellence as measurement parameters.
I imagine that Universities, publishers, and societies will support search engines and sponsor an increasing amount of web content under well recognized quality "banners" thus making these issues less important.
Medical Matrix, www.medmatrix.org
Subject: Re: Quality of medical information on the Internet
Date: Wed, 26 Aug 1998 13:43:58 -0400
From: Michael Jacobson <email@example.com>
To: "AMIA Internet Working Group" <firstname.lastname@example.org>
Thanks, as usual, for your input. I would like to make two general points before replying to your specific comments.
No matter how the issue of quality control is approached, I believe it is useful to consider the assessment of content separate from all other issues (which are primarily issues of accountability and "bookkeeping"). These latter issues are also important but are relatively well-circumscribed, easy to define and straightforward to apply. Who sponsors a site, who is responsible for its content, how those individuals can be reached, how well-linked the references are, how changes in content are documented, how medical advice is handled -- these are all vital matters. They represent a floor, a minimum, without which the assessment of content is fairly meaningless.
The assessment of content, although obviously important, is very different from the above issues. It is likely to generate heated controversy and will be more difficult to define. Not impossible perhaps, but clearly more difficult. As such, content assessment may well benefit from a different approach than the other issues defined above. This is the reason that I believe these issues should at least conceptually be kept distinct. And this is why I consider the Health On the Net code of conduct a very valuable concept. Not that it is all we need -- of course we need to address the issue of content as well. But it does address the whole realm of non-content issues, and does it quite nicely.
As for how we should actually address the quality of medical content
on the Internet, I would suggest that we make use of one of the most valuable
properties the Internet has to offer: interactivity. The ability
to gather and disseminate different viewpoints is something the online
medium is uniquely suited to, and it is something we have not yet begun
to exploit adequately. As Steven McGeady from Intel said, at the Harvard
conference on the Internet and Society, "browsing the Web is a bit like
walking through a shopping mall that's been neutron-bombed: lots of merchandise,
but there's nobody there". Let's put some people and points of view
To respond to your points individually:
>I am surprised that you believe that it is difficult to impartially
>judge medical content. Isn't the "impartial evaluation of medical
>content" called medical science.
>I imagine that both clinicians and patients usually seek information
>from medical knowledge databases that are scientifically-verifiable.
>They are looking for therapy and management of conditions that is
>experienced, evidence-based, and built on hierarchies of excellence
>such as consensus views of specialists and practititioners.
In the October 30, 1997 issue of the New England Journal of Medicine are two articles, under the heading "Clinical Debate. Should thrombolytic therapy be the first-line treatment for acute ischemic stroke?". The first article is entitled "Thrombolysis -- not a panacea for ischemic stroke", the second is "t-PA -- the best current option for most patients". Even at the top of the evidentiary hierarchy, truth remains elusive. How are we to rate the content of a medical website that cites exclusively evidence-based articles, but does so in a biased fashion?
Bear in mind that what I'm talking about is rating the quality of websites
that present medical information primarily to consumers. Can we apply
the same criteria that we use when evaluating clinical trials reported
in peer-reviewed journals? How would you "rate" the content of an
industry sponsored website that cites multiple journal articles and states
that (example made up) "calcium channel blockers have been conclusively
shown to be safe for the treatment of hypertension and are as beneficial
as other agents"? What about if the wording is softened a bit (drop the
'conclusively'; change 'beneficial' to 'effective')? It's not impossible
to come up with criteria, but it certainly represents a quantum leap in
difficulty compared with determining whether or not a website's sponsorship
is clearly indicated.
>As a medical practitioner, who is attempting to meet the
>challenge of keeping up on diverse guidelines of professional
>medicine, why would I go to a registry that combined sites and
>comments from faith healers and homeopaths.
>If I was a homeopath why would I care about randomized clinical
>trials of pharmaceutical products.
As a patient with an open mind, if I had a problem that "conventional"
medicine wasn't treating to my satisfaction, I would certainly want to
know what options were available from sites that scored highly on, say,
acupuncture and homeopathy ratings. At the same time, I would want to be
sure that those same sites weren't accused of dangerous practices by respected
conventional medical authorities. As a physician, if I were caring
for a patient with a malignant but localized colon polyp who insisted on
taking coffee enemas, it certainly would be nice to be able to show that
a website advocating such practices was given the thumbs down by some alternative
>Unfortunately, the HON code does not acknowledge the hierarchy of
>knowledge in medical science. Perhaps medical practitioners can take
>hold of the evaluation of web content. I believe that they have an
>appropriate role to communicate their experience and study of what
>works. Clinicians seem to share a common view of effective medical
>therapy. I am not sure I see the difficulty in developing consensus
>quality rankings for health and medicine sites on the web with
>effectiveness, evidence, and excellence as measurement
>I imagine that Universities, publishers, and societies will support
>search engines and sponsor an increasing amount of web content under
>well recognized quality "banners" thus making these issues less
See my comments above on the HON approach, and on the difficulty of
judging the content of medical websites directed at consumers.
Bottom line -- I think that a non-content related code of conduct is crucial and is enforceable. I agree with you that the rating of content is also possible, but I think it is much trickier, especially when you're talking about information directed at consumers. Also, it might benefit from a multi-individual and multi-institutional, open and interactive approach.
Please excuse the length of this post.
Subject: content vs. non-content evaluation
Date: Mon, 07 Sep 1998
From: "Brian S. Alper M.D." <email@example.com>
Ultimately, what consumers are looking for on the Web site is the content.
An evaluation system that addresses links, currency, sponsorship, etc. is useful in that it can be used as a general screening to avoid Web sites that are too cumbersome to use or are more likely to have poor content.
However, the crafty huckster can be sure to meet these criteria and get the "best" rating, regardless of the quality of content.
Evaluation of content will need to be in place to make using the "evaluated
website database" useful in meeting your ultimate goal of finding content.
Brian S. Alper M.D.
Medical Director, DynaMed
Subject: proposal for website registry
Date: Tue, 08 Sep 1998
From: EDDIE VOS <firstname.lastname@example.org>
A registry using as criterion a HON type code of conduct might be useful especially in relation to the declaration of financial interests and of financial sponsoring. The database of medical websites would also be useful.
However, I am worried about the proposed "content EVALUATION" by, among others, "medical specialty societies and evidence-based medicine groups" as they may hamper dissemination of some content. Content CLASSIFICATION might be more useful because at all cost evaluation bias must be avoided if the Web is to remain useful.
I wonder how well my own webpage about cardiovascular diseases and their
link (i.e. cause and prevention) to foods, supplements and vitamins would
do when "content evaluated" by "medical specialty societies and
evidence-based medicine groups" ( http://www.health-heart.org/ ).
These are the same groups that authored the excellent August 27 1998
Heart Failure Supplement of The Lancet (Vol. 352s1) and concerning which
the Editor's discussion admits: "... that despite the advances in
management of coronary heart disease there is no clear evidence that therapeutic advances have made any impact on the overall burden of disease. ([link no longer available]).
Evidence based medicine was an unsuccessful judge as to the effects of treatment advances. Why would such groups be any better at evaluating websites relating to "unconventional" techniques.
A few years ago ulcers were not thought to be caused by a bacterium (H pylori) while the importance of Lp(a) and vitamin E in vascular diseases and of homocysteine (i.e. B vitamins and betaine) in NTD's and in cardiovascular diseases is just now becoming generally accepted. Such and other ideas need and deserve prominent outlets.
Eddie Vos, M. Eng.
Actually, one of the potential advantages of a website registry with multi-participant content evaluation (including, but not limited to, medical specialty societies and ebm groups) is the increased likelihood that non-mainstream medical sites get consideration and exposure. -- mj
Return to Journal Club home page