Quality control on the medical Internet: proposal for a website registry

Michael Jacobson, MD, MPH
 

The past few years have witnessed a dramatic expansion in the amount of publicly available medical information on the Internet. Such information, often targetting consumers, varies greatly in format (discussion groups, websites), in sponsorship, and in quality. Concerned with the potential for misinformation, a number of groups are investigating quality control methods, to help consumers find appropriate and valid medical information online (see references).  There are two basic approaches to the problem, which are often combined: specification of a "code of conduct" and evaluation of medical content.
 

Codes of conduct are valuable guidelines and should assure a minimum of accountability. On the other hand, centralized attempts at rating the quality of medical information on the Internet are unlikely to succeed. Consumers who turn to the Internet for information (medical or other), are often looking for independent sources and tend to be skeptical of central authorities. Even if such a system were to gain consumer acceptance, its implementation is questionable.  With vast sums of money at stake for pharmaceutical companies and provider organizations, and with so much clinical uncertainty in medicine, any attempt at evaluating medical content by a single organization is likely to become mired in acrimonious discussions. One need only think of the "calcium channel blocker" controversy, and imagine attempting to rate a pharmaceutical industry-sponsored website that provides consumer information about the treatment of hypertension.

Nevertheless, the problem of evaluating medical information on the Internet needs to be addressed. One solution, to be described here, makes use of the Internet's capacity for interactivity to allow multi-participant evaluation of medical resources.
 
 

Proposal for a website registry system

  The advantage of this system is that it makes use of one of the most powerful features that the Internet has to offer -- the ability to collate information and feedback from multiple sources. The lack of reliance on a single rating authority obviates the need for the development of all-inclusive quality criteria and decentralizes the rating process (any centralized rating system would give too much power and authority to the rating organization).
 
 
 
Send feedback for posting to this page  (see comments below)
 

 


References

Overview of the problem:
An attempt to objectify the quality evaluation of medical information on the Internet: Health on the Net code of conduct:
  Other articles:
 

Posted August 24, 1998
Author: Michael Jacobson, MD, MPH (for information about author, see Journal Club on the Web, "about" page).
 




 

Feedback


Subject:  quality of health information on the Web
Date: Tue, 25 Aug 1998 15:51:01 -0500
From:  Myrlah Olson <myrlah.olson@sunny.health.state.mn.us>
Organization:  Minnesota Department of Health
 

This is an interesting topic.  To prepare for a recent presentation about health information on the Web, I reviewed quite a lot of sites offering information about immunization and vaccine preventable disease.  In addition to reliable sites offering credible information (HHS's Healthfinder, American Academy of Pediatrics, CDC, Minnesota Department of Health) there were many offering information that ranged from self-serving and therefore skewed (the Advil site) to blatant misinformation (anti-vaccine sites).  I also looked at several review sites for health information and the HONcode site.  I was left with several impressions.

The issues we are dealing with in attempting to regulate the accuracy of health information are not new.  As long as there has been health information, there has been misinformation.  In Britain three centuries ago, it was decided that an educated populace was best able to deal with a public health crisis of staggering proportions-outbreaks of bubonic and pneumonic plague.  Bills of mortality were published (with trepidation) that informed the public how many people were dying in what
parts of London.  At the same time, a plethora of misinformation was disseminated about causes of plague and methods of preventing transmission.  As long as we believe an informed public is a healthier public, we will have misinformation.

The other impression I had from reviewing Web health resources is that, although people have to some extent learned to distinguish reliable information from misinformation, the Web changes the rules.  As a general rule, I believe the public has developed visual cues that they use to decide whether printed information they read is credible.  Does the brochure or flyer they are seeing look professional; that is, is it attractive, printed on good paper, and designed and written clearly.  This is the reason our organizations spend money on good communications departments and printing budgets.  On the Web, those same rules don't work to discern quality information.  Very nearly everything looks good.  Some of the anti-vaccine sites I reviewed had the clearest design and writing, the most intuitive navigation throughout the site, and the best graphics;  certainly better than some of the government sites with minimal budgets.  We can no longer (if we ever could) judge health information by its appearance and yet that is how most Web users determine quality. (“How users read on the Web” by Jakob Nielson).

These observations lead to no answers, of course, but it has convinced me that any effort to educate consumers of health information on the Web must include an emphasis on criteria to replace the visual cues that no longer work.  JAMA's list (authorship, attribution, disclosure, currency) is a good place to start although there are other good ones.  They aren't difficult concepts to communicate.

Sorry for the long ramble, but it's an interesting subject. I look forward to hearing more discussion about a registry.  My first reaction is that it would add one more layer between me and what I want to reach. We Web users have also become very impatient. Although I used to wait 2 or 3 days for a literature search from the library, a ten second delay at Medline now seems interminable.
 
 

I think the "problem" of how cheap and easy it is to look professional on the Web is a major one. It is one reason why some sort of quality control is necessary on the Internet, even though nothing remotely approaching it is available for consumers browsing the "health" section of their neighborhood bookstores.

As for the registry adding another layer and delay -- if implemented properly it shouldn't have to add any time at all to your searches and browsing (depending on how the registry is implemented and how you decide to use it).  -- mj


Subject:  Feedback on Web Medicine Quality Control
Date:  Tue, 25 Aug 1998 09:43:08 -0700
From: Beverley Kane <bkane@medgrid.philips.com>
 

Michael -

Overall the document is well-worded, judicious, and realistic. I especially like the paragraph which starts "Codes of conduct are valuable guidelines and should assure a minimum of accountability..."

I believe that ultimately there are 3 sources of valid on-line medical information for the average, intelligent consumer:

1. An agency that is known personally, such as one's own doctor or clinic.

2. A state or national agency such as the Board of Public Health, the AMA, the State Medical Society, the Americian Heart Assn, etc.

3. Folksy newsgroup discussions where suggestions are low tech, low risk: like, "I use Celestial Seasonings Tummy Mint Tea for upset stomach. Works every time."

Other than that, caveat emptor!
 

Beverley
 

Beverley Kane, MD
Chief Medical Officer
MedGRiD Program
Philips Medical Systems
 


The following comments are in response to a message I posted to the Internet Working Group mailing list.  -- mj
 

Subject: Re: Quality of medical information on the Internet
Date: Wed, 26 Aug 1998
From: gmalet@surfer.win.net (Gary Malet)
To: "AMIA Internet Working Group" <int-wg@mail.amia.org>

>In a nutshell, I believe that centralized "rating" of medical websites will not be successful because
>of both consumer skepticism and the difficulty of impartially judging medical content.

I am surprised that you believe that it is difficult to impartially judge medical content. Isn't the "impartial evaluation of medical content" called medical science?

I imagine that both clinicians and patients usually seek information from medical knowledge databases that are  scientifically-verifiable.

They are looking for therapy and management of conditions that is experienced, evidence-based, and built on hierarchies of excellence such as consensus views of specialists and practititioners.

>As a possible solution, I suggest a medical website registry which would
>collect, collate and publicize the ratings of various organizations
>along with open viewer feedback.

As a medical practitioner, who is attempting to meet the challenge of keeping up on diverse guidelines of professional
medicine, why would I go to a registry that combined sites and comments from faith healers and homeopaths?

If I was a homeopath why would I care about randomized clinical trials of pharmaceutical products?
 

>In order to be listed with the
>registry, medical internet resources would have to comply with a basic
>code of conduct (such as the HON code).
 

Unfortunately, the HON code does not acknowledge the hierarchy of knowledge in medical science. Perhaps medical practitioners can take hold of the evaluation of web content. I believe that they have an appropriate role to communicate their experience and study of what works. Clinicians seem to share a common view of effective medical therapy. I am not sure I see the difficulty in developing consensus quality rankings for health and medicine sites on the web with effectiveness, evidence, and excellence as measurement parameters.

I imagine that Universities, publishers, and societies will support search engines and sponsor an increasing amount of web content under well recognized quality "banners" thus making these issues less important.

Respectfully,

Gary Malet
Medical Matrix, www.medmatrix.org
 
 
 

Subject: Re: Quality of medical information on the Internet
Date: Wed, 26 Aug 1998 13:43:58 -0400
From: Michael Jacobson <mjmd@journalclub.org>
To: "AMIA Internet Working Group" <int-wg@mail.amia.org>
 

Gary,

Thanks, as usual, for your input.  I would like to make two general points before replying to your specific comments.

No matter how the issue of quality control is approached, I believe it is useful to consider the assessment of content separate from all other issues (which are primarily issues of accountability and "bookkeeping"). These latter issues are also important but are relatively well-circumscribed, easy to define and straightforward to apply. Who sponsors a site, who is responsible for its content, how those individuals can be reached, how well-linked the references are, how changes in content are documented, how medical advice is handled -- these are all vital matters.  They represent a floor,  a minimum, without which the assessment of content is fairly meaningless.

The assessment of content, although obviously important, is very different from the above issues. It is likely to generate heated controversy and will be more difficult to define. Not impossible perhaps, but clearly more difficult.  As such, content assessment may well benefit from a different approach than the other issues defined above. This is the reason that I believe these issues should at least conceptually be kept distinct.  And this is why I consider the Health On the Net code of conduct a very valuable concept. Not that it is all we need -- of course we need to address the issue of content as well. But it does address the whole realm of non-content issues, and does it quite nicely.

As for how we should actually address the quality of medical content on the Internet, I would suggest that we make use of one of the most valuable properties the Internet has to offer: interactivity.  The ability to gather and disseminate different viewpoints is something the online medium is uniquely suited to, and it is something we have not yet begun to exploit adequately. As Steven McGeady from Intel said, at the Harvard conference on the Internet and Society, "browsing the Web is a bit like walking through a shopping mall that's been neutron-bombed: lots of merchandise, but there's nobody there".  Let's put some people and points of view there.
 

To respond to your points individually:
 
 

>I am surprised that you believe that it is difficult to impartially
>judge medical content. Isn't the "impartial evaluation of medical
>content" called medical science.
>
>I imagine that both clinicians and patients usually seek information
>from medical knowledge databases that are scientifically-verifiable.
>
>They are looking for therapy and management of conditions that is
>experienced, evidence-based, and built on hierarchies of excellence
>such as consensus views of specialists and practititioners.
 
 

In the October 30, 1997 issue of the New England Journal of Medicine are two articles, under the heading "Clinical Debate. Should thrombolytic therapy be the first-line treatment for acute ischemic stroke?". The first article is entitled "Thrombolysis -- not a panacea for ischemic stroke", the second is "t-PA -- the best current option for most patients".  Even at the top of the evidentiary hierarchy, truth remains elusive.  How are we to rate the content of a medical website that cites exclusively evidence-based articles, but does so in a biased fashion?

Bear in mind that what I'm talking about is rating the quality of websites that present medical information primarily to consumers.  Can we apply the same criteria that we use when evaluating clinical trials reported in peer-reviewed journals?  How would you "rate" the content of an industry sponsored website that cites multiple journal articles and states that (example made up) "calcium channel blockers have been conclusively shown to be safe for the treatment of hypertension and are as beneficial as other agents"? What about if the wording is softened a bit (drop the 'conclusively'; change 'beneficial' to 'effective')?  It's not impossible to come up with criteria, but it certainly represents a quantum leap in difficulty compared with determining whether or not a website's sponsorship is clearly indicated.
 
 

>As a medical practitioner, who is attempting to meet the
>challenge of keeping up on diverse guidelines of professional
>medicine, why would I go to a registry that combined sites and
>comments from faith healers and homeopaths.
>
>If I was a homeopath why would I care about randomized clinical
>trials of pharmaceutical products.
 

As a patient with an open mind, if I had a problem that "conventional" medicine wasn't treating to my satisfaction, I would certainly want to know what options were available from sites that scored highly on, say, acupuncture and homeopathy ratings. At the same time, I would want to be sure that those same sites weren't accused of dangerous practices by respected conventional medical authorities.  As a physician, if I were caring for a patient with a malignant but localized colon polyp who insisted on taking coffee enemas, it certainly would be nice to be able to show that a website advocating such practices was given the thumbs down by some alternative medicine authorities.
 
 
 

>Unfortunately, the HON code does not acknowledge the hierarchy of
>knowledge in medical science. Perhaps medical practitioners can take
>hold of the evaluation of web content. I believe that they have an
>appropriate role to communicate their experience and study of what
>works. Clinicians seem to share a common view of effective medical
>therapy. I am not sure I see the difficulty in developing consensus
>quality rankings for health and medicine sites on the web with
>effectiveness, evidence, and excellence as measurement
>parameters.
>
>I imagine that Universities, publishers, and societies will support
>search engines and sponsor an increasing amount of web content under
>well recognized quality "banners" thus making these issues less
>important.
>

See my comments above on the HON approach, and on the difficulty of judging the content of medical websites directed at consumers.
 

Bottom line -- I think that a non-content related code of conduct is crucial and is enforceable. I agree with you that the rating of content is also possible, but I think it is much trickier, especially when you're talking about information directed at consumers. Also, it might benefit from a multi-individual and multi-institutional, open and interactive approach.

Please excuse the length of this post.

Michael
 


Subject: content vs. non-content evaluation
Date: Mon, 07 Sep 1998
From: "Brian S. Alper M.D." <alper@earthlink.net>
 

Ultimately, what consumers are looking for on the Web site is the content.

An evaluation system that addresses links, currency, sponsorship, etc. is useful in that it can be used as a general screening to avoid Web sites that are too cumbersome to use or are more likely to have poor content.

However, the crafty huckster can be sure to meet these criteria and get the "best" rating, regardless of the quality of content.

Evaluation of content will need to be in place to make using the "evaluated website database" useful in meeting your ultimate goal of finding content.
 

Brian S. Alper M.D.
Medical Director, DynaMed
http://www.DynamicMedical.com


Subject: proposal for website registry
Date: Tue, 08 Sep 1998
From: EDDIE VOS <vos@virtuel.qc.ca>
 

A registry using as criterion a HON type code of conduct might be useful especially in relation to the declaration of financial interests and of financial sponsoring. The database of medical websites would also be useful.

However, I am worried about the proposed "content EVALUATION" by, among others, "medical specialty societies and evidence-based medicine groups" as they may hamper dissemination of some content. Content CLASSIFICATION might be more useful because at all cost evaluation bias must be avoided if the Web is to remain useful.

I wonder how well my own webpage about cardiovascular diseases and their link (i.e. cause and prevention) to foods, supplements and vitamins would do when "content evaluated" by "medical specialty societies and
evidence-based medicine groups"  ( http://www.health-heart.org/ ).

These are the same groups that authored the excellent August 27 1998 Heart Failure Supplement of The Lancet (Vol. 352s1) and concerning which the Editor's discussion  admits: "... that despite the advances in the
management of coronary heart disease there is no clear evidence that therapeutic advances have made any impact on the overall burden of disease. ([link no longer available]).

Evidence based medicine was an unsuccessful judge as to the effects of treatment advances.  Why would such groups be any better at evaluating websites relating to "unconventional" techniques.

A few years ago ulcers were not thought to be caused by a bacterium (H pylori) while the importance of Lp(a) and vitamin E in vascular diseases and of homocysteine (i.e. B vitamins and betaine) in NTD's and in cardiovascular diseases is just now becoming generally accepted. Such and other ideas need and deserve prominent outlets.

Eddie Vos, M. Eng.
vos@virtuel.qc.ca
http://www.health-heart.org/
 

Actually, one of the potential advantages of a website registry with multi-participant content evaluation (including, but not limited to, medical specialty societies and ebm groups) is the increased likelihood that non-mainstream medical sites get consideration and exposure.  -- mj


 

Return to Journal Club home page
 
 

Site Meter