Evaluation of Joint Services Installation Pilot Project and BioNet Syndromic Surveillance Systems --- United States, 2004

Nicola Marsden-Haug, V. Foster, S. Hakre, A. Anderson, J. Pavlin
Walter Reed Army Institute of Research, Silver Spring, Maryland

Corresponding author: Nicola Marsden-Haug, Walter Reed Army Institute of Research, Division of Preventive Medicine, 503 Robert Grant Avenue, Silver Spring, MD 20910. Telephone: 301-319-7355; Fax: 301-319-9104; E-mail: nicola.marsdenhaug@na.amedd.army.mil.

Disclosure of relationship: The contributors of this report have disclosed that they have no financial interest, relationship, affiliation, or other association with any organization that might represent a conflict of interest. In addition, this report does not contain any discussion of unlabeled use of commercial products or products for investigational use.

Abstract

Introduction: Key elements that can be measured objectively to evaluate the effectiveness of a surveillance system include sensitivity, specificity, and timeliness. Statistical algorithms and data sources have been evaluated frequently; however, factors that affect whether public health professionals will use a system (e.g., acceptability, generalizability, flexibility, representativeness, and reliability) are more difficult to assess and are reported less frequently. Through the Joint Services Installation Pilot Project (JSIPP), the Department of Defense provided nine military installations with enhanced capabilities to detect and respond to weapons of mass destruction. The Department of Homeland Security is funding BioNet to improve outbreak management in San Diego, California, by integrating military and civilian information. Both programs implemented versions of the Electronic Surveillance System for the Early Notification of Community-based Epidemics (ESSENCE) IV for medical surveillance and required evaluation within these environments.

Objectives: This study measured JSIPP and BioNet users' perspectives of syndromic surveillance and the use of ESSENCE IV.

Methods: For JSIPP, registered ESSENCE users were surveyed regarding system usage, utility of features, user friendliness, and suggested improvements. For BioNet, potential ESSENCE users were identified by program administrators and surveyed before implementation to identify qualities and features they viewed as important to syndromic surveillance.

Results: The JSIPP survey response rate was only 34% (17 of 50), probably because registered but infrequent users were included in the survey population. The majority of respondents found the system easy to use and valued having access graphics and summary statistics of disease trends. Seven (41%) of 17 reported concern about the inability to obtain patient identifiers in a timely manner, which diminished their ability to investigate suspicious alerts. Barriers to system use included dislike of the layout and difficulties in interpreting nonclinical data sources. The BioNet survey response rate was 59% (13 of 22). Respondents ranked the usefulness of four elements of syndromic surveillance: 1) usefulness during an outbreak investigation; 2) ability to detect outbreaks rapidly; 3) reassurance that no ongoing outbreaks are occurring; and 4) capability to generate summary reports. Major weaknesses included 1) difficulty in interpreting and responding to alerts; 2) uncertainty in outbreak detection capability; 3) use of abstract data sources; and 4) difficulty in sustaining a system for long-term use. Additional concerns included high false-positive rates, timeliness, reliability, and cost.

Conclusion: Users often continue to have reservations about the utility of these systems. Feedback on which data sources and system features users value can help system developers direct resources for development.

Use of trade names and commercial sources is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services. References to non-CDC sites on the Internet are provided as a service to MMWR readers and do not constitute or imply endorsement of these organizations or their programs by CDC or the U.S. Department of Health and Human Services. CDC is not responsible for the content of pages found at these sites. URL addresses listed in MMWR were current as of the date of publication.

Disclaimer   All MMWR HTML versions of articles are electronic conversions from ASCII text into HTML. This conversion may have resulted in character translation or format errors in the HTML version. Users should not rely on this HTML document, but are referred to the electronic PDF version and/or the original MMWR paper copy for the official text, figures, and tables. An original paper copy of this issue can be obtained from the Superintendent of Documents, U.S. Government Printing Office (GPO), Washington, DC 20402-9371; telephone: (202) 512-1800. Contact GPO for current prices.

Date last reviewed: 8/5/2005