Promoting Evidence-Based Decision Making in a Local Health Department, Pueblo City–County, Colorado

Anna K. Hardy, RN, MPH; Christine Nevin-Woods, DO, MPH; Sylvia Proud, MS; Ross C. Brownson, PhD

Suggested citation for this article: Hardy AK, Nevin-Woods C, Proud S, Brownson RC. Promoting Evidence-Based Decision Making in a Local Health Department, Pueblo City–County, Colorado. Prev Chronic Dis 2015;12:140507. DOI: http://dx.doi.org/10.5888/pcd12.140507.

PEER REVIEWED

PEER REVIEWED

Abstract

Background

Evidence-based decision making (EBDM) is an effective strategy for addressing population health needs. Assessing and reducing barriers to using EBDM in local health departments may improve practice and provide insight into disseminating EBDM principles among public health practitioners.

Community Context

Administrative leaders at the Pueblo City–County Health Department, Pueblo, Colorado, used a systematic approach for implementing EBDM. Research partners engaged staff to understand factors that increase or deter its use.

Methods

A survey was distributed to staff members at baseline to identify gaps in administrative and individual practice of EBDM. In-depth interviews were also conducted with 11 randomly selected staff members. Results were shared with staff and administration, after which activities were implemented to improve application of EBDM. A follow up survey was administered 1 year after the initial assessment.

Outcome

Survey data showed evidence of progress in engaging and educating staff members, and data showed improved attitudes toward EBDM (ie, several items showed significant improvement from baseline to follow-up). For example, staff members reported having the necessary skills to develop evidence-based interventions (73.9%), the ability to effectively communicate information on evidence-based strategies to policy makers (63.0%), access to current information on improving EBDM processes (65.2%), and a belief that evidence-based interventions are designed to be self-sustaining (43.5%).

Interpretation

Within a local health department in which leaders have made EBDM a priority, addressing the culture and climate of the department may build EBDM. Future research may provide insight into tailoring EBDM within and across local health departments.

Top

Background

Evidence-based decision making (EBDM) provides a framework to address many critical challenges (eg, setting priorities, making efficient use of resources) facing the public health system (1). EBDM is a process used to determine the best intervention for a population; it is rooted in community needs, practitioner experience, and existing evidence (2). EBDM produces high-quality information on what works in populations, resulting in implementation of successful programs and policies, greater workforce productivity, and more efficient use of funding (1). Public Health Accreditation Board Standard 10 requires the use and dissemination of evidence (3), which contributes to the momentum for EBDM.

Although the key tenets of EBDM are now well established (1,4,5), information is sparse on how to implement EBDM concepts in day-to-day public health practice. Research shows that EBDM practice does not happen organically (5); there are common barriers to engaging local health departments in EBDM. These include philosophical differences between practitioners and researchers; lack of time, money, or incentives; and insufficient organizational support (5). One survey of 447 state and territorial health care practitioners treating chronic disease showed that the strongest barriers to using EBDM were organizational factors (6). Other national data show the importance of agency size and make-up (ie, types of staff members and their qualifications) in EBDM (7,8).

Organizational structures and activities associated with EBDM performance, called administrative evidence-based practices (AEBPs) (5), fall into 5 domains: workforce development, leadership, organizational culture and climate, relationships and partners, and financial characteristics of the agency (7). This article describes an assessment of AEBPs and individual staff-member factors in a local health department involved in adopting EBDM, the activities related to promoting EBDM that took place over 1 year, and the impact of those activities on staff attitudes and perceptions toward EBDM.

Top

Community Context

The Pueblo City–County Health Department (PCCHD), located in southeastern Colorado, was selected for this case study because of the public health director’s interest in establishing EBDM as standard practice throughout the department. Initial contact was made between the director and the research team through an EBDM training. PCCHD is a mid-sized health department governed by a local board of health. It serves a population of 159,000 that is 54% non-Hispanic white and 42% Hispanic. Approximately 18% of residents in PCCHD’s jurisdiction have incomes below the federal poverty limit, and 72% have completed high school (9). PCCHD has approximately 90 employees in 8 departments: Women, Infants, and Children program; budget and finance; vital statistics; laboratory; environmental health; community health services; administration and disease prevention; and emergency preparedness. The department’s operating budget is $6.7 million, with funding from local, state, and federal government and from private foundations. At the start of this study, the director was a physician with a master’s degree in public health who had been at PCCHD for 22 years. The director had invested in building the EBDM capacity of her staff and wished to create the infrastructure and organizational culture that would make EBDM the norm in the department. The objective of this study was to improve EBDM among PCCHD staff members, which would be evidenced by changes in staff members’ perceptions and knowledge of EBDM from baseline to follow-up 1 year after community engagement began.

Top

Methods

This study used a mixed-methods approach consisting of a quantitative survey, individual staff member interviews, and a focus group. The survey was based in part on the project team’s prior research (5,10). We measured a set of AEBPs and EBDM skills by asking respondents to rank them according to importance and availability.

On the basis of results of the initial quantitative survey, gaps between scores for importance and availability and low-ranking AEBPs were identified, and qualitative interview questions were developed to explore those gaps. The individual interviews consisted of 24 questions, which were reviewed for comprehension with 1 staff member and 1 public health professional from a different local health department.

In addition to the individual interviews, 1 focus group was conducted and opened to all staff members in order to allow those who were not interviewed a chance to express their views. A shortened 12-question version of the individual interview was given to focus group participants. The institutional review board of Washington University in St. Louis approved this project.

Data collection

In February 2013 a baseline survey was sent to 74 PCCHD staff members chosen by administrators on the basis of how pertinent EBDM was to the type of work the staff members performed. Email addresses were acquired from PCCHD, and surveys were distributed via individualized links generated using the Qualtrics online research suite (www.qualtrics.com). PCCHD administrators encouraged the staff to complete the surveys through face-to-face interaction or email. No incentives were given. The follow-up survey contained the same questions as the initial one with an additional 4 questions to assess the impact of activities that occurred during the time between surveys. The follow-up survey was administered to all staff members who received the initial survey and were still employed by PCCHD in April 2014 (n = 46).

Drawing from respondents to the quantitative survey, a random sample of 11 staff members was selected to participate in a semistructured face-to-face interview.

Data analysis

We calculated frequencies and proportions for both participant characteristics and for the various survey items. Several categorical items were recoded as binary (1 = yes, 0 = no or do not know), and χ² tests were used to assess differences in proportions between the 2 surveys. To examine mean differences, we conducted paired sample t tests (repeated measure) for items related to organizational and individual skills. To describe the magnitude and direction of any significant differences, we calculated effect sizes. Cohen’s d values were calculated for mean differences observed in t tests. Cohen suggests the following effect ranges for d: 0.2, small; 0.5, medium; and 0.8, large (11). In addition, we calculated Cramér’s V values for effect sizes for differences in proportions. Cramér’s V may be interpreted as 0.1, small effect size; 0.3, medium effect size; and 0.5, large effect size (12). All analyses were conducted in SPSS version 20 (IBM Corp) or Microsoft Excel version 2010 (Microsoft Corp).

All responses were recorded and were confidential and anonymous. The interviewer transcribed the qualitative interviews. Open coding was completed for each transcript to ascertain general themes. The interviewer next used focused charting to develop a codebook that was distributed to a team of researchers. The team made changes to more adequately capture codes and subcodes. The final codebook was then used by the interviewer to analyze each script. In addition, 1 research assistant analyzed 3 scripts for interrater reliability. Discrepancies between reviewers were minimal. A nonparticipant transcribed the content of the focus group, which was then analyzed by verbally verifying content for accuracy among the group, and then capturing major themes and concluding remarks by using open coding.

Top

Outcome

Initial survey, staff interviews, focus group

Of the 74 staff members who received the baseline survey, 59 completed it, an 80% response rate. The majority of respondents were technical experts (ie, planner, grant writer, epidemiologist, health educator, environmental health specialist, emergency preparedness educator [49.2%]) or other (23.7%), which included nurses, a registered dietician, administrative assistants, accountants, and the vital statistics staff. On average, participants had spent 5.7 years in their current position (standard deviation [SD], 6.9), and 9.5 years in public health (SD, 8.5). Two AEBP domains ranked lowest in the surveys, leadership and organizational climate and culture ( Table 1). The personal skill for which respondents reported the highest proficiency was the ability to develop evidence-based interventions. Staff members reported that training in EBDM would most encourage them to use EBDM ( Table 2).

Among 11 staff members interviewed, the length of employment at PCCHD ranged from 1.5 to 21 years. Nine interviewees were line staff, and 2 held manager or supervisory roles. Five of 8 departments were represented.

Qualitative findings were grouped into benefits and barriers to EBDM use and action steps to promote EBDM at PCCHD ( Table 3). The benefits to using EBDM cited most often were that it fosters a targeted use of limited resources and that it improves service to the community. Barriers to EBDM use at PCCHD were a lack of capacity and internal and external inflexibility (ie, resistance to change).

Staff recommended several action steps for improving the use of EBDM at PCCHD. First, health department leaders and managers should clearly communicate EBDM messages to various departments and model EBDM as a process, in a manner that crosses categorical programs. A second suggestion was to listen to and follow up on staff suggestions. Third, staff members requested positive feedback from leadership to encourage use of EBDM. Finally, staff members wanted to understand the relevance of EBDM to their work. As for training in EBDM, and consistent with adult education theories (13), staff respondents indicated that the most useful training experiences are those that are hands-on and have practical application to their professional duties.

Focus group findings were also clustered into 3 general topics related to EBDM: benefits, barriers, and action steps. Respondents said that benefits to using EBDM in a local health department were “smart business”: EBDM can improve programs, help government agencies stay accountable, determine program effectiveness, and improve efficiency. The barriers were internal and external politics, funding and time limitations, personal agendas, differences in recognizing the value of EBDM across departments within local health departments, and limited knowledge and self-efficacy for using the process. When asked, “If you were the administrator and wanted to make EBDM a priority, what would you do to bridge the gap between benefits and barriers to EBDM?,” the overwhelming majority agreed that securing sustainable funding, rather than grant-focused funding, would be the most beneficial. Other responses included explaining how EBDM is pertinent to each department, demonstrating and explaining specific steps in EBDM, and improving the public’s understanding of the purpose of public health.

Activities between surveys

Baseline survey findings were presented to key administrators at PCCHD, the local board of health, and to all PCCHD staff in several meetings. Presentations were a collaborative effort between the research team and PCCHD administration and staff. The results were shared in order to engage the PCCHD staff in developing policies and activities aimed at reducing the barriers to EBDM.

The following is a summary of the recommendations given at the conclusion of the research assistant’s (A.H.’s) time with PCCHD:

• Clearly define and differentiate workforce improvement initiatives.

• Pay attention to EBDM messaging, modeling, and follow-up activities, because they are vital for the success of EBDM.

• Ensure every person and division understands the value of EBDM and how each staff member can use the process to improve service.

• Ensure that administration and management regularly discusses EBDM ideas and progress in meetings.

• Create an EBDM promotion team from among staff members trained in EBDM.

Two-thirds of the staff agreed or strongly agreed that these recommendations accurately captured staff behaviors and feelings about EBDM.

Following the assessment period, the research team maintained communication with the PCCHD community via conference calls and email to learn about, and track progress on, their plans for addressing improvement in the uptake of EBDM. Notably, many strategies were operationalized through collaboration between staff members and external partners. For example, PCCHD partnered with a state university to tailor a 3-day EBDM training for PCCHD staff that was based on an established training model (2). Agency leaders invited all staff members who held positions that allowed them to make evidence-based decisions. An EBDM team in PCCHD made up of staff members with EBDM training and experience was identified as a resource for staff members to contact for consultation as they learned the EBDM process. Administrators clarified messages about EBDM (ie, that it is a process, not a specific program) as the results of the survey were shared with staff members. To minimize confusion, messages and concepts related to EBDM, quality improvement, and the plan-do-check-act model (a management method used for continuous improvement) were merged into 1 decision flowchart and presented to the entire staff (Figure). EBDM was then incorporated into the 5-year strategic plan by the PCCHD strategic planning committee, where it now states that PCCHD “will implement a quality improvement plan using EBDM.” The local board of health was asked to promote the process by asking the PCCHD administration to formalize the use of EBDM in PCCHD as they made decisions on policies and programs. For example, EBDM language was incorporated into several PCCHD policies, namely the formal grant process, personnel policies, the strategic plan, and supervisor guidelines (in process).

Figure. Integration of evidence-based decision making, quality improvement, and plan-do-check-act at Pueblo City–County Health Department, Colorado. [A text description of this figure is also available.]

Follow-up survey

The follow-up survey was sent to the 46 staff members who were respondents to the initial survey and who were still employed at PCCHD; the survey had a 100% response rate. Like the initial survey, the majority of respondents were technical experts (50.0%) or other (23.9%). Participants had an average of 6.9 years in their current position (SD, 7.0), and 10.3 years in public health (SD, 8.5). Between surveys, 55.9% of the respondents participated in an all-staff meeting where information on EBDM was presented, and 52.5% attended the tailored 2-day training.

Significant changes were made in the AEBPs in the follow-up survey for access to training in EBDM and quality improvement processes, access to current information on EBDM, and knowing PCCHD allocated resources for quality improvement (Table 1). Significant improvement in individual skills assessed in the follow-up survey were having skills in developing evidence-based interventions, possessing communication skills, and acknowledging the self-sustaining nature of evidence-based interventions.

Changes between pre- and post-survey data on activities that would most encourage staff to use EBDM (Table 2) are noteworthy. For example, at baseline survey, training in EBDM was ranked as the activity most likely to motivate staff members to use EBDM. At the post-training survey, training in EBDM had shifted to third highest priority, and encouragement in use of EBDM was ranked highest. This change suggests that, after administrators provided training opportunities, the next phase in increasing use of EBDM was for administrators to provide positive feedback or encouragement in employees’ use of EBDM.

The success of PCCHD staff engagement was largely due to administrative support. Administrators were dedicated to delivering messages about EBDM in ways their staff members found helpful. They combined EBDM and quality improvement processes in order to make using both less confusing. The partnership between a state university and the PCCHD was crucial to delivering the 2-day EBDM training, which was tailored to address the needs identified by staff during the assessment phase.

Top

Interpretation

This case study provides evidence that a local health department can improve attitudes toward EBDM among its staff in a relatively short time. First, through qualitative and quantitative evaluation, we were able to understand the barriers a local health department faced in using EBDM. Second, we showed evidence of staff engagement in improving attitudes and perceptions of EBDM principles as shown by two-thirds of the staff reporting that the team accurately captured their attitudes toward EBDM and significant improvement in several quantitative outcomes.

This study had several limitations. The main limitation relates to community engagement in that not all staff members were included in the survey. Because the pool of respondents was chosen by PCCHD administrators on the basis of job descriptions, results may be skewed in favor of EBDM practice. Another limitation is that 17% (10/59) of the original sample was lost to follow-up because they were no longer employed at PCCHD. In addition, organizational activities occurred during the year from baseline to follow-up, and activities occurring close in time to follow-up data collection may have been recalled more clearly, allowing the potential for bias. Finally, the context-specific nature of a case study limits the generalizability of any findings to other local health departments.

The activities conducted by PCCHD show evidence of improvement in attitudes and perceptions toward EBDM over 1 year. Previous studies suggest that strong leadership is a key ingredient in creating a work climate conducive to EBDM in a local health department (14). At baseline, the PCCHD staff members ranked organizational culture and climate as one of the lowest AEBPs. However, in 1 year, through continuing community engagement, and with an administration committed to making EBDM standard practice, organizational culture and climate were no longer among the lowest ranked AEBPs. The workforce development efforts described here may have influenced this improvement. Enhancing the climate and culture of a local health department is likely a pivotal activity for enhancing the uptake of EBDM (6,7).

This case study of a local health department community builds on previous work (14,15) by providing insight into steps that may improve EBDM adoption within local health departments. Additional observational studies may examine the dissemination of EBDM among PCCHD’s network with other local health departments to assess dissemination and implementation within the public health system (16). EBDM training might focus on hands-on case studies and use of a range of potential resources (4) that pertain to various topics covered by different departments within local health departments and other community-level agencies (eg voluntary health organizations). In practice, leadership may seek to concentrate on fostering a culture and climate conducive to using EBDM by establishing its relevance to all staff members, reiterating that EBDM includes community and practitioner input, and including staff members in key decision making processes.

Top

Acknowledgments

This project was funded in part by contract no. U48/DP000060 (Prevention Research Centers Program) and the Evidence-Based Public Health training program supported in part by the National Association of Chronic Disease Directors, contract no. 482012. This article is a product of a Prevention Research Center and was also supported by cooperative agreement no. U48/DP001903 from the Centers for Disease Control and Prevention.

The authors and Kathleen Duggan conceived and designed this study. We thank Laurie Schneider, Janna West-Kowalski, the Colorado Department of Public Health and Environment, and the Colorado School of Public Health for consulting on and providing EBPH training to research staff and PCCHD staff; Rachel John at the Washington University Brown School Fieldworks Office; Mary Adams and Linda Dix at the Prevention Research Center in St. Louis for coordinating logistics of the fieldwork; Robert Fields, Carson Smith, and Rebekah Jacob for assistance with data analysis and Qualtrics; Beth Dodson for reviewing early manuscript drafts; Carol Brownson for reviewing the full manuscript; and the PCCHD staff and all local board of health members for their participation and support.

Top

Author Information

Corresponding Author: Anna K. Hardy, RN, MPH, Saint Louis University, 1312 Carr Lane Ave, Education Union, Rm 110, St Louis, MO 63014. Telephone: 314-915-4404. Email: hardyak@slu.edu.

Author Affiliations: Christine Nevin-Woods, Sylvia Proud, Pueblo City–County Health Department, Pueblo, Colorado; Ross C. Brownson, Prevention Research Center in St. Louis, Brown School, Washington University in St. Louis, Division of Public Health Sciences and Alvin J. Siteman Cancer Center, Washington University in St. Louis School of Medicine, St. Louis, Missouri.

Top

References

1. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health 2009;30(1):175–201. CrossRef PubMed
2. Baker EA, Brownson RC, Dreisinger M, McIntosh LD, Karamehic-Muratovic A. Examining the role of training in evidence-based public health: a qualitative study. Health Promot Pract 2009;10(3):342–8. CrossRef PubMed
3. Public Health Accreditation Board. Standards: an overview. Alexandria (VA): Public Health Accreditation Board; 2012.
4. Jacobs JA, Jones E, Gabella BA, Spring B, Brownson RC. Tools for implementing an evidence-based approach in public health practice. Prev Chronic Dis 2012;9(1):E116. CrossRef PubMed
5. Brownson RC, Allen P, Duggan K, Stamatakis KA, Erwin PC. Fostering more effective public health by identifying administrative evidence-based practices: a review of the literature. Am J Prev Med 2012;43(3):309–19. CrossRef PubMed
6. Jacobs JA, Dodson EA, Baker EA, Deshpande AD, Brownson RC. Barriers to evidence-based decision making in public health: a national survey of chronic disease practitioners. Public Health Rep 2010;125(5):736–42. PubMed
7. Brownson RC, Reis RS, Allen P, Duggan K, Fields R, Stamatakis KA, et al. Understanding administrative evidence-based practices: findings from a survey of local health department leaders. Am J Prev Med 2014;46(1):49–57. CrossRef PubMed
8. Lovelace KA, Aronson RE, Rulison KL, Labban JD, Shah GH, Smith M. Laying the groundwork for evidence-based public health: why some local health departments use more evidence-based decision-making practices than others. Am J Public Health 2015;105(Suppl 2):S189–97. CrossRef PubMed
9. State and county quick facts: Colorado. US Census Bureau; 2013. http://quickfacts.census.gov/qfd/states/08/08101.html. Accessed August 8, 2014.
10. Stamatakis KA, McQueen A, Filler C, Boland E, Dreisinger M, Brownson RC, et al. Measurement properties of a novel survey to assess stages of organizational readiness for evidence-based interventions in community chronic disease prevention settings. Implement Sci 2012;7(1):65. CrossRef PubMed
11. Cohen J. A power primer. Psychol Bull 1992;112(1):155–9. CrossRef PubMed
12. Cramér H. Mathematical methods of statistics. Princeton (NJ): Princeton University Press; 1946.
13. Fenwick TJ. Expanding conceptions of experiential learning: a review of the five contemporary perspectives on cognition. Adult Educ Q 2000;50(4):243–72. CrossRef
14. Peirson L, Ciliska D, Dobbins M, Mowat D. Building capacity for evidence informed decision making in public health: a case study of organizational change. BMC Public Health 2012;12(1):137. CrossRef PubMed
15. Aronson RE, Lovelace K, Smith M, Shah GH. Differences in definitions of EBPH and evidence: implications for communication with practitioners. Am J Public Health 2014;104(12):e40. CrossRef PubMed
16. Harris JK. Communication ties across the national network of local health departments. Am J Prev Med 2013;44(3):247–53. CrossRef PubMed

Top

Tables

Table 1. Responses of Administrative and Individual Staff Members to Baseline (N = 74) and Follow-Up (N = 46) Surveys, Evidence-Based Decision Making Practices (EBDM), Pueblo City–County, Colorado, Health Department, February 2013 and April 2014

Abbreviation: SD, standard deviation; —, not applicable.
^a P values are reported for χ² test, paired-sample t test, or independent t tests.
^b Seven-point Likert scale response option (7 = strongly agree; 1 = strongly disagree).
^c Expressed as a percentage of yes for yes/no/don’t know and “strongly agree” or “agree” for Likert response options within each domain.
^d Yes/no/don’t know response option.

Table 2. Rankings of Activities That Would Most Encourage Staff to Use Evidence-Based Decision Making Practices (EBDM),^a Baseline and Follow-Up Surveys of Administrative and Individual Staff Members (N = 46), Pueblo City–County, Colorado, Health Department, February 2013 and April 2014

^a The 5 items were ranked on a scale of 1 to 5 (1 = highest ranking; 5 = lowest ranking).

Table 3. Responses of Administrative and Individual Staff Members (N = 46) to Open-Ended Interviews, Evidence-Based Decision (EBDM) Making Practices, Pueblo City–County, Colorado, Health Department, June–July 2013

Top