Recommendations

The CAAC recommends that US EPA pursue the development and implementation of a high quality, long-term measurement framework that will provide CA practitioners nationwide with tools and resources for collecting, aggregating, and presenting CA outcomes. This is a long-term goal that the Agency should aggressively pursue. To achieve this goal, the CAAC offers these recommendations which complement and reinforce the recommendations made by the previous CAAC and outline a number of interim steps that are necessary to achieve this goal. In developing and implementing such a system, the CAAC feels it is critical that:

§ The system emphasizes clear, simple measures, such as whether entities are in compliance with environmental requirements;

§ The measures should be used as a complement to counting the number of non-CA related enforcement actions or the amount of penalties collected; thus enabling the evaluation of environmental progress to focus on compliance success and more accurately target areas of failure for which CA could be an effective tool;

§ EPA must commit the resources necessary to produce effective CA measurement if CA is to be used as a long-term strategy for developing continuous environmental improvement;

§ EPA must aggregate information provided by local, state, tribal, and regional CA providers, as well as EPA to produce effective tools and analysis of CA activities.

The CAAC recommends the following specific actions on the part of EPA to pursue the objectives described above:

Keep Measurement Simple & Universal

· The Agency needs to take the necessary steps to ensure that all organizations within EPA that are involved in providing CA services institute adequate and appropriate CA performance measures. While EPA has identified state and tribal partnerships among the cross-goal strategies in its Strategic Plan, the Agency needs to identify cross-program, sector-based performance goals and focus on implementing the necessary partnerships with identified sectors to ensure that all programs offices work together to achieve those goals.

· The Agency should focus on establishing baseline data and evaluating compliance trends and using compliance changes and improvements as key measures of CA success.

· EPA should work with state, tribal, and local CA providers to develop a set of simple measures that focus on the types of activities and programs that they undertake. The Agency should focus resources toward research that evaluates the effectiveness of compliance assistance methodologies, rather than asking each program to independently and continuously measure the effectiveness of each approach. This research should underpin recommended approaches to delivering CA at the state and local level.

· EPA should examine the wealth of experience and information that is available from other agencies that have not had the coexistence of an enforcement approach, e.g., energy efficiency, tobacco consumption, nutrition, food safety, AIDS prevention, auto safety, consumer awareness, and others. The USDA cooperative extension program is grounded in methodology to diffuse new technologies. EPA should review and identify the best management practices in public education at other federal agencies and in some successful private sector efforts to learn from and adopt those approaches that others have tried and found effective.

· EPA needs to rigorously examine the measurement data that is currently available and analyzed and clearly communicate the limits of the information that is collected. If CA measurements are to be credible, they should not be done piecemeal but should be made as a joint priority for EPA, and other scientific and economic entities as are appropriate. There is an important unrealized opportunity for EPA, in association with other federal agencies, to research, test, and evaluate the effectiveness of various CA techniques/ practices so that this information could be shared across all CA programs to assist them in more efficiently delivering their services. Some programs may tend to broadly claim affects beyond the actual measurable results of an activity, e.g., tons of non-emissions due to the delivery of a fact sheet. Too often, project managers are seeking environmental result measures from activities that have no direct causal relationship with those impacts. This does not mean that such activities do not contribute to environmental improvements, but the influence of these activities cannot be easily measured.

· EPA should adopt and promote the “logic model” (Appendix B) as the basis for developing outcome measures, and educate its staff on its use. EPA should require a completed logic model for all substantially large grants, contracts, and external program funding in support of CA.

· The Agency needs to start by investing in the implementation of a set of relatively straightforward ways of assessing whether regulated entities are generally in compliance with environmental requirements and the prevalence of various categories of violations and sectors where chronic non-compliance tends to be high, particularly associated with the potential for environmental harm. Furthermore, EPA should use its Agency-wide measurement capacity to identify areas of greatest need for protection of human health and the environment, and CA programs should prioritize efforts to focus on those top needs.

· EPA should develop a tiered approach to reporting/measurement for CA projects dependent on the level of funding received from the Agency. As discussed by the previous CAAC, we support a monetary threshold for mandatory measurement activities to reduce the overburdening of CA programs with operating budgets below the threshold.

· As recommended by the previous CAAC, a request for follow-up information from the regulated entity receiving assistance should be made from the compliance assistance provider regarding resulting impacts on environmental performance (e.g., waste/emissions/discharge reductions); however, we feel the requests for information should be voluntary and correlated to the level of assistance provided. Environmental outcome measures may be appropriate for in depth consultations and hands on assistance; however, the majority of compliance assistance activities being provided would not create the data this recommendation is seeking. Again, improvement in compliance and improvement to the environment are not generally synonymous. Mandatory requests for information could have a negative impact on the number of companies seeking assistance if it is seen as burdensome or intrusive.

Focus on Compliance Behaviors & Trends in the Regulated Community

· EPA should use a consistent and systematic approach to measurement to help identify the mix of CA activities that is most effective in promoting and producing increased compliance; measures that can differentiate preferred approaches that deliver prioritized outcomes would serve to improve the effectiveness of service delivery both within and external to the Agency.

· CA providers need improved ways to track and measure compliance improvement as a result of CA activities; examples of possible measures are provided in Appendix C.

· While CA efforts do not always result in measurable environmental impacts, efforts should be made to measure and document improvements when they occur.

· Recognition should also be given to effectiveness of maintaining compliance with regulations as a measure of CA.

· EPA should research and study which activities induce educational and behavioral changes that have causal impacts on the environment; where the success of the activities has been demonstrated, continued demonstration should not be needed. For example, if, after a number of high quality studies of the results of compliance assistance activities (i.e., workshops, one-on-one assistance, or publications) targeted toward a particular sector or regulatory requirement have found that a certain percentage of the participants in that activity implemented compliance improvements, the Agency should allow CA programs to utilize that rate for extrapolating the results of other, similar activities with the sector or regulatory requirement. This would enable the programs to avoid having to extensively measure the results of similar activities each time they are conducted.

· EPA should develop and implement a single, consistent set of measures for assessing the outcomes and impacts of all four components of its integrated compliance assurance program (monitoring, CA, incentives and enforcement).

· EPA needs to develop improved baseline data on compliance rates for targeted sectors or regulations to help evaluate the effectiveness of CA (see Appendix D for one example of a state that has undertaken such as effort).

Provide the Necessary Support & Resources

· Quantifying the value of CA activity is resource intensive; EPA must ensure that adequate resources are allocated to CA measurement. EPA should budget measurement funding within programs commensurate with expectations of measurement objectives. Service delivery may be impacted as much more emphasis is placed on measuring outcomes and on measurement. This should be understood and communicated to stakeholders and collaborating CA programs. The infrastructure necessary to accomplish compliance assistance measurement must be in place at the initiation of CA activities so that the measures can be implemented throughout the CA project or program.

· EPA should use cost-effective, statistically valid measurement systems that demonstrate the outcomes and impacts of efforts, rather than attempting complete measurement of every aspect of every program. By using agency resources to test and recommend best practices instead of requiring each program to verify the results of each activity, resource utilization can be extended. In instances where the Agency is able to gather statistically valid performance data for CA activities, the Agency should allow individual CA programs to use this performance data as a surrogate for actual performance measurement.

Training CA Managers & Providers in Implementing Measures is Critical

· EPA needs to continue to develop and deliver training to CA program staff on basic measurement techniques and strategies, including the use of logic models described in Appendix B. EPA must take the necessary steps to provide measurement training for all staff involved in providing CA services. Such training should focus in particular on effective data collection, data management, and data reporting techniques and skills.

· EPA should develop a strategic planning and performance measurement training component for CA mangers with an orientation on the development and use of strategic planning and program performance measurement.

CA Measurement Clearinghouse

EPA should support and improve CA program measurement through its web-based Compliance Assistance Clearinghouse by identifying effective CA measurement techniques that can help CA programs determine:

· Which CA techniques are better suited for short-tem vs. long-term outcome shifts in a target audience.

· Which techniques are more suited for individual target audiences or individual environmental contexts, e.g., non-point source v. point source pollution sources.

· If it can devise a list of techniques or a matrix to help CA programs best develop an outcome-oriented CA plan.

· If there is a specific set of techniques that most effectively leads target audiences through the “Stages of Change” (Appendix B).

· If retail CA programs (those that deliver compliance assistance directly to end-clients) are accountable for environmental outcomes.

· If these programs set outcome priorities for which CA is a useful tool.

· What data is it logical to expect from retail CA programs.

· Where a CA provider seeks only to bring the target audience into regulatory compliance, if there is a logical and/or causal linkage to environmental performance.

· If there is a link between regulatory compliance and environmental impact.

· The resources CA programs need to achieve their goals and focus their efforts on the top needs.

There are many examples of useful and interesting CA measurement projects that have been undertaken by regional and state CA programs around the country. Examples of case studies of effective performance measures that were described at the Compliance Assistance Providers Forum in 2003 are the following:

· Hazardous waste - tracking pounds reduced through a voluntary reporting program.

· Small business - track how much assistance people are getting, how many took advantage of visits, and the results of follow-up visits.

· Texas has developed a performance measurement system that evaluates the impact of EMS and Regulatory Flexibility programs - based on self-reported information.

· Clean Texas 2000—voluntary program involving 200 industries. Tracked regulatory measurements under TRI and hazardous waste generation reports (e.g., tons of emissions reduced, off- site disposal).

· Auto dealers - track money saved in reduced disposal costs and reduced product costs from product substitutions. Track benefits to the bottom line. Relate money to reduced emissions through disposal costs.

· Used the Web as a tool for gathering measurement information, but there were challenges, including: tracking who uses your site, surveying users, and how to measure sector performance from various sources.

· Environmental outcomes from CA and other activities including the EPA Region 1-NE Charles River initiative that focused on tracking improvements in water quality, a Minnesota project that tracked changes in VOC emissions from waste a MA DEP Environmental Results Projects project that focused on a set of specific environmental outcome results and changes in compliance rates, and an initiative in Maryland that targeted marinas and tracked the changes in behavior associated with compliance.

EPA should examine the measures used in these and many other examples and promote their greater utilization and catalog them in the online national compliance assistance measurement clearinghouse.