POP Center Tools Assessing Responses to Problem, 2nd Ed Appendix C
Appendix C: Problem-Solving Assessment Checklist
This checklist provides a summary of the issues that need to be considered when you evaluate a problem-solving effort. It is general guidance, not a set of rigid rules. It will be most useful if you use it throughout the problem-solving process, beginning in the scanning stage.
I. Pre-Evaluation Considerations
The items here should be considered during the scanning, analysis, and response stages.
A. What Decision Will the Evaluation Help You Make?
1. Should this problem-solving effort be continued? If this is the only question the evaluation will help
decide, a simple evaluation design will be sufficient (see Question III.A).
2. Should this response be used for similar problems (either by your agency or by other agencies)? If
checked, you should consider using a control group in the impact evaluation design (see Question III.A).
3. No decision. If checked, an evaluation will not be helpful. Stop here.
B. Do You Know the Problem?
These questions help develop a cost-effective response and evaluate the response. If you cannot answer these questions with some precision, you need to do more to analyze the problem.
Who is harmed by the problem? Who is not harmed?
How can the harm be measured?
Where does the problem occur? Where is the problem absent?
When does the problem occur? When doesn’t it occur?
What causes the problem? What prevents or suppresses it?
C. Do You Know How the Response Works?
These questions need to be answered in order to determine whether the response is likely to be effective, and to assure accountability during implementation. If you cannot answer them, plans for the response are inadequate and more attention needs to be put into the response stage.
How does the response influence the causes of the problem?
Who is responsible for carrying out the response?
When is the response supposed to be implemented?
Where is the response supposed to be implemented?
How long will it be before the response is likely to have a noticeable impact on the problem?
Who has the legal authority to carry out the response?
What are the likely barriers to implementing the response?
II. Process Evaluation
Process evaluations begin toward the end of the response stage and continue well into the assessment stage.
A. Was the Response Implemented?
The closer the actual implementation is to the planned response, the greater confidence you have that the response was the cause of the changes in the problem detected by the impact evaluation (section III, below). The more variation between what was intended and what occurred, the greater the likelihood that changes in the problem are due to things other than the response.
Was it implemented when it was supposed to be implemented?
Was the response implemented where it was supposed to be implemented?
Was the response implemented for the appropriate people?
Was the implementation carried out as planned?
B. Was Enough of the Response Implemented?
The response might be implemented as planned, but lacked the resources, duration, or intensity needed to make it effective.
Were there sufficient resources available to fully implement the response?
Was the response carried out for a sufficient duration to have an impact?
Was the response carried out with sufficient intensity?
III. Impact Evaluation
Many of the decisions needed to conduct an impact evaluation must be considered in the analysis and response stages. This is particularly the case with measurement decisions.
A. Do You Need a Control Group?
Answering these questions helps decide on the complexity of the evaluation design.
1. Did you check Question I.A.1? If YES, then you do not need a control group.
2. Did you check Question I.A.2? If YES, then you should use a control group.
B. How Often Can You Measure the Problem?
Answering these questions helps you decide whether a time series design is possible.
1. Can you measure the problem consistently for many time periods before (at least 10 periods, though at least 30 is better) and after (at least 10, but 30 is preferred) the response was implemented? If checked, a time series design is feasible.
2. Can you only measure the problem a few times before the response and a few times after the response? If checked, a time series design is not feasible and some form of a pre-post design will be needed.
3. Do you have some measures of the problem that can be examined for many time periods before and after the response, and other measures that can only be examined for a few time periods before and after the response? If checked, then you can use a time series design and a form of a pre-post design.
C. What Type of Evaluation Design Should I Pick?
Your answers to questions in A and B, immediately above, provide some basic guidance answering this question. This is shown in the following table. Obviously, precise answers depend on the particular circumstances of each problem-solving effort.
D. What Type of Contro Group Do I Need?
(only applies if one of the designs from column 2 in Table D1 was selected. If the design comes from column 1, skip this section and go directly to part IV.)
1. Will the response be applied in an identifiable geographic area (place, neighborhood, etc.)? If YES,
the control group should be a very similar geographic area that has similar problems that will not get the response, and preferably is located at some distance from the problem area (to prevent the response from leaking into the control area and contaminating it).
2. Will the response be applied to a set of identifiable potential victims (young males, elderly women,
commuters, etc.)? If YES, the control group should be a very similar group of potential victims that will not be given the response.
3. Will the response be applied to a set of identifiable potential offenders? If YES, the control group should be a very similar group of potential offenders that will not be given the response.
4. Will the response be applied to some other identifiable set of people or things? If YES, the control group should be a very similar group of potential people or things that will not be given the response.
5. It is impossible to identify a control group for this evaluation. If this is checked, go back to Table D1
and pick the appropriate design from column 1. Then proceed to part IV.
If one of the questions 1 through 4 were answered YES, systematically compare the characteristics of the response group to the characteristics of the control group, and list the major differences. In the last part of this checklist (part V) you are asked to make a judgment regarding the possibility that other factors might have caused the change in the problem relative to the control. Your list of differences is a list of potential other factors that could account for the change in the problem.
Table C1: Which Evaluation Design Makes the Most Sense?
IV. Drawing Conclusions
These items fall within the assessment stage and are applicable once evaluation results have been documented. These questions are designed to help formulate conclusions that are consistent with the results from your process and impact evaluations and with your evaluation design. You will have to ask more questions than listed here to fully interpret your particular evaluation results.
A. What Are Your Findings From the Process Evaluation?
1. The response was not implemented.
2. The response was implemented in a radically different manner than was planned.
3. The response was implemented with inadequate resources, too limited duration, or without the intensity required.
4. The response was implemented nearly as planned and with adequate resources, for the necessary time, and with the required intensity.
B. What Are Your Findings From the Impact Evaluation?
(Select the design(s) you applied — pre-post, pre-post with control, time series, or multiple time series. If multiple designs were used, interpret Tables 2 and 3 for each design separately.)
Pre-post design (no control) — use Table 2 to interpret your evaluation.
1. Problem got worse after the response.
2. Problem unchanged after the response.
3. Problem declined after the response.
Pre-post design with control — Use Table 3 to interpret your evaluation.
1. Problem with the response got worse, relative to control.
2. Problem with the response remained unchanged, relative to control.
3. Problem with the response declined, relative to control.
Time series design (no control) — Use Table 3 to interpret your evaluation.
1. Problem got worse after the response.
2. Problem unchanged after the response.
3. Problem went down after the response.
Multiple time series design (with control) — Use Table 3 to interpret your evaluation.
1. Problem with the response got worse, relative to control.
2. Problem with the response remained unchanged, relative to control.
3. Problem with the response declined, relative to control.
V. Overall Impact Evaluation Conclusions
1. Did the people or places that received the response show less of the problem than people or places that did not receive the response?
2. Did the problem decline at a faster rate after the response than it did before the response? This requires a time series design (with or without a control group) to answer. Do not check if a pre-post design with or without a control group was the only design used.
3. Can you eliminate all other plausible explanations for the change in the problem, other than that the
response caused the problem to decline? Use the list of differences (see bottom of part III) between the response and control groups to help answer these questions. Do not check if a pre-post design without a control group was the only design used.
These are judgment calls; the answers should be seen as your degree of confidence in the findings, rather than a totally objective assessment of what occurred. Other people who examine the same evidence could come to different conclusions. For this reason, these questions (and the question that follows) are best answered after several individuals with different perspectives have examined the assessment information.
Based on your answers to these three questions, ARE YOU REASONABLY CONFIDENT THAT THE
RESPONSE YOU EVALUATED WAS PRINCIPALLY RESPONSIBLE FOR THE CHANGES IN THE PROBLEM?
YES — If a thorough examination of these questions has been conducted and all of them have been checked, use Table 3 to interpret your results.
NO — If you cannot check all boxes to questions 1, 2, and 3, this is the appropriate answer. You must interpret Table 3 with extreme caution. Use Table 2 to interpret your results.
