Research Review Criteria

Research Description

Educators are continually inundated with information about programs claiming to be effective violence prevention and attendance improvement programs.

These program profiles can assist educators in making informed decisions about which programs really work and can best meet the needs of their school. The information included in the profiles covers basic research areas such as the type of evaluation design used by each program, the sample size, the instruments used, the number of outcomes examined, the statistical significance of the results, and the consistency of results over time and across multiple studies.

The Research Description contains information on the quality of each program's evaluation design and its evidence. This information will help educators to distinguish among programs with modest effects and weaker designs, and those with exemplary outcomes and stronger designs. The "ideal" research effort would

use an experimental design with careful comparisons between experimental and control groups;
have a sample size of 50 or more students in the experimental group;
gather information through valid and reliable instruments;
demonstrate statistically significant results in more than one outcome area;
demonstrate consistency of results across multiple studies;
provide follow-up data to show that the program's effects last beyond the immediate treatment; and
publish the results of the effort in a peer-review journal.

Obviously not all programs can meet these standards, but the more research standards addressed, the greater the confidence one can place in the results. Listed below is a definition of terms used on the profiles.

Design Design refers to the experimental plan/model for measuring program impact. A good evaluation design addresses two questions: did a change occur? is the change due to the program? Three types of designs are indicated: experimental (most rigorous); quasi-experimental with control group (rigorous); other (least rigorous).

Multiple Studies Multiple studies refers to repeated assessments of program outcomes through annual evaluations, longitudinal investigation or multiple site assessments. Consistency of results across studies significantly increases confidence in those results.

Sample Size The sample size refers to the number of experimental students for whom data were gathered. Research studies based on large sample sizes are generally more reliable, hence, studies with sample sizes greater than 50 (>50) are more rigorous.

Valid/Reliable Instruments Validity and reliability refer to the quality of the evaluation instruments -- whether questionnaires, interviews, performance-based assessments, or tests. An instrument is reliable if it provides consistent measurements, and it is valid if it provides accurate and relevant information.

Statistically Significant Results Statistical significance is determined as part of the data analysis process. It signifies whether or not a program has had an effect beyond which one would expect from chance alone. In social science research, results that occur less than five times out of 100 (p>.05) are considered statistically significant.

Outcomes Outcomes refer to the number of impact areas assessed by the program. Programs with more than one outcome are generally considered more powerful.

Published Published means whether or not a program's research was published in a reputable, professional peer review journal (see List of References). Research data that undergo this level of review generate a greater degree of confidence in the results.

Follow-up Data Follow-up data refer to data collected from participants after their initial involvement in the program (i.e. data on lasting/long-term effects). Programs that document sustained or lasting effects can be considered highly effective.

Awards Awards refer to any formal, external recognition received by the program.

Research Review Category Research review category is a classification system developed by this pilot project to help educators understand the scale of the program's research effort and the degree to which the effort has undergone external review.

Individual Program Profile Information

Home Page | About Sharing Success | Nominate A Program

Adult Education Programs | Early Childhood & K-12 Programs | Whole School Reform | Violence Prevention Programs

Design	Design refers to the experimental plan/model for measuring program impact. A good evaluation design addresses two questions: did a change occur? is the change due to the program? Three types of designs are indicated: experimental (most rigorous); quasi-experimental with control group (rigorous); other (least rigorous).
Multiple Studies	Multiple studies refers to repeated assessments of program outcomes through annual evaluations, longitudinal investigation or multiple site assessments. Consistency of results across studies significantly increases confidence in those results.
Sample Size	The sample size refers to the number of experimental students for whom data were gathered. Research studies based on large sample sizes are generally more reliable, hence, studies with sample sizes greater than 50 (>50) are more rigorous.
Valid/Reliable Instruments	Validity and reliability refer to the quality of the evaluation instruments -- whether questionnaires, interviews, performance-based assessments, or tests. An instrument is reliable if it provides consistent measurements, and it is valid if it provides accurate and relevant information.
Statistically Significant Results	Statistical significance is determined as part of the data analysis process. It signifies whether or not a program has had an effect beyond which one would expect from chance alone. In social science research, results that occur less than five times out of 100 (p>.05) are considered statistically significant.
Outcomes	Outcomes refer to the number of impact areas assessed by the program. Programs with more than one outcome are generally considered more powerful.
Published	Published means whether or not a program's research was published in a reputable, professional peer review journal (see List of References). Research data that undergo this level of review generate a greater degree of confidence in the results.
Follow-up Data	Follow-up data refer to data collected from participants after their initial involvement in the program (i.e. data on lasting/long-term effects). Programs that document sustained or lasting effects can be considered highly effective.
Awards	Awards refer to any formal, external recognition received by the program.
Research Review Category	Research review category is a classification system developed by this pilot project to help educators understand the scale of the program's research effort and the degree to which the effort has undergone external review.