How to Use SUPER

When deciding what prevention practices to implement, the Substance Abuse and Mental Health Services Administration (SAMHSA) developed guidance on Selecting Best-Fit Programs and Practices: Guidance for Substance Misuse Prevention Practitioners.

SAMHSA recommends that prevention practitioners consider three key factors when selecting prevention practices — Conceptual Fit, Practical Fit, and Evidence of Effectiveness. The selection of an appropriate and effective prevention practice will be based on your community or organization’s prevention planning process to address the local priorities and short-term and long-term outcomes.

When deciding what practices to implement, you should consider all three of these factors—Conceptual Fit, Practical Fit, and Evidence of Effectiveness. Ideally, you can find practices that are good fits for your program in all three of these areas. However, it is also important to realize that the best practice for your program is going to be the one you can implement well.

Research shows that practices are more effective if they are implemented with appropriate staffing, resources, and fidelity. So while Conceptual Fit and Evidence of Effectiveness are important, Practical Fit is also critical when choosing prevention practices.

If you implement a practice that is not a good Practical Fit for your program, you are unlikely to see the kind of positive results that research shows. It is critical to strike a balance between Practical Fit, Conceptual Fit, and Evidence of Effectiveness when choosing which practices to implement. SUPER is designed to give you the information you need to make these decisions.

Empirical Evidence of Impact

SUPER presents evidence using a framework that prioritizes scientific rigor while also accounting for the importance of flexibility and the value of emerging and community-driven practices.

For each practice, SUPER includes a table summarizing empirical evidence of effectiveness as documented in peer-reviewed research studies or other evaluations. To be considered as evidence for SUPER, studies/evaluations need to have been conducted in the United States or Canada. Information on implementation outside of the United States or Canada is included in the ‘Adaptations’ section of each practice’s page.

Tables list outcomes that have been tested by outcome (substance use, substance use knowledge/attitudes/beliefs, positive relationships/social connectedness, mental health behavior/attitudes/functioning, mental health symptoms), the age of participants in studies/evaluations, and if studies and evaluations indicated effectiveness for any specific populations of focus (as defined above). If a population of focus comprised at least 80% of a study or evaluation sample, this information is listed in evidence tables.

In many cases, practices are designed for general populations. Often, details of populations in studies or evaluations do not allow for any firm conclusions to be drawn about intervention effectiveness (or lack thereof) with any specific population of focus. In these cases, populations of focus are not included in evidence tables (“None indicated” is written).

Please note that just because evidence tables do not include information about practices’ impacts on a specific outcome or population of focus, it does not necessarily mean that a practice does not have a positive effect on that population—it only means that its effectiveness has not been rigorously tested or documented.

Detailed Evidence Criteria

Practice-Based Evidence

Aligns with best practices and strategies in prevention, has a logic model or strong theoretical rationale

No pre/post data

Promising Evidence

Meets several criteria for strong, but not designed to be able to draw firm conclusions about intervention effectiveness

Has pre/post data or rigorous qualitative data

Moderate Evidence

Studies meet most criteria for Strong Evidence

Strong research evidence from just one quasi-experimental study

Strong Evidence

Eight criteria related to intervention clarity, evaluation, and statistically significant or meaningful impact

At least one randomized controlled trial or two quasi-experimental studies (non-random assignment, pre/post comparisons) that meet criteria

SUPER has four ratings for the strength of empirical evidence supporting each practice’s effects: Strong Evidence, Moderate Evidence, Promising Evidence, and Practice-Based Evidence.

Strong Evidence

Strong Evidence indicates that most evidence from at least two quantitative studies or evaluations that meet high-quality research criteria demonstrates positive impacts. The preponderance of high-quality research must demonstrate these impacts. For comparative effectiveness studies that do not have a treatment-as-usual comparison group, practice outcomes that are favorable or equal to outcomes for comparisons that have an established evidence base are considered positive impacts. To be considered high-quality for SUPER, research studies and evaluations must meet these three criteria:

• Have descriptions of both experimental and comparison groups.
• Use an intent-to-treat design and report outcomes for all participants (even if they did not complete a study protocol). If studies/evaluations demonstrate positive effects for subgroups (but not the entire sample), they are only considered evidence-effective if subgroup analyses were part of a pre-specified protocol or analytic plan.
• Use appropriate analytic methods for quantitative analyses and account for missing data. Appropriate methods are defined in the American Psychological Association’s Journal Article Reporting Standards for Quantitative Research Design.

To be considered high-quality for SUPER, quantitative research studies and evaluations must meet at least three of these criteria as well:

• Have clear descriptions of how participants were assigned to experimental and control groups.
• Control for baseline differences between groups and covariates.
• Use self-report measures from participants.
• Reliable outcome measures and/or measures with strong face or construct validity, with data collection methods described in detail.
• Report outcomes that are clinically and/or statistically significant.

Moderate Evidence

Moderate Evidence indicates that quantitative evidence meets one of the following criteria:

• One study meets high-quality research criteria (described above) demonstrating positive impacts.
• 50% or more of high-quality research (described above) demonstrates positive impacts, but some high-quality research does not demonstrate these impacts.
• High-quality research (described above) demonstrates positive impacts, but they do not reach clinical or statistical significance.
• Research meets several high-quality research criteria (described above) but does not compare outcomes between experimental and comparison groups.

Promising Evidence

Promising Evidence indicates that quantitative studies and evaluations demonstrate positive effects but do not meet several high-quality research criteria as described above, so they cannot be used to draw firm conclusions about practice effectiveness. At a minimum, practices with promising quantitative evidence need to demonstrate positive differences in pre/post data collected from the same group of participants. Qualitative data is considered promising evidence if it meets methodological and reporting standards outlined in the American Psychology Association’s Journal Article Reporting Standards for Qualitative Research Design.

Practice-Based Evidence

Practice-Based Evidence indicates that evidence from real-world practice and experience indicate that a practice is effective and a good fit for the community where it is implemented. Many practices with practice-based evidence are considered Community Defined Evidence Practices (CDEPs), which are practices that have a strong level of support within specific communities.

Forms of practice-based evidence include aggregate experiences of service recipients or providers, practices that are accepted as effective by local communities through consensus, and practices that address the therapeutic and healing needs of individuals and families in culturally specific frameworks. Whereas Strong, Moderate, and Promising evidence is established by researchers who collect and analyze data using methods that are generally standardized across the scientific community, practice-based evidence involves practitioners and community members collaborating with researchers to generate insights into programs’ real-world effectiveness and impacts.

For more information on practice-based evidence, see the California Evidence-Based Clearinghouse for Child Welfare’s commentary on evidence-based practice and practice-based evidence. Frequently, practice-based evidence can be a first step toward identifying practices and interventions that work well and are good candidates for more empirically rigorous research. Practices that have not been researched but are known to utilize components that other research has proven effective are also considered practice-based evidence.