How to Use SUPER

SUPER presents evidence using a framework that prioritizes scientific rigor while also accounting for the importance of flexibility and the value of emerging and community-driven practices.

For each practice, SUPER includes a table summarizing empirical evidence of effectiveness as documented in peer-reviewed research studies or other evaluations. To be considered as evidence for SUPER, studies/evaluations need to have been conducted in the United States. Information on implementation outside of the United States is included in the “Adaptations” section of each practice’s page.

Tables list outcomes that have been tested by substance, substance-related behavior (either reduced use/delayed initiation or changes in knowledge/attitudes/beliefs), the age of participants in studies/evaluations, and if studies/evaluations indicated effectiveness for any specific populations of focus (as defined above). If a population of focus comprised at least 80% of a study or evaluation sample, this information is listed in evidence tables.

In many cases, practices are designed for general substance use-related behaviors and general populations. Often, details of populations in studies or evaluations do not allow for any firm conclusions to be drawn about intervention effectiveness (or lack thereof) with any specific population of focus. In these cases, populations of focus are not included in evidence tables (“None indicated” is written).

Please note that just because evidence tables do not include information about practices’ impacts on a specific substance use behavior or population of focus, it does not necessarily mean that a practice does not have a positive effect on that population—it only means that its effectiveness has not been rigorously tested or documented.

SUPER has four ratings for the strength of empirical evidence supporting each practice’s effects: Strong Evidence, Moderate Evidence, Promising Evidence, and Practice-Based Evidence.

Strong Evidence indicates that most evidence from at least two quantitative studies or evaluations that meet high-quality research criteria demonstrates positive impacts. The preponderance of high-quality research must demonstrate these impacts. For comparative effectiveness studies that do not have a treatment-as-usual comparison group, practice outcomes that are favorable or equal to outcomes for comparisons that have an established evidence base are considered positive impacts.

To be considered high-quality for SUPER, research studies and evaluations must meet these three criteria:

• Have descriptions of both experimental and comparison groups.
• Use an intent-to-treat design and report outcomes for all participants (even if they did not complete a study protocol). If studies/evaluations demonstrate positive effects for subgroups (but not the entire sample), they are only considered evidence-effective if subgroup analyses were part of a pre-specified protocol or analytic plan.
• Use appropriate analytic methods are used for quantitative analyses and account for missing data. Appropriate methods are defined in the American Psychological Association’s Journal Article Reporting Standards for Quantitative Research Design.

To be considered high-quality for SUPER, quantitative research studies and evaluations must meet at least three of these criteria as well:

• Have clear descriptions of how participants were assigned to experimental and control groups.
• Control for baseline differences between groups and covariates.
• Use self-report measures from participants.
• Reliable outcome measures and/or measures with strong face or construct validity, with data collection methods described in detail.
• Report outcomes that are clinically and/or statistically significant.

Moderate Evidence indicates that quantitative evidence meets one of the following criteria:

• One study meets high-quality research criteria (described above) demonstrating positive impacts.
• 50% of more of high-quality research (described above) demonstrates positive impacts, but some high-quality research does not demonstrate these impacts.
• High-quality research (described above) demonstrates positive impacts, but they do not reach clinical or statistical significance.
• Research meets several high-quality research criteria (described above) but does not compare outcomes between experimental and comparison groups.

Promising Evidence indicates that quantitative studies and evaluations demonstrate positive effects but do not meet several high-quality research criteria as described above, so they cannot be used to draw firm conclusions about practice effectiveness. At a minimum, practices with promising quantitative evidence need to demonstrate positive differences in pre/post data collected from the same group of participants. Qualitative data is considered promising evidence if it meets methodological and reporting standards outlined in the American Psychology Association’s Journal Article Reporting Standards for Qualitative Research Design.

Practice-Based Evidence indicates that evidence from real-world practice and experience indicate that a practice is effective and a good for for the community where it is implemented. Many practices with practice-based evidence are considered Community Defined Evidence Practices (CDEPs), which are practices that have a strong level of support within specific communities.

Forms of practice-based evidence include aggregate experiences of service recipients or providers, practices that are accepted as effective by local communities through consensus, and practices that address the therapeutic and healing needs of individuals and families in culturally specific frameworks.

Whereas Strong, Moderate, and Promising evidence is established by researchers who collect and analyze data using methods that are generally standardized across the scientific community, practice-based evidence involves practitioners and community members collaborating with researchers to generate insights into programs’ real-world effectiveness and impacts.

For more information on practice-based evidence, see the California Evidence-Based Clearinghouse for Child Welfare’s commentary on evidence-based practice and practice-based evidence.  Frequently, practice-based evidence can be a first step toward identifying practices and interventions that work well and are good candidates for more empirically rigorous research. Practices that have not been researched but are known to utilize components that other research has proven effective are also considered practice-based evidence.