Efficacy of a Two-Year Intensive Reading Intervention for Middle School English Learners With Reading Difficulties

Primary Hypothesis: Students receiving the RISE intervention will outperform those receiving Business as Usual (BAU) instruction across reading-related elements, including word reading, fluency, and comprehension at the end of year two of treatment.

Secondary Hypothesis: Students receiving the RISE intervention will outperform those receiving Business as Usual (BAU) instruction on measures of word reading and reading fluency following one year of treatment.

Research Design Overview

Participants in Cohort 1 will be recruited in fall of 2018 (6th and 7th grade) and Cohort 2 participants will be recruited in fall 2020 (6th and 7th grade). Participants who meet eligibility criteria will be randomly assigned to one of two conditions: (1) the RISE intervention (one 45 to 50-minute class, 5 days per week) or (2) a BAU comparison condition. Students identified as having disabilities and receiving special education who otherwise meet all criteria will be allowed to participate in the study and additional interventions will be documented. Treatment teachers are hired and trained by the investigators. Participants in the control condition typically participate in an elective during that class period (e.g., cooking, music), and sometimes in a preparation class for the state high stakes test. Teachers will be assigned to one school and responsible for providing instruction to students in groups ranging from 3 to 12 students.

Participants

The Project 3 sample will be composed of 6th and 7th grade students from six moderate to high poverty middle schools in Austin/San Antonio and in Houston with large numbers of students who are learning English as a second language. To evaluate the primary hypothesis, one randomized control trial integrating two nonoverlapping cohorts of participants (two-year intervention) will be conducted. The investigators will identify sample participants using extant school records and as described in the inclusion/exclusion criteria.

Cohort 1 will consist of 410 Els with significant reading difficulties, 205 assigned to Intervention (RISE) and 205 to the no intervention BAU comparison condition beginning in Fall 2018 (Year 1) and continuing in their assigned condition in year 2 (students 7th or 8th grade year). Cohort 2 will be a nonoverlapping sample of 410 students identified using the same inclusion/exclusion criteria and also randomly assigned to Intervention (RISE) or the no intervention BAU comparison condition and remaining in that condition through year 2. Students assigned to control may receive BAU services (researcher documented) from their schools, but will not receive researcher-provided reading intervention. These two cohorts (N = 820 students) will provide a sample to fully power data analysis.

Measures and Assessment Procedures

The data for Project 3 is collected at 4 time points (beginning and end of year for year 1 and year 2) for each cohort, permitting an analysis of treatment effects following one and two years of treatment (see intervention description). The research team that is responsible for hiring, training, and supervising data collection in schools, and for recruiting schools and participants, is experienced and has worked together for the past 10 years. Data collectors are blind to participants' randomly assigned condition.

Primary and secondary outcome measures are described in the section 9 (outcome measures).

Intervention Procedures

The intervention is described in detail in section 8 (Arms, Groups, and Interventions).

Fidelity of Implementation. Fidelity of implementation will be evaluated a minimum of three times per year per intervention teacher. All intervention teachers will audio-record all intervention lessons. A random sample of recordings will be selected and key indicators of intervention implementation adherence and quality of implementation will be evaluated by a coder. All coders will be trained and reliability will be established prior to independent coding.

Intervention Teachers. At each site, highly trained personnel hired and supervised by the investigators will deliver all treatments. These teachers will be provided 20 hours of training prior to implementation and then ongoing training and on-site support for at least 10 hours per month. All treatments will be provided daily in group sizes of about 8-11 students for 50 minutes a day.

Business as Usual (BAU) Comparison Condition. Students assigned to the BAU or comparison condition will participate in an elective class that includes such options as music, cooking, film, study time, or high stakes test preparation. These students will participate in the full normative educational program at their schools, but will not receive any instruction from the research team.

Data Management

Data Acquisition. The data management team will construct data acquisition forms and guidelines for completing those forms. When modifications to procedures or forms are necessitated, data management staff will communicate these changes to staff at each site and ensure that procedures and forms manuals are appropriately updated. Data management staff will also work to ensure consistency in forms layout, forms revision, and forms numbering. Data management and statistical staff also work with data managers to conduct data audits to ensure that error rates are negligible for all data fields.

Data Management. Data management personnel conduct all database design, management, and collection-related activities in a manner that results in all project data being written to the Texas Institute for Measurement, Evaluation, and Statistics (TIMES) data warehouse. The warehouse is designed for maximum data integrity and standardization within and across projects, while allowing for necessary flexibility across TIMES projects in their designs and specific measures and methods. The use of the warehouse as the primary project management data structure increases standardization across projects. The warehouse also automatically constructs an electronic audit trail of all data management activities that result in the modification of even a single data element, which together with standardization is essential to quality control in electronic databases.

Data Analysis

Overall Strategy, Methodology and Analyses. Descriptive features of the data will be examined prior to analysis. Non-normal dependent variables will be transformed (logarithmic, square root, inverse, etc.) as necessary and appropriate. Outliers will be identified using modified z-score analysis, and handled on a case-by-case basis according to their leverage and influence in specific models. Assumptions that errors are normally distributed, homoscedastic, and independent across sampling units and levels of the model will be evaluated by analyzing residuals. Residuals at higher levels are typically assumed to be multivariate normal and independent of lower level errors. The investigators will augment residual analyses with influence diagnostics. Heterogeneity will be addressed according to its apparent source(s), using nonlinear transformations of predictor and/or dependent variables as appropriate and necessary. The investigators will address the primary research hypothesis in the context of multiple group (RISE intervention v. BAU), multilevel regression and structural equation models (SEM).

Analytic Plan. Multiple group, multilevel regressions and SEMs will be fit to estimate the main effect of treatment on 1) word reading and fluency outcomes after 1 year of intervention and word reading, fluency, and comprehension after 2 years of intervention For each model, pretest scores will be centered as appropriate and used as level-1 covariates. School-level means at pretest will be included as "contextual effects" to minimize school-level variability, improving statistical power. Treatment main effects (assuming an intent-to-treat model) will be estimated at the student level by comparing conditional posttest means for the RISE group and the BAU group in the context of nested models. Fit indices for the group-specific model (i.e., unique posttest means estimated for each group) and the constrained model (i.e., posttest means fixed as equal across treatment and groups) will be compared and the difference (Δχ2) will be tested against the critical value that corresponds to the difference in degrees of freedom across models (Δdf). Effect sizes will be calculated as Hedges g.

Power and Effect Size. Data simulation was utilized to estimate statistical power for the main effect on student outcomes. These are intended as examples that can be applied across all primary and secondary outcomes. For estimating power, nesting at the site and cohort levels is ignored, as Intra Class Correlations in previous studies in similar schools have been negligible when controlling for pretest. Additionally, partial nesting was ignored based on previous findings for multiple-cohort, multi-year interventions and partially nested designs implemented in these two sites. When blocking on schools and when controlling for covariates at the school-level, clustering effects of site, cohort, and small instructional groups were trivial for comprehension-related outcomes (.00-.01 for the 2-year intervention with partial nesting). Variability in effect size did not differ statistically from 0 (σδ2 < .0001), suggesting fixed treatment effects.

Power analyses assumed a time 1 sample of 820 students across 6 schools (n=205/school). In the population model, the investigators randomly assigned cases to treatment or BAU within schools. The investigators specified student attrition as .10 annually for both conditions across sites and across cohorts with no differential attrition by condition or by interactions involving condition, suggesting a sample of approximately 740 students at assessment time 2 (spring year 1) and 670 at assessment time 4 (spring year 2). An annual attrition rate of 10% is consistent with previous studies conducted by the research team at similar schools. The investigators modeled a school-level covariate for the pretest to maximize power. In previous work, pretest measures of student reading outcomes (i.e., word reading, comprehension, etc.) accounted for about 75% of the outcome variance. Posttest reading scores were modeled as multivariate normal with a mean of 0 and variance of 1 in the untreated population. For the subpopulation assigned to treatment, the posttest mean was set at .20, which represents a population-level standardized mean difference between treatment and control). A literacy-related treatment effect of .20 represents a meaningful impact in this population, on both word-level reading measures and on reading comprehension, spelling and writing outcomes.

In the sample model, the investigators estimated posttest means as free parameters in the treatment and comparison conditions, using population values as starting values. The probability of rejecting the null hypothesis when it is false (i.e., statistical power) was .99 when modeling effects as fixed (σδ2 = 0) for samples of 740 and 670. Bias for the estimator72 was less than 1%. Under the same assumptions, a standardized mean difference of .10 was associated with power of at least .80. Power estimates were calculated for independent tests of statistical significance. However, many of the above contrasts are not independent. To control for inflated Type 1 error associated with multiple comparisons, the investigators propose the Step M approach for multilevel and mixed effects models and hierarchical data, which takes advantage of the dependence structure of individual test statistics and offers a more powerful approach than alternatives for handling family-wise error in a nested data.