Description of substantive hypotheses: Let’s Move It intervention
mechanisms of impact on physical activity
Date: 22 June 2017
Author group: Matti Heino, Vera Araujo-Soares, Reijo Sund, Tommi Vasankari, Wendy Hardeman, Falko
Sniehotta, Nelli Hankonen
The purpose of this document: This document describes the substantive hypotheses for testing the LMI
intervention impact on (main) hypothesised mediators of physical activity change, prior to accessing the
data. These hypotheses are based on the programme theory, which describes the assumed mechanisms of
impact of the intervention (developed during 2013-14).
The registration is associated with: A cluster-randomised controlled trial testing the effectiveness of Let’s
Move It (LMI) intervention to increase physical activity and decrease sedentary behaviour among
adolescents in vocational schools in Finland (see details in protocol; Hankonen et al., 2016). The primary
hypotheses included that “the intervention is effective in increasing moderate-to-vigorous-intensity
physical activity (MVPA), particularly among those with low or moderate baseline levels, and decreasing SB
[sedentary behaviour] among all students”. The current hypotheses focus on the student programme
component aimed at increasing adolescents’ leisure-time physical activity.
State of data analysis: Data has been collected (January 2015-April 2017). It has not yet been analysed by
co-authors apart from the internal pilot study (first two out of six batches of data collection, collected
January 2015-May 2015), conducted as planned in the study protocol to inform power calculations
(Hankonen et al., 2016). The first author, who conducted analyses for the internal pilot, as well as inspected
some baseline properties of the data (Heino, Knittle, Vasankari, & Hankonen, 2016), has not had access to
the full data. Changes in some of the primary and secondary outcomes (objective and self-reported physical
activity, bioimpedance) have been analysed by statisticians outside this author group (in January 2017).
Goal of research: The primary goal of the paper this document relates to, is to identify potential pitfalls and
solutions in quantitative process evaluation of complex behaviour change interventions, especially with
regard to mechanisms of change. The hypotheses presented below feed to this objective by investigating
the processes postulated by the Let’s Move It intervention programme theory.
Logic model
The Figure below depicts a simplified logic model for the programme theory under evaluation.
Primary substantive hypotheses
Hypotheses marked with “a” will be investigated as the main postulates of how the Let’s Move It
programme theory assumes the intervention would affect the hypothesised determinants. Hypotheses
marked under “b” indicate how the determinants would subsequently affect each other and physical
activity. In addition to testing these hypotheses, exploratory analyses will be conducted to describe the
processes which took place during the trial.
All of the following constructs are referred to in relation to PA (e.g. PA self-efficacy). The questionnaire
items defined the activity in question as “Physical activity during leisure time, which makes one feel out of
breath and increases heart rate”.
Time points to be examined are T1 (baseline), T3 (post-intervention; 2-month after baseline) and T4 (12month follow-up). All hypotheses refer to substantive assumptions of the programme theory.
1) Self-efficacy:
a. Intervention increases self-efficacy.
b. Self-efficacy positively affects:
i. Intention
ii. Use of behaviour change techniques (BCTs)
iii. PA
2) Autonomous motivation:
a. Intervention increases autonomous motivation
b. Autonomous motivation positively affects:
i. BCT use
ii. PA
3) BCT use:
a. Intervention increases BCT use
b. BCT use positively affects PA
4) Outcome expectations:
a. Intervention increases positive outcome expectancies
b. Positive outcome expectancies positively affect:
i. Intention
ii. Autonomous motivation
5) Descriptive norms:
a. Intervention increases descriptive norms
b. Descriptive norms positively affect:
i. Intention
ii. PA
6) Intention:
a. Intervention increases intention
b. Intention positively affects PA
Additional hypotheses
The following additional hypotheses are to be investigated, if allowed by time constraints:
1) Percentage of intervention sessions attended is positively related to intervention
effectiveness, i.e. increases in mediators and PA.
2) Action and coping planning:
a. Intervention increases action and coping planning
b. Changes in action and coping planning are positively related with changes in PA
3) Environment:
a. Intervention increases perceived environmental opportunities and resources
b. Perceived environmental opportunities & resources are positively associated
with PA
c. Perceived environmental opportunities & resources are positively associated
with self-efficacy
d. Perceived environmental opportunities & resources are positively associated
with intention
4) Autonomy support (only measured in intervention arm):
a. Intervention increases perceptions of autonomy support
b. Changes in autonomy support are positively related to changes in Autonomous
Motivation
5) Changes in autonomous motivation are positively related to changes in PA, particularly in
the presence of:
a. Sufficient self-efficacy (answer ≥ 4 on a 1-7 scale)
b. Moderate-to-high use of BCTs
c. Low controlled motivation and amotivation
d. Sufficient environmental opportunity (answer ≥ 4 on a 1-7 scale)
6) Changes in intention are positively related to changes in PA, particularly in the presence
of:
a. Sufficient self-efficacy (answer ≥ 4 on a 1-7 scale)
b. Self-regulatory BCT use
c. BCT use
d. Sufficient perceived environmental opportunities & resources (answer ≥ 4 on a
1-7 scale)
Analysis
As the programme theory assumes that increase in physical activity will happen particularly among those
with low or moderate baseline PA levels, only those participants will be included in the analyses. The same
definition will be used, as in the main outcome analyses.
Mediation analyses will be conducted, and their assumptions (Bullock, Green, & Ha, 2010) examined, unless
data indicates that analyses and not feasible. If time and staff resources allow, we will conduct e.g. path
analysis using structural equation modeling, mixed models and Bayesian estimation. A cross-validation
approach will be explored if deemed possible.
1-tailed tests will be used for theory evaluation, as they are more appropriate than 2-tailed tests in this
case, where the intervention programme theory makes directional predictions (Cho & Abe, 2013).
Exploratory findings will be presented as point estimates with 95% confidence intervals (and/or 95%
highest probability density or credibility intervals, in the case of Bayesian models). Gender, age
(intervention targeted 15-19 year-olds) and language proficiency (self-reported understanding and/or
speaking Finnish) will be examined as covariates in sensitivity analyses. Nestedness of participants in classes
and/or schools will be accounted for where possible.
Alpha will be set at 5%. To control for type 1 error rate, Holm-Bonferroni corrections will be used separately
for claiming the intervention had a direct effect on postulated determinants (action theory; correct for six
tests) and the effect of the determinants on each other and PA (conceptual theory; correct for 11 tests for
both T1-T3 and T1-T4  22 tests). Simple mediation models will be corrected for 11 tests. All estimates are
reported with the corresponding uncertainty intervals.
Analyses will be conducted using the latest RStudio (RStudio Team, 2015) running the latest available
version of R (R Core Team, 2015).
Discussion
Behavioural interventions which target smoking or physical inactivity, are often described as “complex”.
This means that they include multiple interacting components, targeting several facets of the behaviour on
different levels of the environment the individual operates in (Moore et al., 2015). This environment itself
can be described as a complex system (Shiell, Hawe, & Gold, 2008). Given that in complex systems, even
small effects can cause large differences (“the butterfly effect” in nonlinear dynamics; Hilborn, 2004), it is
difficult to determine a “smallest effect sizes of interest” (SESOI) for hypothesised mediators of behaviour
change interventions. This leads to the postulated processes, as described in intervention programme
theories, being largely unfalsifiable with traditional testing paradigms: If a point biserial correlation of 0.001
between intervention participation and a continuous variable would corroborate a theory, one would need
more than six million participants to detect the correlation at 80% power and an alpha of 5%. If researchers
are unable to reject a null hypothesis of no effect, they cannot determine whether there is evidence for a
null effect, or if a more elaborate sample was needed (e.g. Dienes, 2008, p. 30).
Equivalence testing (Lakens, 2016) remedies this in situations, where a SESOI can be determined, but when
testing programme theories of complex interventions, it is not clear that this is an option. Thus, researchers
are faced with an uncomfortable tradeoff: Either they must specify a SESOI (and thus, a hypothesis) which
does not reflect the theory under test or, on the other hand, unfalsifiability. With well-defined alternatives,
one can use Bayes factors to compare whether a point null data generator (effect size being zero) would
predict the data better than, for example, a model where most effects are near zero but half of them over d
= 0.2 (more technically; placing a half-cauchy distributed prior with a scale of 0.2 on the alternative model).
Researcher degrees of freedom
The many choices affecting data analysis, which can be made on an arbitrary basis and are shown to lead to
type 1 error rates exceeding 50%, are referred to as researcher degrees of freedom (Simmons, Nelson, &
Simonsohn, 2011). The rationale of pre-registration is to differentiate exploratory analyses from
confirmatory ones, and the less degrees of freedom, the more confirmatory the analyses.
In the current analysis, 11 out of 34 researcher degrees of freedom (with numbering below, as listed by
Wicherts et al., 2016) remain. To our knowledge, we are among the first to explicate this. None of the
following are rare in current practice of conducting quantitative process evaluations of complex behaviour
change interventions, and many of them would have been difficult to avoid completely, even in the
presence of abundant resources.
-
-
-
-
D3: Measuring the same dependent variable in several alternative ways
o Physical activity is measured in several self-report scales, as well as 7-day accelerometry.
Note: this is necessary as the two tap into different facets of PA.
o Some of the self-report scales of the postulated mediators of effect have been designed for
this study, and items may need to be dropped when deemed appropriate.
D6: Failing to conduct a well-founded power analysis
o Power analyses were calculated based on hypothesised changes in physical activity, not
changes in the process measures.
o There is very large uncertainty as to the size of expected—or necessary—changes in the
process measures, in order to achieve the intervention effect.
C2: Insufficient blinding of participants and/or experimenters
o Blinding of the participants or outcome assessors was impossible due to the nature of the
intervention and the financial resources available.
A1: Choosing between different options of dealing with incomplete or missing data on ad hoc
grounds
-
-
-
A2: Specifying pre-processing of data (e.g., cleaning, normalisation, smoothing, motion correction)
in an ad hoc manner
A3: Deciding how to deal with violations of statistical assumptions in an ad hoc manner
o Particularly relevant concerns include multicollinearity of mediators in larger path models
(which should contain all relevant variables for causally relevant conclusions), lack of
variance in variables, (e.g. time-) invariance of scales, etc.
o Some of the self-report scales of the postulated mediators of effect have been designed for
this study, and items may need to be dropped when lacking adequate psychometric
properties.
A4: Deciding on how to deal with outliers in an ad hoc manner
o See A12.
A5: Selecting the dependent variable out of several alternative measures of the same construct
o See D3.
A12: Using alternative inclusion and exclusion criteria for selecting participants in analyses
o There is room for interpretation on several grounds:
 It is probable that some of the questionnaires have been filled out in a non-serious
manner. An analysis of answering styles and time spent on questionnaire might
reveal this.
A13: Choosing between different statistical models
A14: Choosing the estimation method, software package, and computation of SEs
Appendix 1: Measures
Appendix 2: causal paths, regression models, levels of differences in
measurement points
Bibliography