Group Dynamics: Theory, Research, and Practice Group interactions and time: Using sequential analysis to study group dynamics in project meetings --Manuscript Draft-- Manuscript Number: GDN-2015-0132R3 Full Title: Group interactions and time: Using sequential analysis to study group dynamics in project meetings Abstract: Video-recorded observations of group interactions present a unique challenge for group researchers. This paper presents methodological advice how to perform sequential analysis when collecting observational timed-event data of group discussions. Sequential analyses is a statistical method that examines dynamic behavioral sequences in group interactions. To exemplify the method, we present data from one industry project team which was video-taped during 24 consecutive meetings. Meeting behaviors were coded into different categories (e.g., procedural and actionoriented communication). We compared sequential behavioral patterns in meetings from the first and second half of the project. We provide guidelines on the topic of interrater reliability and report a detailed psychometric analysis of the observational instrument. Overall, we show that positive procedural communication can inhibit dysfunctional communication patterns in group meetings. Our results also show that communication patterns of negative action-orientation only appeared in the second half of the project. This study extends previous group research on micro-sequential patterns with respect to larger scale macro-temporal group dynamics. Overall, we provide practical suggestions for researchers who aim to run observational research and aim to look for sequential dynamics in video-recorded team interactions. Article Type: Special Issue Article Keywords: group interaction patterns; timed-event data; sequential analyses; inter-rater reliability; video-recordings Corresponding Author: Florian Erik Klonek Technische Universität Braunschweig Braunschweig, GERMANY Corresponding Author E-Mail: [email protected] Corresponding Author Secondary Information: Corresponding Author's Institution: Technische Universität Braunschweig Other Authors: Vicenç Quera Manuel Burba Simone Kauffeld Author Comments: May 18th, 2016 Dear David Marcus, Thank you for your quick response and final feedback for our manuscript entitled "Group interactions and time: Using sequential analysis to study group dynamics in project meetings" to the call for papers on "Statistical Methods in Group Psychology and Group Psychotherapy" in Group Dynamics: Theory, Research, and Practice. We followed your recommendation and included the wording "supplementary online file" for all supplemental materials that we mention throughout manuscript. Thank you for allowing us to contribute to Group Dynamics: Theory, Research, and Practice. Best regards, The author team Powered by Edit orial Manager® and ProduXion Manager® from Aries Syst em s Corporat ion Title page with All Author Information Group interactions and time: Using sequential analysis to study group dynamics in project meetings Florian E. Kloneka, Vicenç Querab, Manuel Burbaa, & Simone Kauffelda a Technische Universität Braunschweig b Universitat de Barcelona Corresponding author: Florian E. Klonek Technische Universität Braunschweig Department of Industrial/Organizational and Social Psychology Spielmannstr. 19, 38106 Braunschweig Germany Phone: +49 531 3912566 Email: [email protected] Masked Manuscript without Author Information Abstract Video-recorded observations of group interactions present a unique challenge for group researchers. This paper presents methodological advice how to perform sequential analysis when collecting observational timed-event data of group discussions. Sequential analyses is a statistical method that examines dynamic behavioral sequences in group interactions. To exemplify the method, we present data from one industry project team which was video-taped during 24 consecutive meetings. Meeting behaviors were coded into different categories (e.g., procedural and action-oriented communication). We compared sequential behavioral patterns in meetings from the first and second half of the project. We provide guidelines on the topic of inter-rater reliability and report a detailed psychometric analysis of the observational instrument. Overall, we show that positive procedural communication can inhibit dysfunctional communication patterns in group meetings. Our results also show that communication patterns of negative action-orientation only appeared in the second half of the project. This study extends previous group research on microsequential patterns with respect to larger scale macro-temporal group dynamics. Overall, we provide practical suggestions for researchers who aim to run observational research and aim to look for sequential dynamics in video-recorded team interactions. Keywords: group interaction patterns, timed-event data; sequential analyses; interrater reliability; video-recordings The quote “No one can whistle a symphony. It takes an orchestra to play it” points out that the temporal dimension of group interactions are noteworthy: In the orchestraexample, we can observe how group interactions unfold over time and produce a continuous behavioral output. The study of groups requires methods that capture these dynamic, changing, and complex interactions (Arrow, Poole, Henry, Wheelan, & Moreland, 2004; Hewes & Poole, 2012). One promising method to adequately assess group dynamics is to use video-recorded observations: This type of research has been conducted in the study of organizational group meetings (Grawitch, Munz, & Kramer, 2003; Lehmann-Willenbrock, Allen, & Kauffeld, 2013; Meinecke & Lehmann-Willenbrock, 2015; Lehmann-Willenbrock, Meyers, Kauffeld, Neininger, & Henschel, 2011), quality improvement teams (Jackson & Poole, 2003), group therapy research (Osilla et al., 2015; Shorey, Martino, Lamb, LaRowe, & Santa Ana, 2015), and laboratory group discussions (Bonito, DeCamp, Coffman, & Fleming, 2006; Kuk, 2000). More recently, Leenders, Contractor, and Dechurch (2015, p. 6) proposed that The holy grail for research on team dynamics is to be able to watch a ‘movie’ of team process as it unfolds, then pause the movie, and be able to answer the questions: what will likely happen next and with what implications for outcomes? Observational methods can thus advance our theoretical and practical reasoning about the way group behavior dynamically changes over time (Weingart, 1997). The aim of this paper is to outline sequential analyses in the longitudinal study of group dynamics. In sum, our paper offers the following contributions. First, we provide readers with some background on observational data structures for video-recorded and coded interactions. Second, we provide guidelines in the assessment of observer agreement. Third, we explain necessary assumptions and common pitfalls for running sequential analyses. Fourth, we illustrate our explanations on micro- and macro-temporal group dynamics using data from 24 2 video-taped organizational project meetings. Fifth, we discuss challenges associated with the analyses of observational data. Studying Dynamic Group Interaction Processes with Observational Methods Observations of group interactions provide multiple information at once: time, location, and participants’ interactions. This depth of information enables the researcher to investigate group processes in detail and from a micro-perspective. When researchers want to look at this interactional stream of behaviors, they need to segment these microinteractions into single events (observational units), record their onset and offset times, and assign codes to each event. The result of this process are timed-event sequential data (Bakeman, Quera, & Gnisci, 2007): A behavioral sequence is represented as a series of time ordered codes for the observed events along with information about their onset and offset times. Both duration and events can be considered. Timed-event sequences are extremely useful because the temporal order of behaviors allows answering questions about dynamic communication patterns (i.e., what happens after what?). Furthermore, timed-event data represent a solution to measure team process in continuous time rather than at multiple discrete time points. This allows the researcher to track the actual development of the team’s process (which is very likely nonlinear and unevenly spaced across time) without having to make arbitrary and largely atheoretical decisions about time intervals. (Leenders et al., 2015, p. 5) Reliability of Group Dynamic Interactions Observational research is labor intensive and dependent on trained human observers (Folger, Hewes, & Poole, 1984; Klonek, Quera, & Kauffeld, 2015; Weingart, 1997). The problem of reliability is usually addressed by double-coding the data by at least two observers and estimating how they agree about a coded behavior. There is an abundance of 3 statistical parameters to be used for inter-rater reliability (IRR). Hallgreen (2012) provides a reader-friendly overview of coding-designs and IRR parameters. Hayes and Krippendorf (2007) also give guidance on calculating IRR when using more than two observers. Both reviews focus on code summary counts (e.g., behavioral frequencies). However, when researchers aim to investigate the micro-level of interactions (e.g., sequential analyses), they also need to record temporal and sequential information and should assess the point-by-point IRR (an example of this problem is given in Klonek, Quera et al., 2015). For this reason, the current study will provide guidance which parameters to report when using timed-event data of group interactions. Micro-temporal Dynamics of Group Interactions Our analysis builds on previous work from Kauffeld and Meyers (2009) who studied micro-temporal interactions of semi-autonomous work groups. The authors coded participants’ action-oriented communication (e.g., “complaining behavior” or “statements that express no interest in change”) using video-taped meetings. They hypothesized a reciprocity principle within these group interactions, that is, complaints should facilitate expression of further complaints. Furthermore, they assumed that positive procedural communication (e.g., “pointing out the main topic”) should inhibit further negative actionoriented communication. Using sequential analysis, the authors found evidence for mutual facilitation of negative action-oriented communication (e.g., complaining cycles) as well as for the hypothesized inhibitory effect of positive procedural communication on negative action-orientation communication (Kauffeld & Meyers, 2009). The current study uses the same coding instrument and seeks to replicate these findings. Based on this previous work, we hypothesize: H1: Positive procedural statements inhibit the emergence of negative action statements in group meetings. 4 H2: Negative action-oriented statements facilitate the emergence of further negative action statements in group meetings. Macro-temporal Dynamics of Group Interactions The study from Kauffeld and Meyers (2009) suggested that negative action-oriented meeting behaviors facilitate complaining behavior. While this research focused on microtemporal group dynamic interactions, our study significantly extends this work by studying the macro-temporal changes of one project team over an extended period. Therefore, this study is embedded in the groups-as-complex-systems framework which focuses on the evolution of team work (Arrow, McGrath, & Berdahl, 2000). This framework “proposes that many emergent patterns of group behavior should be independent of the characteristics of individual group members, although they may vary systematically, depending on constraints in the system’s environment.” (Arrow et al., 2004, p. 25). In extended project team work, larger temporal dimensions (e.g., the overall project phase as an environmental constraint) might moderate the interactive relation between group interactions. This problem refers to the challenge of temporal stability in team dynamic process research (Leenders et al., 2015). Leenders et al. argue that project teams are not temporally stable and that a static hypothesis like (e.g., “complaining behavior facilitates further complaining”) does not accurately reflect the effect of larger temporal group developments. In contrast to Kauffeld and Meyers (2009), we collected video-recordings from one single project team over the course of several consecutive meetings, that is, from the beginning of the project until the end of it. In line with Leenders et al. (2015), we assume that group dynamic interactions change over the larger temporal course of the project. Overall, the mutual working history that group members develop over the course of a project 5 should activate different types of group dynamics. As there are no observational studies on how group dynamic patterns evolve over the project-cycle, we formulate a research question about the relationship between group dynamic communication and project phases. Research question: Are group dynamic micro-communication patterns (e.g., complaining cycles) stable over the course of a project? Method Source of Data We videotaped 24 consecutive project team meetings from one organization from the automotive industry. Meetings were recorded from beginning until project termination. The project team consisted of 29 participants coming from different organizational departments. The majority of project members had a University degree (N = 21). Mean age was 38 years (Min = 26; Max = 53), majority of participants were male (N > 25) which is representative of the industry examined. Each meeting was attended on an average by 10 participants (SD = 2.1; Min = 6; Max = 16). The number of participants who were present in the first half of project meetings (i.e., t1 = meeting 1 – 12) was not different from number of participants in the second project half (i.e., t2: meeting 13 – 24; Mt1 = 11.33, SDt1 = 1.92; Mt2= 10.58, SDt2 = 2.43; t(22) = 0.84, p =.41). The number of participants that were present during meetings showed no relation with summary scores of act4teams group communication behavior (all p’s > .10). Group Communication Behavior within Meetings Project group communication behavior was coded with the act4teams coding scheme (e.g., Meinecke & Lehmann-Willenbrock, 2015). This coding scheme has been applied in numerous field studies about team meeting communication (e.g., Lehmann-Willenbrock et 6 al., 2011; Kauffeld & Lehmann-Willenbrock, 2012) and distinguishes seven broad categories of group communication. Problem-focused communication captures problem- or solution-oriented statements (e.g., “We have not received the single parts for building the prototype” – problem; “Then we need to postpone the test phase to next week” – solution). Positive procedural communication captures statements that help a project group to structure the meeting (e.g., Procedural question: “I guess, it’s my turn now?”, Procedural suggestion: “I’d say we get through this from top to bottom”). Negative procedural communication happens when participants discuss subjects that are off topic. Positive socio-emotional communication covers verbal interactions that build interpersonal relations (e.g., Encouraging participation: “Do you have some information for us?” or Expressing feelings: “This makes me really happy”). Negative socio-emotional communication captures negative feelings or critical statements (e.g., Criticizing: “Honestly, this department presents the same boring slides every year”). Positive action-oriented communication captures statements that promote change and the willingness to take project-related actions (e.g., Taking responsibility: “We will get this done – no worries!”; Action planning: “I will put this on my list to call Mr. X.”). Negative action-orientation encompass statements that express resistance to changes (e.g., Complaining: “This never works.” or No interest in change: “I don’t care.”). The supplemental material “Coded Transcript” provides a meeting transcript that exemplifies how the data was segmented and coded. Each category can also be subdivided into more fine-grained codes (cf., Table 1). Depending on the research question, researchers can either use macro-categories (e.g., Meinecke & Lehmann-Willenbrock, 2015) or the more fine-grained behavior codes (e.g., Kauffeld & Meyers, 2009). 7 Coding and Analysis Software: “How to run sequential analysis” Meetings were coded using the INTERACT software (Mangold, 2010; the supplementary online file “Software Options” provides a link for alternative coding tools). After coding of the verbal interaction stream, we converted the data into the SDS-format using ActSds-program (Bakeman & Quera, 2008; the supplementary online material provides a commented SDS file: “Sample SDS Data File”). We used the SDS-files for running sequential analyses in GSEQ (i.e., transition matrices and statistical test parameters, Bakeman & Quera, 2011). We have included two video tutorials in the supplementary online materials that give step-by-step instructions how to run sequential analyses: For GSEQ, see the supplementary online file “Sequential Analyses with GSEQ.WMV”; for running sequential analyses in SPSS, see the supplementary online material “Sequential Analyses with SPSS.WMV”). Both programs can be downloaded for free (the link is provided in the supplementary online file “Software Options”). Assessing IRR with Behavioral Coded Data How many sessions should be used for assessing IRR? To estimate sample size for IRR analysis, we used the following guidelines: 15 – 20 % of a corpus should be sampled for calculation of kappa (Bakeman, Deckner, & Quera, 2005). Yoder and Symons (2010) recommend to use ten or more sessions to calculate IRR based on summary measures. In our case, we double-coded five meetings, that is, about 300 minutes (5 hours). It is also possible to time-sample 20% of all meetings for the assessment of IRR. The latter approach provides a larger sample (N = 24 instead of N = 5) for calculating IRR based on summary scores (e.g., intra-class correlations, ICC). Parsing versus coding of behavioral data streams. Observers need to parse (i.e., “sequence” or “unitize”) and code (i.e., “categorize”) the observational stream (Guetzkow, 1950; Weingart, 1997). Parsing involves decisions how to segment speaker utterances 8 (Quera et al., 2007). Coding schemes can also differ in terms of parsing rules (e.g., Auld & White, 1956; Kauffeld & Lehmann-Willenbrock, 2012; Schermuly & Scholl, 2012; used all different parsing rules). Some researchers also use pre-parsed sessions, that is, one observer parses (and codes) the interaction stream, while the second observer (used for calculation of IRR) only codes pre-parsed segments. Calculating classical indices of IRR (e.g., Cohen’s kappa) is easier with pre-parsed data streams for standard statistical software packages (e.g., SPSS) as both observers code an equal amount of segments (cf., Bakeman et al., 2007, for a detailed discussion). Keeping observers independent and blind. Researchers should ensure independence and blindness of observers (Yoder & Symons, 2010). Independence means that an observer’s decision to code should not be influenced by another observer. Blindness means that observers should not know which sessions are chosen in the IRR analysis (Yoder & Symons, 2010). An observer who codes a pre-parsed observational stream is partly influenced by the observer who parsed the data because this “parser” determined the length of the observational unit. Keeping the main observer (who parses and codes) blind for IRR analysis can circumvent this problem. In our case, the entire sample of project meetings was parsed and coded independently by the main coder who was blind to the IRR sample. Calculation of IRR parameters. Observer agreement was estimated with Cohen’s kappa based on the seven categories using the following formula: κ = Po − Pc ⁄ 1 − Pc , where Po denotes the observed proportion of agreement and Pc denotes the expected proportion of chance agreement, that is, kappa estimates the observed level of agreement between two coders and corrects for chance agreement. As a second index of IRR, we computed ICCs to evaluate how observers agreed about summary measures of group communication (Bakeman & Quera, 2011). Agreement about summary measures can be either absolute (criterion-referenced agreement) or relative 9 (norm-referenced agreement). Absolute ICCs indicate the degree of strict agreement, that is, whether observers obtained exactly the same values for the measures, whereas relative ICCs simply indicate whether the values can be rank-ordered the same way. Analytical Approach: Sequential Analysis To test our hypotheses, we used sequential analysis. This method examines how spontaneous behaviors are associated with subsequent behaviors and allowed to systematically test whether negative action-oriented behaviors facilitate further negative action-oriented statements. The association between behaviors may be analyzed at different time lags. Lag1 indicates the association between a given behavior (utterance) and the immediately following target behavior. We generated lag-sequence matrices containing joint frequencies for sequentially adjacent behaviors with given behaviors (i.e., preceding) at lag0 and subsequent behaviors (i.e., targets) at lag1. We first ran sequential analyses on individual sessions: However, table totals for the 7x7 matrices for each meeting were too small (M = 1107, SD = 352, Min = 423, Max = 1858) to allow interpretation of sequential analysis based on these individual sessions. Therefore, we computed statistics based on pooled sessions (i.e., meetings) (cf., Klonek, Lehmann-Willenbrock, & Kauffeld, 2014; Kauffeld & Myers, 1997). Note that sequential analyses “from pooled data may differ from the same statistics computed as the mean of individual cases” (i.e., meetings, Bakeman & Quera, 2011, p. 125). However, pooling is always recommendable when the events of interest (e.g., most meetings had session-row sums < 30 for negative action-oriented behavior) are so rare that the accuracy of sequential statistics become questionable (for a detailed argument about using pooled results, see Bakeman & Quera, 2011, p.125-126). 10 We used log-linear analysis (Bakeman & Robinson, 1994) to test homogeneity (i.e., similarity of transition matrices across meetings, cf., Gottman & Roy, 1990) of lag1 sequential associations across meetings. A three-dimensional (24 meetings x 7 given behaviors x 7 target behaviors) lag1 frequency table showed that given-target associations were not independent from meetings (log-likelihood G2(1104)= 2176.6, p < .01). Since many table cell frequencies were low (39.8% of the 1,176 cells had a 0 or 1 frequency) that result cannot be taken as conclusive. In order to reduce non-homogeneity and since we were interested in macro-temporal influences on the project group’s dynamics, we pooled sequential analysis across meetings from the first versus the second project half. While most psychological research designs focus on individual differences, the current analytic approach focused on the behavioral talk turn as the relevant unit – it theoretically aligns with the groups-as-complex systems framework (Arrow et al., 2000). Sequential analyses can also be used to compare interaction processes between different group types. For example, a researcher might use an experimental design to test sequential associations in meetings for teams that received a training (i.e., an intervention about running better meetings) in comparison to teams with no training (i.e., control group). The researcher could compare sequential associations (e.g., mean Yule’s Q values for negative actionoriented behavior) between the intervention and the control group. Results IRR Analysis Table 2 shows the agreement matrix for the two observers. Disagreements (i.e., confusions of codes) are shown in off-diagonal cells and agreements in diagonal ones, that is, IRR is high when in-diagonal cell counts are high and off-diagonal cell counts are low. ------------------------------------ 11 Insert Table 2 about here ----------------------------------Since off-diagonal cells indicate codes that cause confusions, the matrix can be useful when developing a coding scheme. In our data, we can see that off-diagonal cells for problem-focused and positive procedural communication are high (Observer 1 and 2 confused between N = 114 – 119 events). That is, definitions for these categories should be refined in future research and we should consider including more rules and examples in the coding manual to help observers better distinguish these categories. We calculated an overall kappa (K = .78, p <. 01) as an index of overall point-bypoint agreement based on the matrix in Table 2. Furthermore, we calculated code-specific kappas. Several authors have provided guidelines to interpret kappa values: Landis and Koch (1977) proposed that values from 0 to .2 = slight agreement, .21 to .40 = fair agreement, .41 to .60 = moderate agreement, .61 to .80 = substantial agreement and .81 to 1.0 = almost perfect agreement. Sachs (1999) is more conservative with <.40 = poor, .41–.61 = considerable, .61–.80 = strong, and .81–1.00 = approaches perfect agreement. So far, there is no clear preference about which of the categorizations should be used (Bakeman, Quera, McArthur, & Robinson, 1997). Overall, a look at the agreement matrix is equally important as kappa when evaluating IRR (cf., Table 2). Kappa interpretation also depends on the prevalence and number of codes (Bakeman & Quera, 2011). In our example, we calculated kappa for an instrument with seven codes and high variability of code prevalence. For example, negative procedural communication was very infrequent (N = 10 for coder 1 and N = 28 for coder 2). The totals for each observer also tell us that there might be a positive or negative observer bias for procedural communication. These issues are ideally discussed during observer training. 12 As a second IRR-measure, we calculated code-specific ICCs based on comparisons of the overall frequencies. The category negative procedural communication showed a low ICC in our example (ICC = .36). Overall, we would not recommend to derive conclusions based on the results for that code. Prerequisites for Interpretation of Sequential Analysis Do I have sufficient data to interpret sequential results? Sequential analyses start by tallying the joint frequencies between adjacent verbal behaviors. In order to obtain statistical parameters that can be trusted, the expected frequency of any transition should be at least five, and row sums of transitions should be larger than 30 (e.g., Yoder & Symons, 2010). When sample size (i.e., the table total) is small, some expected frequencies can be less than five. In that case, there are two strategies to ensure that the expected frequencies are at least equal to five. The first is to form combinations of categories that are theoretically meaningful: Since we calculated transition matrix based on seven macro-categories and not the 44 micro-codes, we have already taken advantage of this first analysis strategy. A second strategy is to pool sessions (i.e., meetings) (cf., Klonek et al., 2014; Kauffeld & Myers, 1997). Sometimes – as in our analysis -, both strategies can be necessary. With GSEQ, we tallied two 7x7 matrices with all act4teams behaviors at lag0 (table rows) and act4teams behaviors at lag1 (table columns). The first sequential matrix was obtained by pooling the joint frequencies over all verbal behaviors from meetings of the first project phase (meetings 1 – 12, with total tables = 14,169), whereas the second sequential matrix was obtained by pooling the joint frequencies of all verbal behaviors from the second project phase (meetings 13 – 24, with total tables = 12,404). -----------------------------------13 Insert Table 3 about here -----------------------------------Lag0 to lag1 joint frequencies and their adjusted residuals (ADJR) from the first project phase meetings are shown in Table 3. Joint frequencies are not equally distributed for all behavioral codes: Row totals for negative procedural communication is less than 30; therefore, we decided not to interpret adjusted residuals for this code. -----------------------------------Insert Table 4 about here -----------------------------------Lag0 to lag1 joint frequencies and their adjusted residuals from the second project phase meetings are shown in Table 4. Again, row totals for negative procedural communication was too small and precluded further analysis of this category. Is there a general association between verbal group behaviors? At first, we tested for a global association within the sequential matrix of joint frequencies. A significant χ²value indicates that associations between lagged variables are not determined by chance (i.e., given codes are followed more often by some codes and less often by others). If there is a global sequential association, researchers can investigate which specific pairs of codes are associated. In our analysis, both matrices indicated an overall significant non-random pattern (for the first project phase: χ²(36) = 298.25, p < .01; for the second project phase: χ²(36) = 623.34, p < .01). Therefore, we further looked for cell-specific associations. How can we test sequential hypotheses about dynamic group interactions? In order to test specific hypotheses about sequential effects, we used GSEQ to calculate adjusted residuals (Bakeman & Quera, 2011). Adjusted residuals are standardized raw residuals (based on the difference between the observed and expected frequency). This cellspecific statistic reveals whether sequential relationships between adjacent behaviors are 14 more or less likely to be expected by chance. If adjusted residuals are > 1.96, there is a significant positive association; if adjusted residuals < -1.96 there is negative association between adjacent behaviors, respectively. In our first hypothesis, we expected that positive procedural statements inhibit the emergence of negative action-oriented statements. Our results support this hypothesis as there was a significant negative sequential association between positive procedural communication and negative action-oriented communication in both the first project phase (ADJR(P+ AO-)= -3.42, p <.01) and the second project phase (ADJR(P+ AO-)= -3.35, p <.01). In our second hypothesis, we expected that negative action-oriented statements facilitate further negative action-orientation. For meetings from the first project phase, we found no significant sequential associations for negative action-oriented behavior (ADJR(AO= -0.1, p = .92), that is, this result does not support our second hypothesis. However, in AO-) the second project phase, negative action-oriented behavior was significantly sequentially associated with itself (ADJR(AO- AO-)= 5.94, p < .01). Overall, meeting interactional patterns from the first project phase are at odds with the findings from Kauffeld and Meyers (2009) whereas meetings from the second project phase underpin a sequential group dynamic patterns of negative action-oriented behavior. In our research question, we asked whether group dynamic communication patterns are stable over the course of a project. In our example, we compared transition matrices in the first half and second half of the project. This comparison revealed that the inhibitory effect of procedural communication on negative action-orientation was consistent across both project phases, whereas the dynamic sequential pattern for negative action-orientation was not stable and only emerged in the second project half. How large are sequential effects of the group interactions? Since values of adjusted residuals are dependent on the overall sample size of the transition matrix, it is not 15 possible to compare effects across tables. To determine the strength of association, we used GSEQ to compute Yule’s Q – an index that is independent of sample size and ranges from +1 for a perfect positive relationship to –1 for a negative relationship. Effect sizes like Yules Q are easier to interpret as it is similar to the more familiar correlation coefficient. In our data, the inhibitory effect of positive procedural communication on negative action statements was moderate (Q(P+ AO-)= -.47 in the first phase) to strong (Q(P+ AO-)= -.61 in the second phase). The self-enhancing effect of negative action-orientated statements on themselves was close to zero in the first phase (Q(AO- AO-)= .02) and strong in the second project phase (Q(AO AO-) = .73) Discussion The current paper has shown how to analyze group dynamics using video-recorded observations of project meetings. Overall, we built on previous work from Kauffeld and Meyers (2009) who reported patterns of negative action-oriented communication in meetings and positive effects for positive procedural statements on complaining behavior. Our study extended this work by investigating the same type of group communication behavior within a project team over an extended period of time. The previous sequential study about team dynamics in work meetings has shown that negative action-oriented behavior is less likely to appear in group meetings if preceded by positive procedural statements (i.e., clarifying goals, time management etc.; Kauffeld & Meyers, 2009). The current study has replicated this finding. Furthermore, we presented an effect size measure (i.e., Yules Q) that can be reported for sequential associations. Kauffeld and Meyers (2009) also reported group dynamic effects for negative actionoriented statements, that is, complaints by one co-worker were more likely if preceded by a previous complaint. The current study investigated these micro-sequential patterns using a larger timeframe (i.e., consecutive meetings). Sequential patterns between negative action16 oriented behaviors did not occur in the first project phase and only emerged in the second project phase. Kauffeld and Meyers (2009) suggested that complaining cycles are more likely for groups who share a common work history (Kauffeld & Meyers, 2009). As the temporal stability of project teams only evolves gradually over time (Hollenbeck, Beersma, & Schouten, 2012), our results support the idea that sequential micro-dynamics of negative action-oriented statements should only occur after the team members were familiar to each other (cf., Kauffeld & Meyers, 2009). In other words, patterns in negative action-oriented behavior were not stable over time and this result theoretically contributes to the assumption that group dynamics can change over larger temporal periods (cf., Leenders et al., 2015). This result also supports the groups-as-complex system approach which assumes that environmental constraints like project phases can impact group dynamic patterns (Arrow et al., 2004). It is noteworthy that the positive effects of procedural communication was independent of the project phase – speaking for the stability of this specific communication behavior. Overall, our results indicate that temporal stability applies only to specific communication behaviors (i.e., procedural statements) and therewith extends theorizing about temporal stability of group dynamics. Furthermore, our analysis exemplifies how sequential dynamics between group interactions might be moderated by macro-temporal group developments and that this temporal stability needs to be considered in group dynamics research (Leenders et al., 2015). Pitfalls and Problems of Sequential Analysis The coding scheme from the present study has been used in several previous studies (e.g., Kauffeld & Lehmann-Willenbrock, 2012; Kauffeld & Meyers, 2009; LehmannWillenbrock et al., 2013). However, these studies only reported a global reliability measure (i.e., overall kappa coefficient) while using category measures in subsequent analyses. Hewes (1985) pointed out that an underreporting of category-specific reliability can result in 17 systematic biases – particularly when testing temporal dependencies in sequential analyses. Therefore, the current study also reported category-specific reliabilities (i.e., ICCs and kappas). We found low reliabilities for negative procedural communication, therewith underpinning Hewes’s claim (1985) to report category specific reliabilities. Overall, we recommend to report category-specific reliability based on summary measures (e.g., ICCs) when interested in static snapshots of group behavior (i.e., aggregating verbal categories over a time, cf., Weingart, 1997). For example, researchers could ask “how is the frequency of action-oriented communication related to meeting satisfaction measured via questionnaires?” (cf., Kauffeld & Lehmann-Willenbrock, 2012). In this case, reporting category-specific ICCs are sufficient as it only takes into account agreement about code frequencies (cf., Klonek, Quera et al., 2015). However, when looking at dynamic interactions (e.g., does negative action-oriented behavior facilitate itself?), we need to ensure high point-by-point reliability. Researchers can either consider statistical techniques that remove bias of unreliable categories or remove them in subsequent analyses (cf., Hewes, 1985). The current study used a pre-parsed data-set. When researchers use independently parsed (i.e., unitized) and coded data, we recommend to also report unitizing reliability (detailed descriptions are given in Guetzkow, 1950; Weingart, 1997). This indicates how well observes consistently segmented the stream of interaction. Overall, sequential analyses can produce systematically biased parameter estimates when assuming that data was coded without error (Hewes, 1985). High quality coder training, reporting of code specific reliabilities, and removing the effect of unreliable categories with statistical techniques (cf., Hewes, 1985; Holsclow, Hallgren, Steyvers, Smyth, & Atkins, 2015) can minimize this problem. 18 In our study, we pooled across several meetings to obtain sequential associations of group interactions. Even though this approach is necessary when table statistics from individual sessions are too small (Bakeman & Quera, 2011) and is in line with previous research (Kauffeld & Meyers, 2009), our results are somehow limited. Our homogeneity test indicated that sequential associations were not similar across all 24 meetings. Therefore, we ran sequential analyses separately for the first and the second project half and showed that sequential associations changed on a macro-temporal scale. Tests of homogeneity or stationarity (O’Connor, 1999) can help to uncover factors that are responsible for nonhomogenous associations between meetings. For example, we cannot exclude if certain individual factors (e.g., particular group compositions) might have contributed to the result that negative action-oriented behavior patterns were absent in the first half of the project. Future longitudinal studies should replicate our results using project meetings that maintain the teams’ participant composition. Finally, our current study has used naturally occurring group interactions and the analysis does not allow us to draw causal conclusions. Causal inferences can only be made by employing a methodological design in which the study coordinator can purposefully vary the conditions of an independent variable (e.g., by using a confederate meeting facilitator that shows positive procedural communication at previously determined time points). Practical Implications for Applied Research Even though observational research is labor-intensive, it has particular value for designs in applied research fields. First, observational methods are less obtrusive in comparison to self-reports in repeated measurement designs (Hill, White, & Wallace, 2014) because participants are disburdened from filling out questionnaires. Especially in applied field research, participants are often reluctant to fill out questionnaires due to time and work priorities (e.g., Barclay, Todd, Finlay, Grande, & Wyatt, 2002). Observational methods are 19 an elegant solution to this problem because they allow for the investigation of interaction processes based on information that participants naturally emit. Second, participants cannot be sensitized for the construct under investigation, that is, they could not know that we studied their procedural or action-oriented behavior in relation to project phases. We could also have focused on participants’ interpersonal styles (Schermuly & Scholl, 2012), change readiness (Klonek, Paulsen, & Kauffeld, 2015), and even linguistic style matching (Taylor & Thomas, 2008). The data captured on video is rich and allows for manifold analytical possibilities. In contrast, filling out surveys requires to cognitively and actively process information when responding to items, which in turn may affect the quality of the data (Rogelberg, Fisher, Maynard, Hakel, & Horvath, 2001). Third, findings from sequential analyses are easier to translate for practitioners. In our research example, we were able to identify specific behaviors (i.e., procedural statements) that can potentially inhibit dysfunctional meeting behaviors. Furthermore, mutual facilitation of dysfunctional behaviors like complaining are more likely to occur when project teams have already worked for an extended period. Practically, these findings help trainers, supervisors, or group facilitators to make deliberate choices about when (i.e., in the second project half) and how to behaviorally intervene (e.g. by using a procedural suggestions) when negative group dynamics like complaining cycles occur. In a similar vein, researchers in the field of clinical psychology have started using sequential analyses to test hypotheses in socio-interactional interventions like motivational interviewing (e.g., Apodaca et al., 2016; Gaume, Gmel, Faouzi, & Daeppen, 2008). The results from this line of research can help to empirically identify which therapist behaviors either facilitate or inhibit motivational client statements and thusly mediate the success of behavior-change interventions (Moyers, Martin, Houck, Christopher, & Tonigan, 2009). While most sequential research in motivational interviewing focuses on dyadic interactions (therapist- 20 clients), future research could further test how these interpersonal dynamics transfer to group-based motivational interventions (e.g., Osilla et al., 2015). Conclusion The present study presented methodological advice how to use sequential analyses for the study of video-recorded meetings. Timed-event data allows researchers to investigate how different group behaviors are interrelated and dependent on contextual factors - instead of looking at static information. Overall, this study showed the potential of studying the dynamic nature of group communication behavior using sequential analyses. We have given methodological guidelines and extended research on the dynamic effects of procedural and action-oriented communication team member micro-interactions. In short, we replicated inhibitory effects of positive procedural communication on negative action-oriented statements. Furthermore, we have provided first evidence that negative action-oriented behavior patterns only occur when team members already share a working history and not in the beginning of a project phase. 21 References Apodaca, T. R., Jackson, K. M., Borsari, B., Magill, M., Longabaugh, R., Mastroleo, N. R., & Barnett, N. P. (2016). Which individual therapist behaviors elicit client change talk and sustain talk in motivational interviewing? Journal of Substance Abuse Treatment, 61, 60-65. Doi: 10.1016/j.jsat.2015.09.001 Arrow, H., McGrath, J. E., & Berdahl, J. L. (2000). Small groups as complex systems: Formation, coordination, development, and adaptation. Newbury Park, CA: Sage. Arrow, H., Poole, M. S., Henry, K. B., Wheelan, S., & Moreland, R. (2004). Time, change, and development the temporal perspective on groups. Small group research, 35(1), 73-105. Auld, F., Jr., & White, A. M. (1956). Rules for dividing interviews into sentences. The Journal of Psychology: Interdisciplinary and Applied, 42, 273–281. doi: 10.1080/00223980.1956.9713040 Bakeman, R., Deckner, D. F., & Quera, V. (2005). Analysis of behavioral streams. In D.M Teti (Ed.), Handbook of research methods in developmental science, (pp. 394–420). Oxford, UK. Bakeman, R., & Quera, V. (2008). ActSds and OdfSds: Programs for converting INTERACT and The Observer data files into SDIS timed-event sequential data files. Behavior Research Methods, 40 (3), 869-872. Bakeman, R., & Quera, V. (2011). Sequential analysis and observational methods for the behavioral sciences. New York, NY: Cambridge University Press. Bakeman, R., Quera, V., & Gnisci, A. (2007). Observer agreement for event sequences: Methods and software for sequence alignment and reliability estimates. Behavior Research Methods, 39(1), 39-49. 22 Bakeman, R., Quera, V., McArthur, D., & Robinson, B.F. (1997). Detecting sequential patterns and determining their reliability with fallible observers. Psychological Methods, 2 (4), 357-370. Bakeman, R., & Robinson, B.F. (1994). Understanding log-linear analysis with ILOG. An interactive approach. Hillsdale, NJ: Erlbaum. Barclay, S., Todd, C., Finlay, I., Grande, G., & Wyatt, P. (2002). Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of GPs. Family Practice, 19(1), 105–111. doi:10.1093/fampra/19.1.105 Bonito, J. A., DeCamp, M. H., Coffman, M., & Fleming, S. (2006). Participation, information, and control in small groups: An actor-partner interdependence model. Group Dynamics: Theory, Research, and Practice, 10(1), 16. doi: 10.1037/10892699.10.1.16 Folger, J. P., Hewes, D. E., & Poole, M. S. (1984). Coding social interaction. Progress in Communication Sciences, 4 , 115–161. Gaume, J., Gmel, G., Faouzi, M., & Daeppen, J. B. (2008). Counsellor behaviours and patient language during brief motivational interventions: a sequential analysis of speech. Addiction, 103(11), 1793-1800. Doi: 10.1111/j.1360-0443.2008.02337.x Gottman, J. M., & Roy, A. K. (1990). Sequential Analysis: A Guide for Behavorial Researchers. Cambridge University Press. Grawitch, M. J., Munz, D. C., & Kramer, T. J. (2003). Effects of member mood states on creative performance in temporary workgroups. Group Dynamics: Theory, Research, and Practice, 7(1), 41. doi: 10.1037/1089-2699.7.1.41 Guetzkow, H. (1950). Unitizing and categorizing problems in coding qualitative data. Journal of Clinical Psychology, 6, 47–58. 23 Hallgreen, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutor Quant Methods Psychology, 8(1), 23-34. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. doi:10.1080/19312450709336664 Hewes, D. E., & Poole, M. S. (2012). The analysis of group interaction processes. In A. B. Hollingshead & M. S. Poole (Eds.), Research methods for studying groups and teams: A guide to approaches, tools, and technologies (pp. 358–385). New York, NY: Routledge. Hill, A. D., White, M. A., & Wallace, J. C. (2014). Unobtrusive measurement of psychological constructs in organizational research. Organizational Psychology Review, 4(2), 148–174. doi:10.1177/2041386613505613 Hollenbeck, J. R., Beersma, B., & Schouten, M. E. (2012). Beyond team types and taxonomies: A dimensional scaling conceptualization for team description. Academy of Management Review, 37(1), 8-106. Holsclaw, T., Hallgren, K. A., Steyvers, M., Smyth, P., & Atkins, D. C. (2015). Measurement error and outcome distributions: Methodological issues in regression analyses of behavioral coding data. Psychology of Addictive Behaviors. Advance online publication. doi:10.1037/adb0000091 Jackson, M. H., & Poole, M. S. (2003). Idea‐generation in naturally occurring contexts. Human Communication Research, 29(4), 560-591. Kauffeld, S., & Lehmann-Willenbrock, N. (2012). Meetings matter: Effects of team meetings on team and organizational success. Small Group Research, 43(2), 130–158. doi:10.1177/1046496411429599 24 Kauffeld, S., & Meyers, R. A. (2009). Complaint and solution-oriented circles: Interaction patterns in work group discussions. European Journal of Work and Organizational Psychology, 18(3), 267–294. doi:10.1080/13594320701693209 Klonek, F. E., Lehmann-Willenbrock, N., & Kauffeld, S. (2014). Dynamics of resistance to change: A sequential analysis of change agents in action. Journal of Change Management, 14(3), 334–360. doi:10.1080/14697017.2014.896392 Klonek, F.E., Paulsen, H. & Kauffeld, S. (2015). They meet, they talk ... but nothing changes: Meetings as a focal context for studying change processes. In J. A. Allen, N. Lehmann-Willenbrock, & S. G. Rogelberg (Eds.), The Cambridge handbook of meeting science. New York, NY: Cambridge University Press. Klonek, F.E., Quera, V. & Kauffeld, S. (2015). Computer-supported coded interactions in Motivational Interviewing: What are the advantages for process researchers? Computers in Human Behavior, 44, 284-292. doi: 10.1016/j.chb.2014.10.034 Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Sage Publications; Beverly Hills, CA. Kuk, G. (2000). "When to speak again": Self-regulation under facilitation. Group Dynamics: Theory, Research, and Practice, 4(4), 291. doi:10.1037/1089-2699.4.4.291 Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. doi:10.2307/2529310 Leenders, R., Contractor, N. S., & DeChurch, L. A. (2015). Once upon a time: Understanding team dynamics as relational event networks. Organizational Psychological Review. Advance online publication. doi:10.1177/2041386615578312 Lehmann-Willenbrock, N., Allen, J. A., & Kauffeld, S. (2013). A sequential analysis of procedural communication in organizational meetings: How teams facilitate their 25 meetings. Journal of Applied Communication Research, 41(4), 365–388. doi:10.1080/00909882.2013.844847 Lehmann-Willenbrock, N., Meyers, R. A., Kauffeld, S., Neininger, A., & Henschel, A. (2011). Verbal interaction sequences and group mood: Exploring the role of planning communication. Small Group Research, 42, 639–668. doi:10.1177/1046496411398397 Mangold (2010). INTERACT quick start manual V2.4. Mangold International GmbH. Retrieved from www.mangold-international.com Meinecke, A. L., & Lehmann-Willenbrock, N. (2015). Social dynamics at work: Meetings as a gateway. In J. A. Allen, N. Lehmann-Willenbrock, & S. G. Rogelberg (Eds.), The Cambridge handbook of meeting science (pp. 325-356). New York, NY: Cambridge University Press. Moyers, T. B., Martin, T., Houck, J. M., Christopher, P. J., & Tonigan, J. S. (2009). From insession behaviors to drinking outcomes: a causal chain for motivational interviewing. Journal of consulting and clinical psychology, 77(6), 1113. Doi: 10.1037/a0017189 O'Connor, B. P. (1999). Simple and flexible SAS and SPSS programs for analyzing lagsequential categorical data. Behavior Research Methods, Instrumentation, and Computers, 31, 718-726. Osilla, K. C., Ortiz, J. A., Miles, J. N., Pedersen, E. R., Houck, J. M., & D’Amico, E. J. (2015). How group factors affect adolescent change talk and substance use outcomes: Implications for motivational interviewing training. Journal of counseling psychology, 62(1), 79. doi:10.1037/cou0000049 Rogelberg, S. G., Fisher, G. G., Maynard, D. C., Hakel, M. D., & Horvath, M. (2001). Attitudes toward surveys: Development of a measure and its relationship to respondent behavior. Organizational Research Methods, 4(1), 3–25. doi:10.1177/109442810141001 26 Sachs, L. (1999). Angewandte Statistik. Anwendung statistischer Methoden [Applied statistics. Application of statistical methods]. Berlin, Germany: Springer. Schermuly, C. C., & Scholl, W. (2012). The discussion coding system (DCS): A new instrument for analyzing communication processes. Communication Methods and Measures, 6(1), 12–40. doi:10.1080/19312458.2011.651346 Shorey, R. C., Martino, S., Lamb, K. E., LaRowe, S. D., & Santa Ana, E. J. (2015). Change talk and relatedness in group motivational interviewing: A pilot study. Journal of substance abuse treatment, 51, 75-81. doi:10.1016/j.jsat.2014.11.003 Taylor, P. J., & Thomas, S. (2008). Linguistic style matching and negotiation outcome. Negotiation and Conflict Management Research, 1(3), 263–281. doi:10.1111/j.17504716.2008.00016.x Weingart, L. R. (1997). How did they do that. The ways and means of studying group processes. Research in Organisational Behaviour, (19), 189-239. Yoder, P. J., & Symons, F. J. (2010). Observational measurement of behavior. Springer Publishing Company: New York. 27 Table 1 Table 1. Coding scheme for verbal behavior in project team meetings (adapted from Meinecke & Lehmann-Willenbrock, 2015) PROBLEM-FOCUSED (PF+) Problem Describing a problem Connections with problems Defining the objective Solution Describing a solution Problem with a solution Arguing for a solution Organizational knowledge Knowing who Question POSITIVE PROCEDURAL (P+) POSITIVE SOCIO-EMOTIONAL (SE+) POSITIVE ACTION-ORIENTED (AO+) Goal orientation Clarifying Procedural suggestion Procedural question Prioritizing Time management Task distribution Visualization Summarizing Encouraging participation Providing support Listening Reasoned disagreement Giving feedback Humor Separating opinions from facts Expressing feelings Offering praise Expressing positivity Taking responsibility Action planning NEGATIVE PROCEDURAL (P-) NEGATIVE SOCIO-EMOTIONAL (SE-) NEGATIVE ACTION-ORIENTED (AO-) Losing the train of thought (running off topic) Criticizing/backbiting Interrupting Side conversations Self-promotion No interest in change Complaining Seeking someone to blame Denying responsibility Empty talk Ending the discussion early Table 2 Table 2. Agreement and Confusion Matrix, Code-specific Kappas and Intra-Class Correlations for Five Double-coded Meetings Observer 2 Observer 1 PF P+ P- SE+ SE- AO+ AO- Total Kappa ICC PF 2,537 119 11 49 12 75 35 2,838 .79 .97 P+ 114 515 1 36 1 29 4 700 .71 .81 P- 2 0 8 0 0 0 0 10 .42 .36 SE+ 76 19 5 970 3 67 6 1,146 .81 .91 SE- 4 1 0 5 288 1 2 301 .94 .98 AO+ 104 22 3 74 2 619 4 828 .73 .97 AO- 22 2 0 6 3 2 87 122 .66 .93 Total 2,859 678 28 1,140 309 793 138 5,945 Note. PF= Problem-focused communication, P = Procedural communication, SE = Socioemotional communication, AO = Action-oriented communication; + = Positive, - = Negative Table 3 Table 3. Joint Transition Frequencies and Adjusted Residuals for Lag1 Analysis from the First Project Phase Joint Frequencies Target Behavior (Lag1) Given behavior (Lag0) PF P+ PSE+ SEAO+ AOTotals PF P+ P- SE+ SE- AO+ AO- Totals 4,343 895 15 1,109 260 1,075 106 7,803 787 318 3 242 65 235 26 1,676 11 2 1 5 1 5 0 25 1,193 245 1 267 72 247 41 2,066 206 37 3 80 51 62 19 458 1,138 188 1 238 54 253 37 1,909 123 11 1 51 16 26 4 232 7,801 1,696 25 1,992 519 1,903 233 14,169 Adjusted Residuals Target Behavior (Lag1) Given behavior (Lag0) PF P+ PSE+ SEAO+ AO- PF P+ P- SE+ SE- AO+ AO- 1.59* -2.03** 0.5 0.58 -2.32* 1.34* -2.96** -7.1** 9.41** 0.03 0.48 0.5 0.76 -0.32 -1.11 -0.61 4.56** 0.86 0.09 0.96 -0.65 2.66** -0.17 -1.5 -1.61 -0.47 -2.13* 1.32 -4.41** -2.61** 2.48** 2.13* 8.65** 0.07 4.28** 4.3** -3.07** -1.39 -2.15* -2.09* -0.24 1.08 -0.63 -3.42** 0.93 3.5** 2.64** -1 0.1 Note. χ²(36)= 298.25**; * p < .05. ** p < .01; PF= Problem-focused communication, P = Procedural communication, SE = Socio-emotional communication, AO = Action-oriented communication; + = Positive, - = Negative Table 4 Table 4. Joint Transition Frequencies and Adjusted residuals for Lag1 Analysis from the Second Project Phase Joint Frequencies Target Behavior (Lag1) Given behavior (Lag0) PF P+ PSE+ SEAO+ AOTotals PF P+ P- SE+ SE- AO+ AO- Totals 3,514 852 4 1,140 210 941 72 6,733 814 263 3 238 53 182 13 1,566 6 0 2 1 2 1 1 13 1,236 234 2 285 43 233 23 2,056 147 42 1 51 46 31 3 321 955 190 0 222 27 161 17 1572 78 5 0 26 7 18 9 143 6,750 1,586 12 1,963 388 1,567 138 12,404 Adjusted Residuals Target Behavior (Lag1) Given behavior (Lag0) PF P+ PSE+ SEAO+ AO- PF P+ P- SE+ SE- AO+ AO- -5.43** -0.48 -1.46 -2.07* 5.08** 1.29 5.68** -2.09* 0.01 -3.14** 0.16 1.25 5.39** -0.89 -1.32 0.03 -3.35** -0.37 3.68** -0.06 4.91** -0.5 -0.73 0.62 -1.29 -1.14 -0.6 -1.38 17.74* * -0.8 2.54** -0.54 2.26* -2.67** -2.96** -1.94* 0.03 0.03 11.68** -1.63 -0.31 -1.98* -3.44** -3.05** -0.13 0.78 1.22 -0.02 5.94** Note. χ²(36)= 623.34**; * p < .05. ** p < .01; PF= Problem-focused communication, P = Procedural communication, SE = Socio-emotional communication, AO = Action-oriented communication; + = Positive, - = Negative Annotated Bibliography Click here to access/download Integral Supplemental Material (made available to referees) Annotated Bibliography.docx Coded Transcript Click here to access/download Integral Supplemental Material (made available to referees) Coded transcript.docx Sample SDS Data File (Word) Click here to access/download Integral Supplemental Material (made available to referees) Sample SDS Data File.docx Sample SDS Data File (SDS) Click here to access/download Integral Supplemental Material (made available to referees) Sample SDS Data File.sds Software options Click here to access/download Integral Supplemental Material (made available to referees) Software options.docx Sequential Analyses with GSEQ Click here to access/download Integral Supplemental Material (made available to referees) Sequential Analyses with GSEQ.wmv Sequential Analyses with SPSS Click here to access/download Integral Supplemental Material (made available to referees) Sequential Analyses with SPSS.wmv
© Copyright 2026 Paperzz