Comparing bivariate and multivariate approaches to testing individual-level interaction effects in meta-analyses: The case of the integration hypothesis

Introduction

Many theories and hypotheses in psychology assume an interaction, that is, a relationship between two variables that changes at different levels of a third variable (Cox, 1984). In primary studies with continuous variables, testing such relationships requires a multivariate approach, typically a multiple regression including the main effects of the predictor and the moderator and a multiplicative interaction term (Aguinis & Gottfredson, 2010; Baron & Kenny, 1986). This approach, however, is difficult to translate to meta-analysis, as it requires individual-level data while meta-analysis usually deals with data aggregated at the study level. Only recently, advances in meta-analytical structural equations modelling (MASEM; Cheung, 2015a; Hagger et al., 2022) led to the introduction of multivariate methods that allow for accurately meta-analysing individual-level interactions. Prior to that, in the absence of better alternatives, attempts to meta-analytically test hypotheses that involve individual-level interactions, such as the integration hypothesis (Berry, 1997; Berry et al., 2006), relied on non-validated bivariate methods operationalising the interaction as a single variable (e.g., a score resulting from adding or multiplying two predictor scores) and computing bivariate correlations of this single variable with an outcome (e.g., Abu-Rayya et al., 2023). Such approaches produced results of unknown accuracy and might have led to misguided conclusions.

In this paper, we aim to re-evaluate the accuracy of results obtained from bivariate approximations of individual-level interactions by comparing them to results from state-of-the-art multivariate tests (Hagger et al., 2022). We do so on the example of the integration hypothesis (Berry, 1997; Berry et al., 2006), arguably one of the most prominent interaction-based hypotheses in psychology, that has been repeatedly meta-analysed using bivariate approximations of interactions (Abu-Rayya & Sam, 2017; Abu-Rayya et al., 2023; Bierwiaczonek & Kunst, 2021). For reasons that we will discuss later, this paper is not intended as a test of the integration hypothesis. Instead, the question we are concerned with is whether past meta-analyses of the integration hypothesis (and other interaction-based hypotheses) using bivariate approximations can really be considered valid evaluations of this hypothesis.

Past Meta-Analyses of the Integration Hypothesis

The integration hypothesis (Berry, 1997; Berry et al., 2006) suggests that migrants and ethnic minority group members must navigate the extent of their engagement with both the mainstream culture and their heritage culture. Optimal adaptation outcomes, such as psychological health and socio-cultural competence, are thought to be associated with engaging with both the heritage and mainstream cultures (the integration strategy), rather than solely the heritage culture (separation strategy), the mainstream culture (assimilation strategy), or neither (marginalisation strategy).

Thus, integration is best operationalised as an interaction between mainstream-culture orientation and heritage-culture orientation where the strongest positive association with adaptation is expected at high levels of both (Rudmin, 2006; Ward & Kus, 2012). Indeed, many primary acculturation studies measure mainstream and heritage orientation separately (Celenk & van de Vijver, 2014; Safa & Umaña-Taylor, 2021), allowing for an interaction test (Baron & Kenny, 1986). Yet, the interactional basis of the integration hypothesis has not been addressed directly in any of the existing meta-analyses of the link between integration and adaptation outcomes. Most meta-analyses[1] so far operationalised integration by combining the two cultural orientations into one integration score using one of four approaches: midpoint split, summative scores, multiplicative scores, or Euclidean Distance (ED; Abu-Rayya & Sam, 2017; Abu-Rayya et al., 2023; Bierwiaczonek & Kunst, 2021). This integration score was then used to calculate bivariate correlations with adaptation outcomes, which meta-analysts pooled to obtain the average effects.

However, a correct test of interaction effects requires controlling for main effects of the predictor and the moderator (Baron & Kenny, 1986), which in meta-analysis is, to date, only possible using meta-analytical structural equations modelling (MASEM; Cheung, 2015a, Hagger et al., 2022). None of the bivariate approaches listed above allows for it, hence they cannot be considered accurate tests of interaction between cultural orientations that the integration hypothesis calls for. Additionally, each method carries its own methodological drawbacks that we describe below. By consequence, although meta-analyses based on these approaches claim to test the integration hypothesis, it is unknown how much they really tell us about the interaction between the mainstream- and the heritage-culture orientation.

Drawbacks of Bivariate Approaches

In the Midpoint Split Approach (Figure 1a; e.g., Bierwiaczonek & Kunst, 2021), researchers dichotomise scores for each cultural dimension into low and high categories, creating a binary integration score where high scores in both dimensions are coded as 1, and others as 0. Dichotomisation may be performed using median, mean, or theoretical midpoint splits, with the theoretical midpoint (the middle of the scale, such as 3 on a 1-5 scale) being preferable (Arends-Toth & van de Vijver, 2006a). While the midpoint split approach preserves the core notion of interaction that integration requires high engagement with both cultures, it has low sensitivity and has thus been criticised for losing information on individual variation (Abu-Rayya et al., 2023).

With the Summative Approach (Figure 1b), scores on each cultural orientation are summed to create an integration score, with higher scores indicating greater integration (e.g., Abu-Rayya & Sam, 2017; Smokowski et al., 2010). Problematically, however, summation goes against the definition of interaction, which states that the effects are not simply additive (Cox, 1984). Indeed, the summative approach assumes independent influences of each cultural orientation on adaptation (West et al., 2017), meaning that a higher score can be driven by either one orientation rather than by both (Rudmin, 2006). For example, one person scores above the scale midpoint (3.5 on a 5-point Likert scale) on both cultural orientations, resulting in a summative score of 7, while another scores above the midpoint (5) on mainstream culture orientation and below the midpoint (2) on heritage culture orientation, resulting in the same summative score of 7. Even though the former fits the definition of integration and the latter that of assimilation, the summative approach gives both individuals the exact same score. Consequently, a meta-analytical result based on the summative approach might represent the association between either of the two dimensions and adaptation.

In the Multiplicative Approach (Figure 1c),an integration score is derived as the product of mainstream and heritage orientation scores (e.g., Abu-Rayya & Sam, 2017). This product is then entered into a bivariate correlation with an adaptation outcome; differently than in the multivariate approach with MASEM (Cheung, 2015a, Hagger et al., 2022), main effects of both cultural orientations are not controlled for. As the comparison of Figure 1b and 1c shows, the multiplicative approach displays the same problems as the summative approach, namely, answers above the scale midpoint on both orientations (consistent with integration) can result in the same or lower score as answers below the midpoint on one orientation and above the midpoint on the other (consistent with assimilation or separation). Thus, the resulting effect size can be inadvertently driven by only one dimension rather than both (Rudmin, 2006).

The ED Approach (Figure 1d; e.g., Abu-Rayya et al., 2023; Arends-Toth & van de Vijver, 2006b) places scores from two cultural orientations on a two-dimensional matrix and calculates the participant’s distance from the ideal integration score (i.e., the highest score on both axes). It is calculated as:

where P1, P2 represent the extreme score, and Q1, Q2 represent a person’s score on the corresponding dimension. Again, this approach fails to fully translate the idea of integration as high scores on both cultural orientations. To illustrate it, consider the answers 3.3 on both orientations (Person A) versus answers 2.7 on heritage orientation and 5 on mainstream orientation (Person B, see Figure 1). The resulting ED scores are, respectively, 2.4 and 2.3, meaning that Person B is closer to the ideal integration score, thus more integrated, whereas conceptually the opposite should be the case. Further, ED scores are, by definition, nonlinear, thus not necessarily compatible with the linear integration hypothesis.  

Despite these variations in operationalising integration, previous meta-analyses found relatively consistent small correlations between integration and adaptation (Kunst, 2021). For example, Bierwiaczonek and Kunst (2021) meta-analysed 19 longitudinal studies using the midpoint split approach and found cross-sectional correlations ranging from .02 to .11. Abu-Rayya and Sam (2017) found positive correlations ranging from .07 to .21 with the multiplicative approach applied to the ICSEY dataset. Subsequently, Abu-Rayya et al. (2023) found positive correlations ranging from .06 to .17 using the ED approach on the same dataset, and from .07 to .23 using the summative approach (referred to as Cultural Involvement, or CI). The similarity of these results might not only be due to using the same dataset (except for Bierwiaczonek & Kunst, 2021), but also to similar biases arising from the above-described problems with bivariate approaches to integration.

Figure 1

Illustration of the four bivariate approaches: Midpoint split (a), summative (b), multiplicative (c), and Euclidean Distance (d), for two hypothetical participants: Person A (scoring 3.3 on both cultural orientations) and Person B (scoring 5 on mainstream orientation and 2.7 on heritage orientation).

The Current Studies

In sum, none of the approaches used in meta-analyses of integration effects so far is without problems: The midpoint split approach lacks sensitivity, whereas the remaining three approaches risk confusing the effects of integration with the effects of either of the two cultural orientations. Thus, such approaches can be considered proxies of an interaction test at best. It remains unknown to which extent using these methods has biased the results of previous meta-analyses of integration and, by consequence, whether these results can be considered trustworthy.

Here, we address these questions by assessing how much the results from the four bivariate approaches described above diverge from the results of a state-of-the-art meta-analytical test of interaction effects, conducted by fitting a full moderation model including both the main effects of each orientation and their interaction term (Hagger et al., 2022; Jak & Cheung, 2020). To do so, we use two existing meta-analytical datasets: a smaller dataset that previously showed relatively small effects (Bierwiaczonek & Kunst, 2021, Study 1), and a larger dataset that previously showed slightly larger effects (ICSEY, Berry et al., 2006, Study 2). Both datasets were obtained directly from the (co-)authors of the original studies with permission to reuse them.

Below we report results obtained by combining psychological and socio-cultural adaptation into one outcome, an approach also employed by other meta-analysts in the field (e.g., Bierwiaczonek & Kunst, 2021; Nguyen & Benet-Martínez, 2013). Separate analyses for psychological adaptation and socio-cultural adaptation are available in the supplementary materials (Table S3, S4, Figure S1, S2 for Study 1; Table S5, S6, Figure S3, S4 for Study 2). Supplementary materials, data and analysis code are available online via OSF,  https://osf.io/ytkez/. The aggregated meta-analytical datasets used in Study 1 and Study 2 have been published with permissions from the authors of the original studies.

Study 1

Methods 

In Study 1, we re-analysed Bierwiaczonek and Kunst’s (2021) data consisting of 19 primary studies examining the relationship between integration and adaptation outcomes. Please refer to Bierwiaczonek and Kunst (2021) for full details of the search procedures, coding, and sample descriptions. Since calculating four different operationalisations of integration required full data including both mainstream-culture orientation and heritage-culture orientation, as well as adaptation outcomes, we used a subset of 110 effect sizes from nine studies for which such data were available. The detailed description of this subset is available in the supplementary materials (Table S1). 

All included studies were longitudinal, with each primary study comprising 2-4 waves of data collection. For the current study, we included all effect sizes within different waves (e.g., acculturation at T1 and outcomes at T1) but excluded lagged effects (e.g., acculturation at T1 and outcomes at T2). Within-study dependency was accounted for in the analysis by using a multilevel approach.

Data Preparation

We transformed all measures to a 1-5 Likert scale for the sake of comparability. This transformation does not affect the relationships between cultural orientations and outcomes but facilitates the characterisation of the dataset (e.g., means, standard deviations). Where necessary, scores were reversed to ensure that higher scores indicated a higher level of the construct of interest. For example, since we were interested in adaptation rather than the lack thereof, any negative indicators of adaptation such as depression were reversed so that higher scores correspond with better adaptation.

Analytical Procedures

Bivariate approaches: Integration scores. Following previous meta-analyses in the field, we computed, in each primary dataset, four variables corresponding with the four bivariate approaches to operationalising integration (i.e., four integration scores):

• summative: sum of mainstream-culture and heritage-culture orientation values.
• multiplicative: product of mainstream-culture and heritage-culture orientation values.
• Euclidean Distance (ED): Euclidean Distance from the maximum value of 5, multiplied by -1 for higher scores to indicate better integration.
• midpoint split: a binary indicator (1 if both orientations ≥3, 0 otherwise).

We then calculated the correlation with adaptation outcomes and the corresponding variance for each operationalisation of integration. For summative, multiplicative, and ED operationalisations, we used Pearson’s correlation. For the midpoint split operationalisation, we calculated biserial correlation with adaptation and its corresponding variance using Soper’s method (Soper, 1914; see also Jacobs & Viechtbauer, 2017), as biserial correlation is statistically equivalent and comparable to Pearson’s correlation.  

For each operationalisation (summative, multiplicative, ED, midpoint split), we conducted three-level meta-analyses on the correlation with adaptation using the metafor 4.6 R package (Viechtbauer, 2010). In these analyses, Level 1 corresponded with variation between participants, Level 2 with the variation of effect within each independent sample, and Level 3 with the variation of effects between independent participant samples. We then compared the resulting effect sizes (variance explained R²) from each operationalisation to each other and to the full interaction model.

Multivariate approach: Interaction test. Prior to analyses, we calculated an interaction term as the product of standardised (z-score) mainstream-culture and heritage-culture orientation scores in each primary dataset. We then calculated Pearson’s bivariate correlations between each pair of variables, namely: mainstream-culture orientation and adaptation, heritage-culture orientation and adaptation, mainstream-culture and heritage-culture orientation, mainstream-culture orientation and interaction term, heritage-culture orientation and interaction term, interaction term and adaptation.

To test for interaction between mainstream-culture orientation and heritage-culture orientation, we used a two-stage MASEM. In Stage 1, we used the three-level meta-analytical approach (similar as in bivariate analyses) to obtain a total of six pooled correlations (i.e., mainstream-culture orientation and adaptation outcome, heritage-culture orientation and adaptation outcome, mainstream-culture and heritage-culture orientation, mainstream-culture orientation and interaction term, heritage-culture orientation and interaction term, and interaction term and adaptation outcome). These effects formed a pooled correlation matrix. These analyses were conducted using the metafor 4.6 package for R (Viechtbauer, 2010). In Stage 2, two structural equation models were fitted to the pooled correlation matrix using the metaSEM 1.3.1 package for R (Cheung, 2015b). First, we fitted the model including only main effects of the two acculturation orientations (Model 1.1):

Adaptation ~ Mainstream-culture orientation + Heritage-culture orientation

Then, we fitted a model including the main effects and the interaction term (Model 1.2):

Adaptation ~ Mainstream-culture orientation + Heritage-culture orientation + Mainstream-culture orientation * Heritage-culture orientation

Note that, since these models were fully saturated (i.e., all parameters were calculated), model fit indices were not applicable.

Finally, to assess whether adding the interaction term to the model improved the prediction, we compared variance explained by the full interaction model to that of the main effect model that excluded the interaction term (ΔR²). We also calculated the f² statistic, which represents the ratio of systematic variance accounted for by the interaction effect relative to unexplained variance (Aiken et al., 1991). According to a review by Aguinis et al. (2005), which examined 30 years of research using moderated multiple regression to detect interaction effects in psychological studies, the mean effect size for f² was 0.009. We compared the f² statistic from our study to this mean effect size to evaluate its relative magnitude.

Results and Discussion

Bivariate Approaches

Detailed results for each bivariate approach, including the average effect sizes as well as their precision and heterogeneity, are provided in Table 1. Meta-regression results showed no significant difference between the effect of integration operationalised using the ED approach (r = .15, p < .001) and the effects using the summative (B = .01, p = .453, 95% CI [-.011, .024]) and multiplicative approaches (B = .01, p = .480, 95% CI [-.011, .024]). However, the effect obtained using the ED approach was significantly larger than the effect measured using the midpoint split approach (B = -.05, p < .001, 95% CI [-.070, -.027]).

Table 1

Average Associations Between Adaptation and Four Different Operationalisations of Integration for Bierwiaczonek and Kunst’s (2021) Data.

Multivariate Approach

In Stage 1 of MASEM, pooled bivariate correlations showed that both mainstream-culture orientation and heritage-culture orientation were significantly related to adaptation (r_mainstream = .18, R²_mainstream = 3.19%, p_mainstream < .001, 95% CI_mainstream [.123, .233]; r_heritage = .06, R²_heritage = 0.38%, p_heritage = .024, 95% CI_heritage [.008, .115]).

In Stage 2 of MASEM, when both cultural orientations were included without the interaction term (Figure 2, Model 1.1), only mainstream-culture orientation was significantly positively related to adaptation while heritage-culture orientation was not. The total variance explained by the model was 3.31%. When the interaction term was added to the model (Figure 2, Model 1.2), there was no significant interaction between mainstream- and heritage-culture orientation on adaptation. The relationship between both cultural orientations and adaptation remained the same as in Model 1.1. Adding the interaction only explained an additional 0.01% of the variance in adaptation outcomes,with an f² = .0001, thus not reaching the average interaction effect size (Aguinis et al., 2005).

To sum up, results from the bivariate approaches indicate that, when interaction is not explicitly considered (as in summative, multiplicative, or ED approaches), the effect sizes tend to be similar and oscillate around .16 in the current dataset. The midpoint split approach, which adheres to the core idea of interaction by requiring participants to engage in both cultural dimensions to be considered integrated, appears to yield a significantly smaller effect. However, when employing a multivariate approach and conducting a full interaction test, the effect of interaction between mainstream- and heritage-culture orientations on adaptation is even smaller and non-significant, with adaptation being driven by mainstream-culture orientation.

Figure 2

Two-Stage MASEM Results on How Mainstream- and Heritage-Culture Orientations Interact to Predict Adaptation in Bierwiaczonek and Kunst’s (2021) Data.

Study 2

Method 

This study uses a subset of the ICSEY dataset (Berry et al., 2006), considered the most comprehensive comparative study of acculturation to date (Abu-Rayya et al., 2023). Similar to Study 1, we only included data from international migrants, requiring both mainstream-culture and heritage-culture orientation measures, along with adaptation outcomes. Details of these measures, as described by Abu-Rayya and Sam (2017), can be found in the supplementary materials (Table S2). In total, we used 240 effect sizes nested within 48 samples across 13 countries. Within-study dependency was accounted for in the analysis by using a multilevel approach. Data preparation and analytical procedures were identical as in Study 1.

Results and Discussion

Bivariate Approaches

Detailed average effect sizes for each of the four bivariate approaches are provided in Table 2. Again, meta-regression results showed no significant difference between the effect of integration operationalised using the ED approach (r = .12, p < .001) and the effects using the summative (B = .01, p = .271, 95% CI [-.007, .025]) and multiplicative approaches (B = .00, p = .701, 95% CI [-.013, .019]). However, midpoint split approach yielded a significantly smaller effect (B = -.02, p = .022, 95% CI [-.041, -.003]).

Table 2

Average Associations Between Adaptation and Four Different Operationalisations of Integration for the ICSEY dataset (Berry et al., 2006).

Multivariate Approach

Similar to Study 1, in Stage 1 of MASEM, bivariate correlations indicated that both mainstream- and heritage-culture orientations were significantly related to adaptation (r_mainstream = .11, R²_mainstream = 1.14%, p_mainstream < .001, 95% CI_mainstream [.083, .131]; r_heritage = .07, R²_heritage = 0.50%, p_heritage < .001, 95% CI_heritage [.045, .097]). In Stage 2, before adding the interaction to the model (Figure 3, Model 2.1), both orientations remained significant predictors of adaptation, collectively explaining 1.68% of the variance of adaptation outcomes. Adding the interaction term to the model (Figure 3, Model 2.2) slightly (by 0.08%, f² = .0008) yet significantly increased variance explained by the model. To further explore the significant interaction, we conducted simple slope analysis (Figure 4). Contrary to the prediction, this analysis showed that mainstream-culture orientation had a stronger association with adaptation at the lowest level of heritage-culture orientation, and vice-versa, heritage-culture orientation had a stronger association with adaptation at the lowest level of mainstream-culture orientation.

Figure 3

Two-Stage MASEM Results on How Mainstream- and Heritage-Culture Orientations Interact to Predict Adaptation in the ICSEY dataset (Berry et al., 2006).

In sum, regarding the bivariate approaches, the results were consistent with Study 1 in that the effect sizes were small across the four operationalisations of integration, and similar when interaction was not explicitly considered (i.e., the summative, multiplicative and ED approaches). Again, the midpoint split approach yielded a smaller effect size. Regarding the multivariate approach, this time we found a significant interaction between mainstream- and heritage-culture orientations. This finding may be due to the larger sample size of the ICSEY dataset than in Bierwiaczonek and Kunst’s (2021) data used in Study 1, allowing for the detection of relatively small effects. However, there are two important points to consider: first, incremental variance in adaptation outcomes explained by interaction was minimal, below 0.1%. Second, simple slope analysis suggested that mainstream-culture orientation only contributes to adaptation when heritage-culture orientation is low but not when it is high, and vice-versa. If that is the case, strongly endorsing both orientations (integration) presents no advantage over strongly endorsing one orientation only.

Figure 4

Simple slope analysis showing the association between mainstream orientation and adaptation at different levels of heritage orientation (3a) and between heritage orientation and adaptation at different levels of mainstream orientation (3b) in the ICSEY dataset (Berry et al., 2006).

We additionally validated these findings in a simulation study (total k = 30,000, total N = 8,981,481; see Supplementary Materials). This simulation confirmed that the difference in results between Study 1 and 2 was only due to statistical power rather than any other meaningful differences in study parameters, and suggested that significant interaction effects, when detected, are negligeable in size and primarily driven by mainstream-culture orientation.

General Discussion

In the absence of methods to meta-analytically test hypotheses based on individual-level interactions, it is understandable that many meta-analysts recurred to approximative operationalisations of the interaction of interest. The set of studies reported here clearly shows that using such approximations can be highly problematic.Specifically, results obtained from bivariate approaches used in meta-analyses of the integration hypothesis appear to be considerably inflated when compared to a correct multivariate test. It must be noted, however, that this paper is not intended as a test of the integration hypothesis and should not be read as such. The main reason has to do with the limitations of the datasets we used here: Study 1 includes only 122 effects; and the dataset in Study 2 is derived from a single project, making it more adequate for a multilevel analysis than a meta-analysis. Thus, neither of the studies is representative for the entire acculturation field. Instead, our goal here was to alert the acculturation field to the fact that previous meta-analyses used flawed methodological approaches that might have produced inflated results, leading to misguided conclusions.

Across the two studies presented here, the results were consistent, even though they used datasets that had previously yielded opposite results: in the case of the ICSEY data, supporting the integration hypothesis (Abu-Rayya & Sam, 2017; Abu-Rayya et al., 2023), and in the case of Bierwiaczonek and Kunst’s (2021) data, failing to support it. Overall, bivariate approaches yield similar average effect sizes, ranging from .10 to .15, regardless of the method employed. Even with the midpoint split approach, the effects, although consistently smaller, remained comparable. The bivariate analyses thus appeared to yield convergent, convincing results. However, when interaction was correctly tested using state-of-the-art methods (Cheung, 2015a, Hagger et al., 2022), the interaction effect itself was minimal and the divergence between both approaches became evident: Variance explained decreased from around 2% (bivariate approaches) to less than 0.10% (multivariate approach).  Further, the pattern of results showed that using bivariate approaches could lead to mistaking variance explained by main effects of one or both predictors for variance explained by the interaction between them. Here, specifically, variance in adaptation was explained mostly by mainstream-culture orientation, while heritage culture orientation seemed to contribute very little, a pattern found previously in an analysis of the heterogeneity of integration effects (Bierwiaczonek et al., 2023). Thus, bivariate approaches do not only consistently inflate the effect sizes but can also obscure the specific role of individual predictors.[2]

Methodological Takeaways

Considering these results, we call for abandoning bivariate approximations and using a correct multivariate approach to testing individual-level interactions in meta-analysis. We acknowledge, however, that this approach has one critical limitation: It requires full participant data. Note that this is not an added requirement as compared to the problematic bivariate approaches. Individual scores are also needed to calculate an approximative interaction score based on a midpoint split, summative, multiplicative, or ED approach, and applying the correct multivariate approach does not pose additional challenges in this regard. Nevertheless, accessing full participant data may be difficult for meta-analysts who usually only have access to aggregated information from research reports (e.g., correlation matrices, means, standard deviations). Despite repeated calls for open science practices, data sharing, though more common now than before, is not yet the standard practice (see Colavizza et al., 2020 for a general overview, and Vu et al., in press, for cross-cultural adaptation research). Alternative methods to directly analyse interactions exist (e.g., meta-analysing interaction terms reported in primary studies; see Aguinis et al., 2011; Murphy & Russell, 2017), but they may also be difficult to implement in practice due to missing information in primary papers. However, our simulation study (see Supplementary Materials) suggests that the multivariate approach to meta-analysing interactions produces relatively reliable results even with a relatively low number of studies (k = 20), and it is clearly preferable over bivariate approaches even under conditions of lower power. Thus, meta-analysts should prioritise the multivariate approach even if it means that they can include fewer studies.

Importantly, while the current set of studies focused on the integration hypothesis, interaction-based hypotheses are common in psychology, and bivariate methods were used to examine interactions in several meta-analyses on other topics (e.g., the interaction between cognitive ability and motivation on performance, Van Iddekinge et al., 2018; or the interaction between adaptive and maladaptive perfectionism, Stoeber et al., 2020). Thus, the implications of our studies go far beyond the field of acculturation. Bivariate approximations of interactions may lead to inflated results and erroneous conclusions, further translating into incorrect theory, misguided policy and ineffective interventions in real-world settings. Therefore, the acculturation field and other fields in which meta-analyses have used similar methodological workarounds would be well-advised to critically re-evaluate the results obtained this way.

Conclusion

Analysing individual-level interaction effects in meta-analyses requires a multivariate approach, where main effects of predictors are tested together with the interaction term (Hagger et al., 2022). However, because these methods are relatively new, many past meta-analyses have relied on bivariate approximations of interactions, likely resulting in biased results and erroneous conclusions. For instance, previous meta-analyses of the integration hypothesis found small effects that they attributed to the interaction between mainstream and heritage culture orientations (i.e., integration) in predicting adaptation outcomes. However, when tested with a state-of-the-art multivariate method, this interaction turned out to be negligeable in size and primarily driven by mainstream culture orientation.

This discrepancy in the case of the integration hypothesis highlights a broader issue: Bivariate approaches to testing individual-level interaction in meta-analysis produced inflated effects and obscured the roles of individual predictors. Such methodological workarounds should be avoided, and past meta-analyses using these approaches should be re-evaluated.

Endnotes

Conflicts of Interest

The authors declare no competing interests.

Author Contributions

D.V. and K.B. designed the study. D.V. conducted the data analysis, wrote up the results, and drafted the initial manuscript. K.B. provided critical feedback at all stages of the research process and revised the final manuscript. Both authors reviewed and approved the final version of the paper for submission.

Data Availability Statement

Supplementary materials and analysis codes can be accessed online: https://osf.io/ytkez/

Acknowledgements

This research was supported by a grant from the Research Council of Norway, Norway, to the second author for the project ACCA (Antecedents of Cross-Cultural Adaptation, Project No. 325260).

The authors thank Dr. David Lackland Sam for sharing the ICSEY dataset and Dr. Mike W.-L. Cheung for his methodological insight.

References

Abu‐Rayya, H. M., Berry, J. W., Sam, D. L., & Grigoryev, D. (2023). Evaluating the integration hypothesis: A meta‐analysis of the ICSEY project data using two new methods. British Journal of Psychology, 114(4), 819-837. https://doi.org/10.1111/bjop.12656

Abu-Rayya, H. M., & Sam, D. L. (2017). Is integration the best way to acculturate? A reexamination of the bicultural-adaptation relationship in the “ICSEY dataset” using the bilineal method. Journal of cross-cultural psychology, 48(3), 287-293. https://doi.org/10.1177/0022022116685846

Aguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. Journal of applied psychology, 90(1), 94-107. https://doi.org/10.1037/0021-9010.90.1.94

Aguinis, H., & Gottfredson, R. K. (2010). Best‐practice recommendations for estimating interaction effects using moderated multiple regression. Journal of organizational behavior, 31(6), 776-786. https://doi.org/10.1002/job.686

Aguinis, H., Gottfredson, R. K., & Wright, T. A. (2011). Best‐practice recommendations for estimating interaction effects using meta‐analysis. Journal of Organizational Behavior, 32(8), 1033-1043. https://doi.org/10.1002/job.719

Aiken, L. S., West, S. G., & Reno, R. R. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage

Arends-Toth, J., & van de Vijver, F. J. R. (2006a). Assessment of psychological acculturation. In D. L. Sam & J. W. Berry (Eds.), The Cambridge handbook of acculturation psychology (pp. 143–160). Cambridge University Press.

Arends-Toth, J., & van de Vijver, F. J. R. (2006b). Issues in the conceptualization and assessment of acculturation. In M. H. Bornstein & L. R. Cote (Eds.), Acculturation and parent-child relationships: Measurement and development (pp. 33–62). Lawrence Erlbaum Associates Publishers.

Arends‐Tóth, J., & van de Vijver, F. J. (2007). Acculturation attitudes: A comparison of measurement methods. Journal of Applied Social Psychology, 37(7), 1462-1488. https://doi.org/10.1111/j.1559-1816.2007.00222.x

Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of personality and social psychology, 51(6), 1173-1182. https://doi.org/10.1037//0022-3514.51.6.1173

Berry, J. W. (1997). Immigration, acculturation, and adaptation. Applied psychology, 46(1), 5-34. https://doi.org/10.1111/j.1464-0597.1997.tb01087.x

Berry, J. W., Lepshokova, Z., Collaboration, M., Grigoryev, D., Annis, R. C., Au, A. K., Bano, S., Boehnke, K., Buholzer, A., & Brylka, A. (2022). How shall we all live together? Meta‐analytical review of the mutual intercultural relations in plural societies project. Applied psychology, 71(3), 1014-1041. https://doi.org/10.1111/apps.12332

Berry, J. W., Phinney, J. S., Sam, D. L., & Vedder, P. (2006). Immigrant youth in cultural transition. Mahwah, NJ: Erlbaum.

Bierwiaczonek, K., Cheung, M. W.-L., & Kunst, J. R. (2023). Revisiting the integration hypothesis again: High heterogeneity complicates the interpretation of cross-sectional evidence. International Journal of Intercultural Relations, 93, 101780. https://doi.org/10.1016/j.ijintrel.2023.101780

Bierwiaczonek, K., & Kunst, J. R. (2021). Revisiting the integration hypothesis: Correlational and longitudinal meta-analyses demonstrate the limited role of acculturation for cross-cultural adaptation. Psychological Science, 32(9), 1476-1493. https://doi.org/10.1177/09567976211006432

Celenk, O., & van de Vijver, F. J. (2014). Assessment of psychological acculturation and multiculturalism: An overview of measures in the public domain. The Oxford handbook of multicultural identity, 205-226. https://doi.org/10.1093/oxfordhb/9780199796694.013.001

Cheung, M. W.-L. (2015a). Meta-analysis: A structural equation modeling approach. John Wiley & Sons.

Cheung, M. W.-L. (2015b). metaSEM: An R package for meta-analysis using structural equation modeling. Frontiers in psychology, 5, 1521. https://doi.org/10.3389/fpsyg.2014.01521

Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K., & McGillivray, B. (2020). The citation advantage of linking publications to research data. PloS one, 15(4), e0230416. https://doi.org/10.1371/journal.pone.0230416

Cox, D. R. (1984). Interaction. International Statistical Review/Revue Internationale de Statistique, 1-24. https://doi.org/10.2307/1403235

Hagger, M. S., Cheung, M. W.-L., Ajzen, I., & Hamilton, K. (2022). Perceived behavioral control moderating effects in the theory of planned behavior: A meta-analysis. Health Psychology, 41(2), 155-167. https://doi.org/10.1037/hea0001153

Jacobs, P., & Viechtbauer, W. (2017). Estimation of the biserial correlation and its sampling variance for use in meta‐analysis. Research synthesis methods, 8(2), 161-180. https://doi.org/10.1002/jrsm.1218

Jak, S., & Cheung, M. W. L. (2020). Meta-analytic structural equation modeling with moderating effects on SEM parameters. Psychological methods, 25(4), 430. https://doi.org/10.1037/met0000245

Kunst, J. R. (2021). Are we facing a “causality crisis” in acculturation research? The need for a methodological (r) evolution. International Journal of Intercultural Relations, 85, A4-A8. https://doi.org/10.1016/j.ijintrel.2021.08.003

Murphy, K. R., & Russell, C. J. (2017). Mend it or end it: Redirecting the search for interactions in the organizational sciences. Organizational Research Methods, 20(4), 549-573. https://doi.org/10.1177/1094428115625322

Nguyen, A.-M. D., & Benet-Martínez, V. (2013). Biculturalism and adjustment: A meta-analysis. Journal of cross-cultural psychology, 44(1), 122-159. https://doi.org/10.1177/0022022111435097

Rudmin, F. W. (2006). Debate in science: The case of acculturation. In AnthroGlobe Journal. Retrieved Jan 28, 2025, from http://www.anthroglobe.info/docs/rudminf_acculturation_061204.pdf

Safa, M. D., & Umaña-Taylor, A. J. (2021). Biculturalism and adjustment among US Latinos: A review of four decades of empirical findings. Advances in child development and behavior, 61, 73-127. https://doi.org/10.1016/bs.acdb.2021.05.002

Smokowski, P. R., Rose, R. A., & Bacallao, M. (2010). Influence of risk factors and cultural assets on Latino adolescents’ trajectories of self-esteem and internalizing symptoms. Child Psychiatry & Human Development, 41(2), 133-155. https://doi.org/10.1007/s10578-009-0157-6

Soper, H. (1914). On the probable error of the bi-serial expression for the correlation coefficient. Biometrika, 10(2/3), 384-390. https://doi.org/10.2307/2331789

Stoeber, J., Madigan, D. J., & Gonidis, L. (2020). Perfectionism is adaptive and maladaptive, but what’s the combined effect?. Personality and Individual Differences, 161, 109846. https://doi.org/10.1016/j.paid.2020.109846

Van Iddekinge, C. H., Aguinis, H., Mackey, J. D., & DeOrtentiis, P. S. (2018). A meta-analysis of the interactive, additive, and relative effects of cognitive ability and motivation on performance. Journal of Management, 44(1), 249-279. https://doi.org/10.1177/0149206317702220

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of statistical software, 36(3), 1-48. https://doi.org/10.18637/jss.v036.i03

Vu, D.-H., Kunst, J. R., Tong, R., & Bierwiaczonek, K. (in press). Methodological challenges in cross-cultural adaptation research:  Insights from a large-scale meta-analysis. In L.A. Arasaratnam-Smith (Eds.), Handbook of Communication & Culture. Edward Elgar Publishing.

Ward, C., & Kus, L. (2012). Back to and beyond Berry’s basics: The conceptualization, operationalization and classification of acculturation. International Journal of Intercultural Relations, 36(4), 472-485. https://doi.org/10.1016/j.ijintrel.2012.02.002

West, A. L., Zhang, R., Yampolsky, M., & Sasaki, J. Y. (2017). More than the sum of its parts: A transformative theory of biculturalism. Journal of cross-cultural psychology, 48(7), 963-990. https://doi.org/10.1177/0022022117709533