Comparing Techniques for Aggregating Interrelated Replications in
Context: Researchers from different groups and institutions are collaborating
towards the construction of groups of interrelated replications. Applying
unsuitable techniques to aggregate interrelated replications' results may
impact the reliability of joint conclusions.
Objectives: Comparing the advantages and disadvantages of the techniques
applied to aggregate interrelated replications' results in Software Engineering
Method: We conducted a literature review to identify the techniques applied
to aggregate interrelated replications' results in SE. We analyze a
prototypical group of interrelated replications in SE with the techniques that
we identified. We check whether the advantages and disadvantages of each
technique -according to mature experimental disciplines such as medicine-
materialize in the SE context.
Results: Narrative synthesis and Aggregation of $p$-values do not take
advantage of all the information contained within the raw-data for providing
joint conclusions. Aggregated Data (AD) meta-analysis provides visual summaries
of results and allows assessing experiment-level moderators. Individual
Participant Data (IPD) meta-analysis allows interpreting results in natural
units and assessing experiment-level and participant-level moderators.
Conclusion: All the information contained within the raw-data should be used
to provide joint conclusions. AD and IPD, when used in tandem, seem suitable to
analyze groups of interrelated replications in SE.