Tertiary review

In software engineering, a tertiary review is a systematic review of systematic reviews.^[1] It is also referred to as a tertiary study in the software engineering literature. However, Umbrella review is the term more commonly used in medicine.

Kitchenham et al.^[1] suggest that methodologically there is no difference between a systematic review and a tertiary review. However, as the software engineering community has started performing tertiary reviews new concerns unique to tertiary reviews have surfaced. These include the challenge of quality assessment of systematic reviews,^[2] search validation^[3] and the additional risk of double counting.^[4]

^ ^a ^b Kitchenham, B. A. (2016). Evidence-based software engineering and systematic reviews. Boca Raton. ISBN 9781482228656.{{cite book}}: CS1 maint: location missing publisher (link)
^ Usman, Muhammad; Ali, Nauman Bin; Wohlin, Claes (2023). "A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering". E-Informatica Software Engineering Journal. 17: 230105. arXiv:2109.10134. doi:10.37190/e-inf230105. S2CID 237581262.
^ Börstler, Jürgen; bin Ali, Nauman; Unterkalmsteiner, Michael (2022). "How good are my search strings? Reflections on using an existing review as a quasi-gold standard". E-Informatica Software Engineering Journal. 16 (1): 220103. arXiv:2402.11041. doi:10.37190/e-inf220103. S2CID 245255682.
^ Börstler, Jürgen; bin Ali, Nauman; Petersen, Kai (2023-06-01). "Double-counting in software engineering tertiary studies — An overlooked threat to validity". Information and Software Technology. 158: 107174. doi:10.1016/j.infsof.2023.107174. ISSN 0950-5849. S2CID 257173845.

[kitchenham2016-1] Kitchenham, B. A. (2016). Evidence-based software engineering and systematic reviews. Boca Raton. ISBN 9781482228656.{{cite book}}: CS1 maint: location missing publisher (link)

[2] Usman, Muhammad; Ali, Nauman Bin; Wohlin, Claes (2023). "A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering". E-Informatica Software Engineering Journal. 17: 230105. arXiv:2109.10134. doi:10.37190/e-inf230105. S2CID 237581262.

[3] Börstler, Jürgen; bin Ali, Nauman; Unterkalmsteiner, Michael (2022). "How good are my search strings? Reflections on using an existing review as a quasi-gold standard". E-Informatica Software Engineering Journal. 16 (1): 220103. arXiv:2402.11041. doi:10.37190/e-inf220103. S2CID 245255682.

[4] Börstler, Jürgen; bin Ali, Nauman; Petersen, Kai (2023-06-01). "Double-counting in software engineering tertiary studies — An overlooked threat to validity". Information and Software Technology. 158: 107174. doi:10.1016/j.infsof.2023.107174. ISSN 0950-5849. S2CID 257173845.

[1]

[2]

[3]

[4]