Test oracle

In software testing, a test oracle (or just oracle) is a provider of information that describes correct output based on the input of a test case. Testing with an oracle involves comparing actual results of the system under test (SUT) with the expected results as provided by the oracle.^[1]

The term "test oracle" was first introduced in a paper by William E. Howden.^[2] Additional work on different kinds of oracles was explored by Elaine Weyuker.^[3]

An oracle can operate separately from the SUT; accessed at test runtime, or it can be used before a test is run with expected results encoded into the test logic.^[4]

However, method postconditions are part of the SUT, as automated oracles in design by contract models.^[5]

Determining the correct output for a given input (and a set of program or system states) is known as the oracle problem or test oracle problem,^[6]^: 507 which some consider a relatively hard problem, and involves working with problems related to controllability and observability.^[7]

^ Earl T. Barr et al; The Oracle Problem in Software Testing: A Survey, 2015
^ Howden, W.E. (July 1978). "Theoretical and Empirical Studies of Program Testing". IEEE Transactions on Software Engineering. 4 (4): 293–298. doi:10.1109/TSE.1978.231514.
^ Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980, pp. 44-49
^ Jalote, Pankaj; An Integrated Approach to Software Engineering, Springer/Birkhäuser, 2005, ISBN 0-387-20881-X
^ Meyer, Bertrand; Fiva, Arno; Ciupa, Ilinca; Leitner, Andreas; Wei, Yi; Stapf, Emmanuel (September 2009). "Programs That Test Themselves". Computer. 42 (9): 46–55. doi:10.1109/MC.2009.296.
^ Cite error: The named reference Oracle survey was invoked but never defined (see the help page).
^ Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing, 2nd edition", Cambridge University Press, 2016, ISBN 978-1107172012

[1] Earl T. Barr et al; The Oracle Problem in Software Testing: A Survey, 2015

[2] Howden, W.E. (July 1978). "Theoretical and Empirical Studies of Program Testing". IEEE Transactions on Software Engineering. 4 (4): 293–298. doi:10.1109/TSE.1978.231514.

[3] Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980, pp. 44-49

[038720881X-4] Jalote, Pankaj; An Integrated Approach to Software Engineering, Springer/Birkhäuser, 2005, ISBN 0-387-20881-X

[5] Meyer, Bertrand; Fiva, Arno; Ciupa, Ilinca; Leitner, Andreas; Wei, Yi; Stapf, Emmanuel (September 2009). "Programs That Test Themselves". Computer. 42 (9): 46–55. doi:10.1109/MC.2009.296.

[Oracle_survey-6] Cite error: The named reference Oracle survey was invoked but never defined (see the help page).

[ammann-intro-7] Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing, 2nd edition", Cambridge University Press, 2016, ISBN 978-1107172012

[1]

[2]

[3]

[4]

[5]

[6]

[7]