Scientific study of digital information
Information theory is the mathematical study of the quantification , storage , and communication of information . The field was established and put on a firm footing by Claude Shannon in the 1940s,[ 1] though early contributions were made in the 1920s through the works of Harry Nyquist and Ralph Hartley . It is at the intersection of electronic engineering , mathematics , statistics , computer science , neurobiology , physics , and electrical engineering .[ 2]
A key measure in information theory is entropy . Entropy quantifies the amount of uncertainty involved in the value of a random variable or the outcome of a random process . For example, identifying the outcome of a fair coin flip (which has two equally likely outcomes) provides less information (lower entropy, less uncertainty) than identifying the outcome from a roll of a die (which has six equally likely outcomes). Some other important measures in information theory are mutual information , channel capacity , error exponents , and relative entropy . Important sub-fields of information theory include source coding , algorithmic complexity theory , algorithmic information theory and information-theoretic security .
Applications of fundamental topics of information theory include source coding/data compression (e.g. for ZIP files ), and channel coding/error detection and correction (e.g. for DSL ). Its impact has been crucial to the success of the Voyager missions to deep space, the invention of the compact disc , the feasibility of mobile phones and the development of the Internet.[ 2] The theory has found applications in other areas, including statistical inference ,[ 3] cryptography , neurobiology ,[ 4] perception ,[ 5] linguistics, the evolution[ 6] and function[ 7] of molecular codes (bioinformatics ), thermal physics ,[ 8] molecular dynamics ,[ 9] quantum computing , black holes , information retrieval , intelligence gathering , plagiarism detection ,[ 10] pattern recognition , anomaly detection ,[ 11] imaging system design,[ 12] epistemology ,[ 13] and even art creation.
^ Schneider, Thomas D. (2006). "Claude Shannon: Biologist" . IEEE Engineering in Medicine and Biology Magazine: The Quarterly Magazine of the Engineering in Medicine & Biology Society . 25 (1): 30–33. doi :10.1109/memb.2006.1578661 . ISSN 0739-5175 . PMC 1538977 . PMID 16485389 .
^ a b Baleanu, D.; Balas, Valentina Emilia; Agarwal, Praveen, eds. (2023). Fractional Order Systems and Applications in Engineering . Advanced Studies in Complex Systems. London, United Kingdom: Academic Press. p. 23. ISBN 978-0-323-90953-2 . OCLC 1314337815 .
^ Burnham, K. P. and Anderson D. R. (2002) Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Second Edition (Springer Science, New York) ISBN 978-0-387-95364-9 .
^ F. Rieke; D. Warland; R Ruyter van Steveninck; W Bialek (1997). Spikes: Exploring the Neural Code . The MIT press. ISBN 978-0262681087 .
^ Delgado-Bonal, Alfonso; Martín-Torres, Javier (2016-11-03). "Human vision is determined based on information theory" . Scientific Reports . 6 (1): 36038. Bibcode :2016NatSR...636038D . doi :10.1038/srep36038 . ISSN 2045-2322 . PMC 5093619 . PMID 27808236 .
^ cf; Huelsenbeck, J. P.; Ronquist, F.; Nielsen, R.; Bollback, J. P. (2001). "Bayesian inference of phylogeny and its impact on evolutionary biology". Science . 294 (5550): 2310–2314. Bibcode :2001Sci...294.2310H . doi :10.1126/science.1065889 . PMID 11743192 . S2CID 2138288 .
^ Allikmets, Rando; Wasserman, Wyeth W.; Hutchinson, Amy; Smallwood, Philip; Nathans, Jeremy; Rogan, Peter K. (1998). "Thomas D. Schneider], Michael Dean (1998) Organization of the ABCR gene: analysis of promoter and splice junction sequences" . Gene . 215 (1): 111–122. doi :10.1016/s0378-1119(98)00269-8 . PMID 9666097 .
^ Jaynes, E. T. (1957). "Information Theory and Statistical Mechanics" . Phys. Rev . 106 (4): 620. Bibcode :1957PhRv..106..620J . doi :10.1103/physrev.106.620 . S2CID 17870175 .
^ Talaat, Khaled; Cowen, Benjamin; Anderoglu, Osman (2020-10-05). "Method of information entropy for convergence assessment of molecular dynamics simulations" . Journal of Applied Physics . 128 (13): 135102. Bibcode :2020JAP...128m5102T . doi :10.1063/5.0019078 . OSTI 1691442 . S2CID 225010720 .
^ Bennett, Charles H.; Li, Ming; Ma, Bin (2003). "Chain Letters and Evolutionary Histories" . Scientific American . 288 (6): 76–81. Bibcode :2003SciAm.288f..76B . doi :10.1038/scientificamerican0603-76 . PMID 12764940 . Archived from the original on 2007-10-07. Retrieved 2008-03-11 .
^ David R. Anderson (November 1, 2003). "Some background on why people in the empirical sciences may want to better understand the information-theoretic methods" (PDF) . Archived from the original (PDF) on July 23, 2011. Retrieved 2010-06-23 .
^ Pinkard, Henry; Kabuli, Leyla; Markley, Eric; Chien, Tiffany; Jiao, Jiantao; Waller, Laura (2024). "Universal evaluation and design of imaging systems using information estimation". arXiv :2405.20559 [physics.optics ].
^ Harms, William F. (1998). "The Use of Information Theory in Epistemology" . Philosophy of Science . 65 (3): 472–501. doi :10.1086/392657 . ISSN 0031-8248 . JSTOR 188281 .