Information-Theoretic Measures in AI: A Practical Decision Framework
A practical decision framework for four foundational measures - Entropy, KL divergence/cross-entropy, Mutual Information, and Transfer Entropy is provided, organized around three prescriptive questions for each: what question does the measure answer and in which AI context; which estimator is appropriate for the data t...