Skip to content

[exponential_families, information_projections] Add two lectures on information geometry - #1129

Merged
mmcky merged 4 commits into
mainfrom
nov_penn8
Nov 17, 2026
Merged

mmcky merged 4 commits into
mainfrom
nov_penn8

Conversation

@thomassargent30

Copy link
Copy Markdown
Contributor

Add two lectures on elementary information geometry to the Statistics and Information section, placed after divergence_measures.

Exponential Family and Maximum Entropy (exponential_families) derives the exponential family from maximum entropy, using Jaynes's Brandeis dice problem, and shows that the normal density maximizes entropy for a given mean and variance. It treats natural and expectation parameters as dual coordinates linked by Legendre duality, and maps that duality onto profit and production functions and Hotelling's lemma. It shows that:

  • the conjugate of the log-partition function is negative entropy, and the multipliers of a maximum entropy problem are shadow prices;
  • the logit inclusive value is a log-partition function, with the identity τ log Σ q exp(v/τ) = max_π {π·v − τ D(π‖q)};
  • KL divergence is a Bregman divergence in either coordinate system, and the Fisher information matrix is its curvature.

A simulation compares the finite-sample behavior of maximum likelihood estimators in the two coordinate systems. Exercises cover the gamma and Poisson families, the logit of a Bernoulli mean, and conjugate priors.

Information Projections (information_projections) proves the Pythagorean theorems for e-projections onto linear families and m-projections onto exponential families. It applies them to:

  • maximum likelihood under correct specification and misspecification, including the identity that the likelihood-ratio statistic equals 2n times a KL divergence;
  • mutual information;
  • Sanov's theorem and the conditional limit theorem, with exact computations for a die;
  • the worst-case models of multiplier preferences in robust control;
  • RAS / iterative proportional fitting, with Csiszár's convergence argument, which does not need the biproportional form and so covers tables with zeros;
  • Sinkhorn's algorithm, whose regularized plans converge to the maximum-entropy optimal transport plan rather than to linprog's vertex;
  • Chernoff information.

A note flags a published error: Nielsen (2018, 2020) pairs each Pythagorean identity with the other identity's orthogonality condition. Exercises cover exponential tilting versus empirical likelihood reweighting, constraint preferences, and a hierarchical decomposition of interactions among three variables.

Also add a return link in divergence_measures and 26 bibliography entries.

Both lectures execute without errors. A subset Jupyter Book build reports no warnings for either lecture.

🤖 Generated with Claude Code

…nformation geometry

Add two lectures to the Statistics and Information section, placed
after divergence_measures.

Exponential Family and Maximum Entropy (exponential_families) covers:
- maximum entropy, with Jaynes's die and the normal distribution as the
  maximum entropy density for a given mean and variance;
- natural and expectation parameters, linked by Legendre duality, with
  a dictionary to profit and production functions;
- the conjugate of the log-partition function as negative entropy;
- the logit inclusive value as a log-partition function;
- KL divergence as a Bregman divergence, and Fisher information;
- finite-sample behavior of maximum likelihood estimators in the two
  coordinate systems.

Information Projections (information_projections) covers:
- e- and m-projections and their Pythagorean theorems;
- maximum likelihood as an m-projection, under correct specification
  and misspecification;
- mutual information;
- Sanov's theorem and the conditional limit theorem;
- worst-case models of robust control;
- RAS/iterative proportional fitting, with Csiszar's convergence proof;
- Sinkhorn's algorithm for entropy-regularized optimal transport;
- Chernoff information.

Also add the bibliography entries and a link from divergence_measures.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved support, interiority, and moment assumptions leave several mathematical claims and the IPF implementation invalid for cases permitted by the text.

5 open issues
What changed in this PR

Adds two connected information-geometry lectures to the Statistics and Information section, extending the existing material on divergence measures with theory, economic applications, and executable Python examples.

Changes:

  • Introduces exponential families, maximum entropy, dual coordinates, and estimation.
  • Develops information projections and applications to robust control, matrix balancing, optimal transport, and statistical inference.
  • Integrates both lectures into navigation and adds supporting references.
File Description
lectures/​information_projections.md Adds projection theory, applications, and worked exercises.
lectures/​exponential_families.md Adds maximum entropy, exponential-family geometry, and estimation examples.
lectures/​divergence_measures.md Links readers to both new lectures.
lectures/​_toc.yml Places both lectures after divergence measures.
lectures/​_static/​quant-econ.bib Adds 26 supporting bibliography entries.

Review effort: Balanced

Comment thread lectures/exponential_families.md Resolved
Comment thread lectures/exponential_families.md Resolved
Comment thread lectures/information_projections.md Resolved
Comment thread lectures/information_projections.md Resolved
Comment thread lectures/information_projections.md Resolved
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

📖 Netlify Preview Ready!

Preview URL: https://pr-1129--sunny-cactus-210e3e.netlify.app

Commit: 4bcd182

📚 Changed Lectures


Build Info

mmcky and others added 3 commits November 18, 2026 09:25
…e pass

Address the five Copilot review comments on PR 1129 with text changes:
- state the maximum entropy existence condition with the relative
  interior and relative boundary of the convex hull;
- restrict the moment-convergence claim to the normal family, and note
  that the moments of the natural-parameter estimator need not exist in
  general;
- require p to vanish wherever h does in the m-projection theorem;
- assume f and g are distinct with a common support before the
  Chernoff characterization, so the minimizer is interior;
- require a strictly positive joint distribution in the interactions
  exercise.

Style, following the QuantEcon style guide:
- italic rather than bold for emphasized statements, keeping bold for
  definitions;
- figure captions shortened to six words or fewer.

No code changes; printed outputs are unaffected.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Say what fig-ip-sanov shows, the decay rate converging to D(p* || q),
rather than naming the experiment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Make each caption say what its figure shows: equal parameter distances
with unequal KL divergences, maximum entropy die distributions by mean,
the grid solution matching the normal, the Bernoulli conjugate pair,
maximum likelihood estimates in both parameterizations, the
moment-matched normal for a bimodal mixture, and the point equally far
from both distributions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@mmcky

mmcky commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

thanks @thomassargent30

@mmcky
mmcky merged commit 0d5c687 into main Nov 17, 2026
2 checks passed
@mmcky
mmcky deleted the nov_penn8 branch November 17, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants