Skip to content

ENH: Add an opt-in QR solver to IV2SLS and IVLIML - #870

Merged
bashtage merged 25 commits into
mainfrom
iv-qr-2sls
Nov 10, 2026
Merged

bashtage merged 25 commits into
mainfrom
iv-qr-2sls

Conversation

@bashtage

@bashtage bashtage commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Adds an opt-in QR method to IV2SLS and IVLIML for 7.2. It supersedes #850, which is closed. The default computation does not change in 7.2; a FutureWarning announces that the default becomes "qr" in 8.0.

What changes

  • New keyword: IV2SLS, IVLIML and their from_formula take a keyword-only method, as in statsmodels, with values "pinv" and "qr" (not case sensitive).
  • method="qr":
    • Tests the rank of [exog, endog] and [exog, instruments] after scaling the columns to unit length. They are rank deficient if the condition number is at least 1e9.
    • Computes the k-class estimates from one Householder QR of the instruments, without forming cross products (kclass_factors). For 2SLS this is the QR least-squares solve of the projected problem, as in R's ivreg. For 0 ≤ kappa < 1, which includes OLS, it is a stacked least-squares problem. Otherwise (2SLS, LIML, and negative kappa) it uses R'GR with G = I - (kappa-1)F'F.
    • Computes the covariance (all four estimators) from R⁻¹, with scores in orthonormal coordinates, so robust, clustered and kernel SEs keep the cond·eps accuracy.
    • Computes the LIML kappa from a QR plus an SVD (σ_min(M₁W R⁻¹)²). This removes case 2 of IV: LIML kappa with an endogenous regressor spanned by the instruments, kappa's unit dependence, scale-dependent instrument rank check #823: the old kappa changed with the units when there were 2 or more endogenous regressors.
  • method="pinv" gives 7.1's results bit for bit. fit computes pinv(z) once instead of three times, and the covariance reuses pinv(z) @ x and x'z pinv(z) x. This about halves the time of fit.
  • The default, None, behaves as "pinv" and issues a FutureWarning that the default changes to "qr" in 8.0, and that method="pinv" keeps the current results and silences it. The warning points at the caller's line, also through from_formula and linearmodels.formula.iv_2sls/iv_liml. When the rank check of "pinv" refuses columns only because of their units, the error suggests method="qr".
  • Lower-level functions: the IV covariance classes, estimate_parameters, cragg_donald(_f), kleibergen_paap(_f) and FirstStageResults take method too, with the same default and warning. Code inside linearmodels always passes method: the models pass their own, and AbsorbingLS and IVGMM pass "pinv". The first-stage results and the tests of IVResults (Sargan, Basmann, Wooldridge, Durbin, Wu-Hausman, Anderson-Rubin, C) use the model's method.
  • Docs: the Notes of IV2SLS explain both methods and their accuracy: about 16 − log10(c) correct digits with "qr" and 16 − 2·log10(c) with "pinv", where c is the condition number of the scaled data (e.g. 6 digits at c = 1e5 and none at 1e8 with "pinv"). The other docstrings point there. The 7.2 notes describe the change briefly. The IV example notebooks pass method="qr", and iv_basic-examples has a cell on the two methods.
  • Not changed: IVGMM, IVGMMCUE, IVJIVE and AbsorbingLS keep today's rank check and computation. They are not k-class estimators; GMM inverts Z'Z and needs its own follow-up.

Evidence

  • Exact references (tests): the k-class estimates and all four 2SLS covariances are exact on Hadamard-built data, with the references computed with fractions. Fixed kappas use data without an error term, since a residual adds a cond² term for any method. Each case also runs with permuted rows, so that the result can't depend on the BLAS summation order.

    Collinearity δ "qr" params "qr" SEs "pinv" params
    2⁻²⁰ 5e-11 9e-12 1.2e-4
    2⁻²⁷ 6e-9 1.2e-9 300% off
    2⁻³⁰ — — LinAlgError
  • Scale invariance (tests): rescaling y, an exog, an endog or an instrument by 2^±60 leaves the "qr" params and SEs bit-identical after mapping back, for all four covariance types and for 2SLS, LIML, Fuller, kappa = 0.5 and kappa = −0.5. So are the diagnostics listed above, kappa included. The one exception is the automatic kernel bandwidth, which sums score columns and so depends on units by definition, with "pinv" too.

  • Simulated data, 7,560 data sets from the BUG: Test the rank of the IV inputs after scaling the columns #850 study. Scored against a 120-digit reference at tolerance 1e-6. A refusal is "needed" if the exact estimate is undefined or QR can't get within 1e-6. The grid is deliberately heavy in collinear and extreme-scale data, so compare the rows rather than reading the shares as frequencies.

    Accurate Refused, needed Refused, unneeded Loud failure Silently wrong
    main = branch default 19.7% 41.6% 28.7% 2.1% 7.9%
    "qr" 50.0% 45.0% 5.0% 0.0% 0.0%

    The "qr" row is the same for the 2SLS params, all four SE types and LIML; the main row varies by at most 0.2 points. Median error with collinear regressors:

    δ main params "qr" params main robust SE "qr" robust SE
    1e-6 5.3e-4 4.1e-11 7.7e-4 2.3e-11
    1e-7 4.7e-2 3.6e-10 7.8e-2 2.5e-10
    1e-8 fails 4.0e-9 fails 2.2e-9

    These were computed with the first version of this PR. The later simplifications don't change the arithmetic of these "qr" paths, and on the 38,504 numbers below "qr" is bit-identical to that version.

  • Default path = main, bit for bit: 38,504 numbers compare equal with exact equality. The fits cover 2SLS, LIML, Fuller, kappa 0/0.5, OLS, IVGMM and AbsorbingLS; unadjusted, robust, debiased, one- and two-way clustered, and automatic and fixed kernel covariances; weighted, scaled and nearly collinear data; and every test statistic and the first-stage results. The same holds with method="pinv" given explicitly.

  • Large models: seconds of fit (best of 3, 1 BLAS thread), as main → branch default / "qr". Peak memory is what fit allocates, measured with tracemalloc.

    n=500,000, k=12, k_z=18 n=50,000, k=110, k_z=130 n=10,000, k=310, k_z=330
    2SLS unadjusted 2.16 → 1.07 / 0.73 3.77 → 1.72 / 1.33 2.13 → 0.98 / 0.60
    2SLS robust 2.14 → 1.26 / 0.76 3.63 → 1.79 / 1.43 2.28 → 1.06 / 0.72
    2SLS clustered 2.28 → 1.18 / 0.86 3.71 → 1.81 / 1.48 2.32 → 1.12 / 0.73
    LIML robust 2.11 → 1.09 / 0.92 3.81 → 1.84 / 1.54 2.24 → 1.11 / 0.88
    peak memory (MB) 256 → 267 / 198 192 → 196 / 146 104 → 105 / 76
  • Test suite: pytest -m "not slow" linearmodels gives 23,385 passed, 2,473 skipped and 47 xfailed (main: 23,095 passed). The new tests are in linearmodels/tests/iv/test_numerical_method.py, including regression tests for negative kappa and a 1-D y with "qr". The other tests don't pass method: filterwarnings in pyproject.toml ignores the default-method warning, and the few tests that turn every warning into an error ignore it explicitly. The IV tests also pass with the warning turned into an error whenever code inside linearmodels, rather than a test, triggers it.

  • Style and types: black, isort, flake8 and ruff are clean on linearmodels, and numpydoc lint is clean on the changed modules. mypy shows only errors that are not in code this PR changes.

Decided

  • 7.2.
  • The keyword is method (as in statsmodels), with values "pinv" and "qr", and both methods stay for now.
  • The models and the lower-level functions warn when method is not given. Code inside linearmodels always passes it.
  • The notebooks pass method explicitly. The tests filter the warning, except the tests of the warning.

Builds on the unit-length rank check proposed by inchang-ing in #850.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.86130% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 99.69%. Comparing base (19bfbcf) to head (1d93fb2).

Files with missing lines Patch % Lines
linearmodels/iv/covariance.py 98.92% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main     #870    +/-   ##
========================================
  Coverage   99.68%   99.69%            
========================================
  Files         201      202     +1     
  Lines       38176    38797   +621     
  Branches     2442     2497    +55     
========================================
+ Hits        38057    38677   +620     
  Misses         51       51            
- Partials       68       69     +1     
Flag Coverage Δ
adder 99.68% <99.86%> (+<0.01%) ⬆️
subtractor 99.68% <99.86%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bashtage
bashtage marked this pull request as ready for review November 7, 2026 10:31
Add the keyword solver to IV2SLS, IVLIML and their from_formula. solver="qr"
tests the rank of [exog, endog] and [exog, instruments] after scaling the
columns to unit length, with a relative tolerance of 1e-9, and computes the
k-class estimates, their covariance and the kappa of LIML from QR
decompositions, as R's ivreg does. The results do not depend on the units of
the variables, and their rounding error grows with the condition number of
the scaled data, not with its square.

The default is solver="legacy", which computes as before. If solver is not
given, a FutureWarning is issued for data that solver="qr" finds rank
deficient, and the error for columns that are only rank deficient because of
their units suggests solver="qr". linearmodels 8.0 will use "qr" by default.

The covariance estimators, proj/annihilate and the Cragg-Donald and
Kleibergen-Paap statistics take the same keyword, and the first stage
results and the tests of IVResults use the solver of the model.

Builds on the unit-length rank check of #850. Fixes case 2 of #823 (kappa
depends on the units with 2 or more endogenous regressors) with solver="qr".
Rename solver="legacy" to solver="pinv", so that both values name the
numerical method. The default, None, still computes as "pinv" and warns.
The covariance estimators accepted solver positionally after extra_df, and
the private _first_stage_rank and _rank did too. Every function that takes
solver now requires it as a keyword, and a test checks the public ones.
…heck

The default rank check computed the singular values of the columns scaled to
unit length for every model, to decide whether to warn, which made creating a
model up to twice as slow. The rank is now computed from the singular values
of the columns as they are, as matrix_rank computes it, and the scaled columns
are only tested if the condition number is at least 1e9 / (2 sqrt(ncol)).
Scaling the columns cannot increase the condition number by more than
sqrt(ncol) (van der Sluis), so the decision does not change.

With solver="qr", fit decomposes the instruments once for kappa, the estimate
and the covariance, OLS (kappa = 0) does not decompose the instruments, and
the covariance uses one QR decomposition of x when z is x.
solver="pinv" (bit-for-bit the same results):
- fit computes pinv(z) once for kappa, the estimate and the covariance,
  which each computed it before, and kappa reuses it for x1 if x1 is z
- the covariance reuses pinv(z) @ x, x'z pinv(z) x and V
- the terms that are multiplied by exactly 0 when kappa is 0 or 1, such as
  the x'x of 2SLS, are not computed (the inputs are finite)
- the kernel covariance computes the projected regressors once

solver="qr":
- the covariance reuses the factors of the estimate, and computes U from R
  instead of decomposing x again for OLS
- kappa < 1 applies the reflections of the QR of x to y without forming Q
- the rank check and kappa use QR without forming Q, and proj uses QR and
  only falls back to the SVD if the scaled columns are rank deficient
Sharing pinv(z) between kappa, the estimate and the covariance kept it alive
while kappa decomposed x1, which raised the peak memory of fit by up to 19%.
Kappa now computes the residuals on x1 first, so that their decomposition is
freed before the one of z is computed and kept. The residuals are
independent, so the results are the same.
With kappa other than 1, the k-class estimator fits a residual of y - Xb,
whose error grows with the square of the condition number. The Hadamard
data only gave exact estimates with kappa = 1.25 because the rounding error
of their products cancelled in the order of the sums of OpenBLAS on x86. It
did not on Windows ARM64 (6.4e-7 and 1.0e-2), and does not on x86 with the
rows permuted. Fixed kappa is now tested on data without the error, where
the error grows with the condition number, with the rows in both orders.
The tolerance of the Fuller test, which has a residual, follows its bound.
With solver="pinv", kappa only uses the cross product of the residuals on
x1, which is now computed before z is decomposed, so that the residuals are
freed. This removes the last 2-4% of extra peak memory of fit compared with
main; the results are the same. Also narrow the optional basis of
kclass_factors for mypy.
The keyword of the numerical method of IV2SLS and IVLIML is now method, as
in statsmodels, with the values "pinv" and "qr". The internal attribute is
_numerical_method, since _method holds the name of the estimator.

If method is not given, a FutureWarning says that the default changes from
"pinv" to "qr" in linearmodels 8.0, that method="qr" gives the new method
now and that method="pinv" keeps the current one without the warning. For
data that method="qr" finds rank deficient, the warning also says so. Only
the models that the user creates warn: the models of the diagnostics and the
covariance estimators get the method of the model.

The test suite ignores the warning in pyproject.toml; the tests of the
warning use pytest.warns. The kappa tests that turn every warning into an
error ignore it too.
The IV covariance estimators, estimate_parameters, cragg_donald,
cragg_donald_f, kleibergen_paap, kleibergen_paap_f and FirstStageResults
default to method=None, which is "pinv". If they are called directly from
outside linearmodels, a FutureWarning says that the default will be "qr" in
linearmodels 8.0. They do not warn when linearmodels calls them: the models
pass their own method, and AbsorbingLS and IVGMM use the default without a
warning. The tests of linearmodels count as callers outside linearmodels.

The docstrings of the models and of fit explain that with "pinv" the
estimates lose digits well below the condition number of 1e9 at which "qr"
rejects the data, about 16 - 2 log10(c) correct digits against 16 - log10(c).
The tests pass method="pinv" wherever they create IV2SLS or IVLIML or call a
lower-level function that takes method, except the tests of the warning,
which use pytest.warns. Where the estimator is a parameter that can also be
IVGMM, IVGMMCUE, IVJIVE or a panel model, the new test helper method_kwargs
adds method only for those that take it. pyproject.toml no longer ignores
the warning, and the kappa tests turn every warning into an error again. The
suite passes with the warning promoted to an error.
The IV examples pass method="qr", so that they do not show the warning, and
the basic examples have a new cell on the two numerical methods: what each
does, the trade-offs (accuracy and units, the stricter rank check, the exact
results of 7.1 with "pinv") and why "qr" is recommended.
The 7.2 notes explain the digits that "pinv" loses below the tolerance of
"qr", why IVGMM, IVGMMCUE and IVJIVE do not have method, and that the
lower-level functions warn only when they are called directly.
Codecov found added lines that no test runs. Those that no input can
reach, or that change nothing, are removed:

- _rank_and_condition_number: the fallback to matrix_rank. NumPy 1
  computes the tolerance as (smax * m) * eps and NumPy 2 as
  smax * (m * eps), which are the same number since eps is a power of 2,
  unless smax * m overflows.
- _estimate_kappa: the checks of the keys of shared, which is always
  empty when kappa is estimated.
- kclass_solve: the check that the factors have t, which every caller
  computes, and the guard for no regressors, since solve_triangular
  without trans handles empty arrays.
- proj: the separate branch for fewer rows than columns. The QR
  decomposition then has a square Q, which is the projection if x has
  full row rank.
- The OLS shortcut of the covariance estimators tests the shape in
  array_equal.

The guards for no regressors in kclass_factors and in the covariance are
kept, with comments: qr_multiply fails if x has no columns, and
solve_triangular with trans="T" fails with empty arrays in SciPy 2.0.dev.
- The rank tests with a constant and a full set of dummies as exog,
  which proj projects on with the SVD. "qr" agrees with pinv, and
  Kleibergen-Paap does not change without the redundant dummy. Without
  the SVD, Cragg-Donald was 307.7 instead of 303.9.
- A model with fewer observations than regressors is rank deficient
  with both methods.
- estimate_parameters and the covariance estimators with no regressors
  return empty arrays with both methods.
- The covariance estimators agree when the model is just identified, so
  that z has the shape of x, which tests that the OLS shortcut is only
  used if z is x.
- proj agrees with pinv for full rank, rank deficient and wide x.

method_kwargs no longer catches errors of inspect.signature, which
cannot fail for the IV models and functions.
bc96124 removed it after checking SciPy 1.18 and 2.0.dev, where
solve_triangular returns an empty array. SciPy 1.13 and earlier raise
LinAlgError for empty arrays, which made test_no_regressors fail in the
minimums (SciPy 1.8), NumPy 1.26 (1.11) and Cython coverage (1.13) jobs.
Checked with SciPy 1.13.1 and 1.11.4, where all 255 tests of
test_numerical_method.py now pass.
bashtage and others added 4 commits November 10, 2026 15:39
Bugs of method="qr":

- A negative kappa (given, or from a large fuller) took the square root
  of kappa and raised "array must not contain infs or NaNs". The G
  factorization is exact for kappa < 0, and is used for it now.
- estimate_parameters with a 1-D y and 0 < kappa < 1 raised in vstack.
  The stacked y is concatenated now.

Simplifications, with the same results (bit for bit for "pinv" and the
default, and for "qr" on well-conditioned data):

- resolve_method warns whenever method is None, with the stacklevel of
  the first frame outside linearmodels/iv. The frame check of the
  package and tests directories is gone: AbsorbingLS and
  IVGMMCovariance pass method="pinv". The models use resolve_method
  too, and the subclasses of the covariance estimators do not repeat it.
- The FutureWarning no longer says whether "qr" would reject the data,
  which removes _rank_and_condition_number, the van der Sluis bound and
  the deferred warning. The rank check of "pinv" is matrix_rank again.
- _estimate_kappa decomposes z after the residuals on the exogenous
  regressors and returns the decomposition, which fit reuses for the
  estimate and the covariance, instead of filling a dict. "pinv" keeps
  the arithmetic of 7.1 and reuses pinv(z) @ x and x'z pinv(z) x,
  without the products dict, the cache of V and the skipped terms.
- kclass_factors always gets the basis of z and always returns U. OLS
  (kappa = 0) uses the stacked factorization, which gives the same R and
  t. The special cases of the covariance for OLS, and the guards for no
  regressors, are gone: "qr" raises for no regressors.
- proj with "qr" takes the singular vectors of a rank deficient x from
  the SVD of R.
- The IVLIML notes were duplicated. The notes of method are in IV2SLS,
  and the descriptions of method elsewhere are shorter.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The tests ignore the FutureWarning of the default method through
filterwarnings in pyproject.toml, instead of passing method in every
test, and method_kwargs is gone. The tests of the warning are in
test_numerical_method.py. The tests that turn every warning into an
error ignore this one explicitly.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Also restore conf.py, whose change is not part of this pull request.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@bashtage
bashtage merged commit 5799ee0 into main Nov 10, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant