Skip to content

[FIX] Leave zero sigmas out of the log-linear sigma fill - #1610

Open
scks1 wants to merge 2 commits into
casact:mainfrom
scks1:fix-loglinear-zero-sigma-1609
Open

scks1 wants to merge 2 commits into
casact:mainfrom
scks1:fix-loglinear-zero-sigma-1609

Conversation

@scks1

@scks1 scks1 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary of Changes

WeightedRegression.loglinear_interpolation fills the sigmas that Development cannot estimate by regressing log(sigma) on development period. It replaced a zero sigma with 1e-320 and fitted its log (-736.8) at full weight, so one zero-variation period could push the fills to 1.8e308 (details in #1609).

This PR changes only that function:

  • Only positive sigmas enter the regression. A zero sigma has no logarithm, and the 1e-320 stand-in put a point near -737 on the log scale that outweighed the other points. Zero sigmas are not filled and come out as before (Mackchainladder's standard error has different estimates compared to R's ChainLadder library #234).
  • Where a triangle has fewer than three positive sigmas, the fill comes from mack_interpolation instead. One point gives no slope, and a line through two points fits both exactly, so nothing is left to judge it by. With a cutoff of two, as in the issue, clrd IncurLoss still got a fill of 1,921,341 extrapolated from two sigmas, 0.0268 and 11.12.
  • Triangles without zero sigmas and with three or more positive sigmas give byte-identical results.

A regression test on clrd CumPaidLoss covers Brotherhood Mut Ins Co / prodliab (one positive sigma, where Mack has nothing to fill from and the old 1.8e308 fills become NaN), Great Amer Grp / medmal (two positive sigmas, where the fallback's 36-48 fill must equal the sigma_interpolation="mack" value) and Buckeye Ins Grp / othliab (five positive sigmas and two zeros). For Buckeye the test refits the five positive sigmas with np.polyfit and checks that the 12-24 and 108-120 fills lie on that line; the 12-24 fill is 7.1819 (it was 2.50e137).

Related GitHub Issue(s)

Closes #1609

AI/LLM Usage

I used Árbol, an AI assistant running on Muse, under my direction to trace the cause, write the change and the test, run the comparisons and draft this description. I reviewed the change and the results.

Additional Context for Reviewers

Before and after on main ee2c24d, default settings:

before after
clrd CumPaidLoss sigma_ cells above 1e10 20 0
clrd IncurLoss sigma_ cells above 1e10 4 0
clrd BulkLoss sigma_ cells above 1e10 12 0
Buckeye othliab total_mack_std_err_ 1.74e139 540.83
Columbia Ins Grp prodliab largest mack_std_err_ 2.90e37 113.23
  • Cells changed: 469 of 7,011 sigma_ cells in CumPaidLoss, 402 in IncurLoss, 492 in BulkLoss. In CumPaidLoss, 20 of the old values were above 1e10, 74 were below 1e-300 and 59 were NaN. 127 of the new values are NaN.
  • sigma_interpolation="mack" results are byte-identical on every triangle checked.
  • raa, genins, mw2008, mw2014, abc and ukmotor give byte-identical sigma_, std_err_ and Mack totals. The quarterly sample changes in four tail fills of one column (4.59e-65 becomes 0.0117), and its second total Mack standard error moves from 768.02 to 768.08.
  • The sparse backend gives the same sigma_ as numpy on clrd CumPaidLoss, and the new test runs under both fixture backends.
  • Tests: chainladder/development and chainladder/methods give 240 passed with this change and 238 passed on main (the difference is the two new test runs). The new test fails on main.

Submitter's Checklist

  • I have reviewed and am adhering to the standards outlined in the project Governing Doc.
  • The PR subject title summarizes the changes, with one proper prefix ([FIX], [FEAT], [DOCS], [TST], [CHORE], or [BRK]).
  • I am a human (not a bot), and this PR form is written by a human.

Reviewer's Checklist

  • The implementation addresses the associated issue(s).
  • The implementation is appropriate, maintainable, and follows ARCHITECTURE.md.
  • PR subject title has the proper prefix and the subject is appropriate.
  • Relevant issue(s) are linked.
  • AI/LLM usage is disclosed and appropriate.
  • Documentation and tests are appropriate.
  • CI tests passed, or any failures are acceptable.
  • Leave a comment with the final recommendation (e.g. approve as is, request a secondary review, or flag an area for more review).

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Pyright Type Completeness

View the full pyright --verifytypes output for this commit

Project (full chainladder package, at this PR's head): 18.1% of exported symbols fully typed (303 / 1675)

Known Ambiguous Unknown Total
Project (head) 303 96 1276 1675

Other symbols referenced but not exported by chainladder: 24

Known Ambiguous Unknown Total
Other (head) 11 1 12 24

Symbols without documentation:

  • Functions without docstring: 331
  • Functions without default param: 0
  • Classes without docstring: 10

Patch (exported symbols added or changed by this PR): 0.0% fully typed (0 / 1)

Known Ambiguous Unknown Total
Patch 0 0 1 1
Patch symbol details
Symbol Status Change
chainladder.development.tests.test_development.test_loglinear_sigma_skips_zero_sigmas ❌ unknown new

@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.00%. Comparing base (c113fd9) to head (eb71331).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1610   +/-   ##
=======================================
  Coverage   95.00%   95.00%           
=======================================
  Files         100      100           
  Lines        6461     6466    +5     
  Branches      820      821    +1     
=======================================
+ Hits         6138     6143    +5     
  Misses        204      204           
  Partials      119      119           
Flag Coverage Δ
unittests 95.00% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@scks1

scks1 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

The (3.14, pandas3, ubuntu-latest) job failed with "The hosted runner lost communication with the server" partway through Run tests, and it left no log. Every other job passed, including 3.14 on pandas 2.3.3. Could you re-run that job when you get a chance? I don't have permission to re-run checks on this repo.

(This comment was written with Árbol, an AI assistant, under my direction.)

@henrydingliu henrydingliu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responding to the questions in the PR summary

R also switches to Mack when the slope's p-value is above 0.05. I did not add that. The largest fill left on clrd is 3,341,035 (CumPaidLoss, 24-36, extrapolated back from four sigmas that fall from 14.7 to 0.00059). Its p-value is 0.048, so R would keep it too.

anyone using sigma_ should understand that smaller triangles or triangles with lots of missing data is going to cause issues. i don't think we need to nanny

R warns when it switches to Mack. This PR does not warn, since clrd would warn on many triangles at once. Should it?

since the sigma calculations are happening on every development even when people are just trying to get to a normal chainladder result, i don't want to warn on a corner case on an unused statistic. if we had a toggle to turn sigma calcs on or off (defaulting to off), then it becomes reasonable to warn this switching behavior when someone intentionaly turns on the sigma calcs.

mack_interpolation still returns 9.9999e-321 where a sigma is 0, because of the same 1e-320 stand-in. Turning those into 0 would make them NaN under "mack" and could change Mack results, so I left it out.

i'm not following. so is mack interpolation not doing anything?


# One positive sigma: the fill falls back to Mack's approximation.
brotherhood = cl.Development().fit(tri.loc["Brotherhood Mut Ins Co", "prodliab"])
assert np.nanmax(brotherhood.sigma_.set_backend("numpy").values) < 1e3

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please double check my logic flow here.

we are here to test mack's approximation for sigma_ when loglinear regression doesn't have enough valid data. have we tested mack's approximatio itself somewhere? i.e. are any of our hardcoded mack tests covering the approximation for sigma?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your logic is right, and the Brotherhood case didn't test it. Its raw sigmas are NaN, NaN, 7.67, 0, then NaN. Mack's approximation needs two positive sigmas just before a gap, so it fills nothing there; the case only checks that the 1.8e308 fills are gone. I fixed its comment and added Great Amer Grp / medmal, whose sigmas are 14.98, 52.90, then a gap: the fallback fills 36-48 with 14.978, and the test checks that against sigma_interpolation="mack" on the same triangle. On main that fill is 1.2e-142. The approximation itself is covered by printed values: test_mack1999_hardcode checks mortgage's filled last sigma (16.887 against 16.89 in Mack 1999), and test_cdr_matches_merz2008 checks mw2008's filled last sigma^2 (0.040 against 0.04). test_mack1994_hardcode only reaches it through the oldest origin's standard error.

(This comment was written with Árbol, an AI assistant, under my direction.)

"""
Fill undefined sigmas by regressing log(sigma) on development period.

As in R's ChainLadder (``estimate.sigma``), only positive sigmas enter

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we are pivoting away from pinning any results to R chainladder implementation, we should also provide independent rationale for design choices, rather than referring back to R chainladder.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten without R. A zero sigma has no logarithm, and the 1e-320 stand-in put a point near -737 on the log scale that outweighed every other point. A line through two points fits both exactly, so nothing is left to judge it by; on clrd IncurLoss one such line gave a fill of 1,921,341 from sigmas of 0.0268 and 11.12.

(This comment was written with Árbol, an AI assistant, under my direction.)

fill_ = xp.nan_to_num(xp.exp(reg.x * slope + intercept) * (1 - estimated))
use_mack = xp.sum(w, axis=self.axis, keepdims=True) < 3
if use_mack.any():
with warnings.catch_warnings():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i didn't see a rationale for this warning suppression in the PR summary. any specific issue this is trying to solve?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default path now calls mack_interpolation, which divides by the square of the sigma two periods back. Where that sigma is 0, numpy warns "divide by zero" and "invalid value" in cells whose fill comes out as 0 anyway. Without a filter, a default fit would gain those two warnings whenever one of its triangles takes the fallback and any of them has a zero sigma. I replaced the blanket RuntimeWarning filter with np.errstate(divide="ignore", invalid="ignore") around that one call. On clrd the default path now gives the same warnings as with the old filter, under numpy and sparse.

(This comment was written with Árbol, an AI assistant, under my direction.)

@scks1

scks1 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. I'll leave out the p-value fallback and the warning.

On "is mack interpolation not doing anything?": it does fill. It fills a missing sigma from the two sigmas just before it, which covers the usual missing last sigma. My bullet was only about one detail: where a sigma is exactly 0, mack_interpolation returns the 1e-320 stand-in, which shows as 9.999889e-321, instead of 0. That is separate from this PR, so I've dropped the bullet and can open an issue for it if you want one.

(This comment was written with Árbol, an AI assistant, under my direction.)

@henrydingliu

Copy link
Copy Markdown
Member

On "is mack interpolation not doing anything?": it does fill. It fills a missing sigma from the two sigmas just before it, which covers the usual missing last sigma. My bullet was only about one detail: where a sigma is exactly 0, mack_interpolation returns the 1e-320 stand-in, which shows as 9.999889e-321, instead of 0. That is separate from this PR, so I've dropped the bullet and can open an issue for it if you want one.

are you referring to this line?

assuming yes, then i'm wondering if the brotherhood test is valid, i.e. if we plan to change that line, wouldn't that likely change brotherhood test?

while reading through the mack interpolation, the docstring makes me wonder what usefulness may remain of this mack-interpolation fallback if we change that line to not creep 0 to 1e-320

loglinear_interpolation replaced a zero sigma with 1e-320 and fitted its
log (-736.8) at full weight, so a single zero-variation period could push
the extrapolated sigmas to 1.8e308. Only positive sigmas now enter the
fit, as in R's ChainLadder estimate.sigma, and zero sigmas are not filled.
Where fewer than three positive sigmas remain, the fill falls back to
Mack's approximation, which R's Mack.S.E also does for one point and for
two points (undefined slope p-value).

Closes casact#1609.

Developed with an AI assistant (Árbol, running Muse).
Give the log-linear fill's reasons in the docstring: a zero sigma has no
logarithm, and a line through two points leaves no residual to judge it by.
Scope the fallback's error suppression to numpy's divide and invalid
flags, which mack_interpolation sets where an earlier sigma is zero.
Add a case where the fallback fills a sigma and compare it with
sigma_interpolation="mack", and correct the comment on the case where
Mack has nothing to fill from.

Developed with an AI assistant (Árbol, running Muse).
@scks1
scks1 force-pushed the fix-loglinear-zero-sigma-1609 branch from a274a90 to eb71331 Compare November 16, 2026 08:46
@scks1

scks1 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Yes, that's the line.

The Brotherhood test passes with and without it. The fill only uses that value squared or to the fourth power, and 1e-320 ** 2 == 0.0, so every filled sigma comes out the same. I ran the test both ways on this branch and compared all of clrd (CumPaidLoss, IncurLoss, BulkLoss) with the line removed:

  • log-linear: no change.
  • mack: sigma_ shows NaN instead of 9.999889e-321 at each zero sigma, as log-linear already does, and std_err_ shows 0 instead of a subnormal value (at most 1e-320) in 3, 10 and 192 cells.
  • MackChainladder ibnr_, mack_std_err_ and total_mack_std_err_ with sigma_interpolation="mack" on CumPaidLoss and IncurLoss: no change.

On the docstring: Mack's rule matches a two-point log-linear extrapolation only while the sigmas fall. Mack (1993, p. 217) offers log-linear regression or, more simply, keeping the ratio of successive sigmas constant as long as they decrease, which gives sigma[k]^2 = min(sigma[k-1]^4 / sigma[k-2]^2, sigma[k-2]^2, sigma[k-1]^2). That cap is what the fallback keeps, and it doesn't depend on the 1e-320 line. With fewer than three positive sigmas the log-linear fit is a line through at most two points, and it can climb: for Great Amer Grp medmal the line gives 186.8 where Mack's rule gives 14.98.

I can correct the mack_interpolation docstring in this PR, along with my own log-linear docstring, which says zero sigmas are left as they are when they come out as NaN. Removing the 1e-320 line could be a separate PR if you want it.

(This comment was written with Árbol, an AI assistant, under my direction.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Log-linear sigma fill regresses on zero sigmas as log(1e-320), giving sigma_ up to 1.8e308

2 participants