Skip to content

[BUG] TweedieGLM fails on an incremental triangle with empty cells since #1637 #1640

Open

Description

@scks1

Are you on the latest chainladder version?

  • Yes, this bug occurs on the latest version (main at 8d8f90c).

Describe the bug in words

Since #1637, TweedieGLM().fit fails on an incremental triangle that has an empty cell inside its observed area, such as a development quarter with no new reports. Reported counts from prism at OYDQ have 17 such cells, and the fit raises ValueError: sample_weight.shape == (203,), expected (220,)! from scikit-learn. Nine of the ten grains of these counts fail the same way, all but OYDY, which has no such cells. BarnettZehnwirth, which always passes the weights, fails on the same nine.

DevelopmentML._prep_X_ml builds one row per cell of the cumulative triangle (learning.py L265-L271), and _prep_w_ml builds one weight per non-empty cell of the triangle it is given (L283-L298). The counts differ on such a triangle. #1637 made TweedieGLM pass the weights on every fit (glm.py L223), so the plain fit now fails. Before #1637 only a weighted fit failed, such as TweedieGLM().fit(tri, sample_weight=tri * 0 + 1) on c113fd9.

The comment at glm.py L221-L222 says the unweighted fit should stay unchanged. Before #1637 the plain fit on this triangle gave the same coefficients as the fit on tri.incr_to_cum(), since _prep_X_ml fills the empty cells with zero increments.

Building the weights from X.incr_to_cum() in _prep_w_ml, for both the base mask and TriangleWeight, fixes it in a local copy. The incremental fit then matches the cumulative fit, with and without drop_valuation="2017Q2", and matches the plain fit on c113fd9 to 10 decimal places.

How can the bug be reproduced?

import chainladder as cl

tri = cl.load_sample("prism").sum()["reportedCount"].grain("OYDQ")
cl.TweedieGLM().fit(tri)

What is the expected behavior?

import numpy as np


def coef(X):
    return cl.TweedieGLM().fit(X).model_.estimator_ml["model"].coef_


np.testing.assert_allclose(coef(tri), coef(tri.incr_to_cum()))

Would you be willing to contribute this ticket?

  • Yes, absolutely!
  • Yes, but I would like some help.
  • No.

(This issue was written with Árbol, an AI assistant, under my direction.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Scope

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions