Skip to content

[FEAT] Check Development parameter values in fit and read_json - #1606

Merged
henrydingliu merged 11 commits into
casact:mainfrom
scks1:feat-validate-params-1569-solo
Nov 16, 2026
Merged

henrydingliu merged 11 commits into
casact:mainfrom
scks1:feat-validate-params-1569-solo

Conversation

@scks1

@scks1 scks1 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary of Changes

Following #1569, Development gains a _check_param_values(X=None) method. fit calls it, and so does read_json for a single estimator and for each pipeline step, on any estimator that defines it. The name avoids scikit-learn's own _validate_params and _check_params. It checks:

  • average must be 'volume', 'simple', 'regression' or 'geometric', the values of a new Average Literal alias read with get_args.
  • n_periods must be -1 or a positive integer; 0 stays an alias for -1.
  • sigma_interpolation must be 'log-linear' or 'mack', the values of a SigmaInterpolation alias.

A bad value raises a ValueError naming the parameter. Lists are checked element by element, and dicts and callables are checked in fit once resolved, through a new Common._assumption_values next to _validate_assumption. Before, average='bogus' loaded and then raised a KeyError in fit, n_periods=-5 fitted as -1, and sigma_interpolation='bogus' left sigma_ as NaN.

The average docstring no longer offers a numeric exponent: fit passed numbers through as weight exponents from v0.7.7 until 59b7f9e (first released in v0.8.15), and on main they raise a KeyError.

Tests in test_development.py cover eight bad values in fit, n_periods=0 fitting like -1, and valid lists, dicts and callables. A new chainladder/development/tests/test_io.py checks two bad values in read_json, that read_json still loads Trend, BerquistSherman and ParallelogramOLF, and that it loads a dict assumption without a triangle.

Related GitHub Issue(s)

Closes #1569

AI/LLM Usage

This change was made with Árbol, an AI assistant running on Muse, working under my direction. It wrote the change and the tests, ran the checks and drafted this description. I reviewed the diff and the results before submitting.

Additional Context for Reviewers

Left for follow-ups: the read_json checks for a non-chainladder __class__ and for nested estimators, a discussion of deprecating n_periods=0, and checks on the length of per-link-ratio parameters.

Other modules and Development's other parameters are left for later PRs. This PR doesn't depend on #1568's PR. It merges cleanly into main. It conflicts with #1389 on adjacent lines of Development.fit, and it and #1610 both add tests at the end of test_development.py, so whichever merges second needs a small rebase. The development, utilities and JSON tests pass with the uv.lock versions, and ruff is clean.

Submitter's Checklist

  • I have reviewed and am adhering to the standards outlined in the project Governing Doc.
  • The PR subject title summarizes the changes, with one proper prefix ([FIX], [FEAT], [DOCS], [TST], [CHORE], or [BRK]).
  • I am a human (not a bot), and this PR form is written by a human.

Reviewer's Checklist

  • The implementation addresses the associated issue(s).
  • The implementation is appropriate, maintainable, and follows ARCHITECTURE.md.
  • PR subject title has the proper prefix and the subject is appropriate.
  • Relevant issue(s) are linked.
  • AI/LLM usage is disclosed and appropriate.
  • Documentation and tests are appropriate.
  • CI tests passed, or any failures are acceptable.
  • Leave a comment with the final recommendation (e.g. approve as is, request a secondary review, or flag an area for more review).

DevelopmentBase and MethodBase gain a _validate_params hook that checks
nothing by default. Development overrides it: average must be 'volume',
'simple', 'regression' or 'geometric', n_periods must be -1 or a positive
integer, and sigma_interpolation must be 'log-linear' or 'mack'. A bad value
raises a ValueError naming the parameter. fit calls the hook with the
triangle, so lists are checked element by element and dicts and callables
after they resolve against it. read_json calls it with no triangle when it
loads an estimator or a pipeline step.

Before this, average='bogus' loaded and then raised a KeyError in fit,
n_periods=-5 or 0 fitted silently as -1, and sigma_interpolation='bogus'
left sigma_ as NaN.

Closes casact#1569.

Developed with an AI assistant (Árbol, running Muse).
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Pyright Type Completeness

View the full pyright --verifytypes output for this commit

Project (full chainladder package, at this PR's head): 18.2% of exported symbols fully typed (304 / 1673)

Known Ambiguous Unknown Total
Project (head) 304 96 1273 1673

Other symbols referenced but not exported by chainladder: 24

Known Ambiguous Unknown Total
Other (head) 11 1 12 24

Symbols without documentation:

  • Functions without docstring: 329
  • Functions without default param: 0
  • Classes without docstring: 10

Patch (exported symbols added or changed by this PR): 25.0% fully typed (2 / 8)

Known Ambiguous Unknown Total
Patch 2 0 6 8
Patch symbol details
Symbol Status Change
chainladder.development.development.Average ✅ known new
chainladder.development.development.SigmaInterpolation ✅ known new
chainladder.development.tests.test_development.test_fit_rejects_invalid_params ❌ unknown new
chainladder.development.tests.test_development.test_n_periods_zero_fits_as_all_periods ❌ unknown new
chainladder.development.tests.test_development.test_valid_params_pass_validation ❌ unknown new
chainladder.development.tests.test_io.test_read_json_leaves_dict_params_to_fit ❌ unknown new
chainladder.development.tests.test_io.test_read_json_loads_estimators_without_the_hook ❌ unknown new
chainladder.development.tests.test_io.test_read_json_rejects_invalid_params ❌ unknown new

@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.00%. Comparing base (ee2c24d) to head (a9def71).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1606      +/-   ##
==========================================
+ Coverage   94.84%   95.00%   +0.15%     
==========================================
  Files          99      100       +1     
  Lines        6424     6461      +37     
  Branches      811      820       +9     
==========================================
+ Hits         6093     6138      +45     
+ Misses        212      204       -8     
  Partials      119      119              
Flag Coverage Δ
unittests 95.00% <100.00%> (+0.15%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Developed with an AI assistant (Árbol, running Muse).
Developed with an AI assistant (Árbol, running Muse).
@henrydingliu

Copy link
Copy Markdown
Member

can you check the codecov? is it cupy again?

@scks1

scks1 commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

It isn't cupy this time. The three lines are new code that no test reaches: base.py 45–46, where _assumption_values returns nothing for a dict or callable assumption when there is no triangle (the read_json path), and utility_functions.py 456, the hasattr(estimator, "_validate_params") check in _estimator_from_json, which is never false.

That check hides a bug. sklearn's BaseEstimator has a _validate_params of its own, so every estimator passes the hasattr test, and sklearn's version reads a _parameter_constraints attribute that chainladder classes don't set. On this head, cl.read_json(est.to_json()) raises AttributeError for Trend, BerquistSherman and ParallelogramOLF, which all load on main.

I've pushed d420695, which calls the hook only on DevelopmentBase and MethodBase instances and adds a regression test over those three classes, and 1f99121, a test that loads a dict n_periods through read_json. In a local coverage run every patch line is covered and test_development.py passes.

One naming question: the hook has the same name as sklearn's private BaseEstimator._validate_params with a different signature. I kept the name you suggested on #1569, and I can rename it if you'd rather avoid the clash.

(This comment was written with Árbol, an AI assistant, under my direction.)

@henrydingliu

Copy link
Copy Markdown
Member

please disregard my name suggestion and use something that doesn't clash with sklearn.

n_periods=0 used to fit as -1 and now raises. Should 0 stay as an alias, or warn first?

let's keep the behavior but you can raise an issue to discuss deprecation

The average docstring says a numeric exponent works, but on main average=1.5 raises a KeyError. Should the docstring drop it, or should numeric support be a separate issue?

my guess is that it may have worked at one point. does average=1.0 even work now? if average=1.0 doesn't work, then we can reasonably conclude that it's not in use and can be removed from docstring

The read_json checks for a non-chainladder class and for nested estimators are not here. Here or in a follow-up?

sounds like distinct scope

scikit-learn's BaseEstimator defines _validate_params, so the chainladder
hook takes a name scikit-learn does not use. read_json still calls the hook
only on DevelopmentBase and MethodBase estimators, the classes that define it.

Developed with an AI assistant (Árbol, running Muse).
Development fitted n_periods=0 as -1 before the parameter check, and the check
now accepts 0 again so that behavior is unchanged. Negative values other than
-1 still raise. A test checks that n_periods=0 gives the same ldf_ and sigma_
as n_periods=-1 on raa.

Developed with an AI assistant (Árbol, running Muse).
The docstring said a numeric average sets the exponent of the regression
weights, but average=1.0, 1.5 and other numbers raise a KeyError on main,
because the weight lookup in weighted_regression.py accepts only the string
names. The docstring now lists only those names.

Developed with an AI assistant (Árbol, running Muse).
Comment thread chainladder/development/development.py Outdated
callable is checked value by value; see
``DevelopmentBase._validate_params``.
"""
averages = ("volume", "simple", "regression", "geometric")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was originally thinking the allowable values would be declared as a Literal constant, which can then be checked against via get_args. open to other ideas if that's not a reliable pattern

``DevelopmentBase._validate_params``.
"""
averages = ("volume", "simple", "regression", "geometric")
for value in self._assumption_values(self.average, X):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somewhere in this module or development/base.py, there's also checking for length of the parameters. do you think that should be in scope for this method? (doesn't have to be this PR)

"params, name",
[({"average": "bogus"}, "average"), ({"n_periods": -5}, "n_periods")],
)
def test_read_json_rejects_invalid_params(params, name):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think we'll eventually get to a sizable suite of I/O tests for development estimators? does it make sense to start a test_io.py?

Comment thread chainladder/development/base.py Outdated
Subclasses override it; the base class checks nothing.
"""

def _assumption_values(self, value, X: Triangle | None, axis: int = 3) -> list:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the way this is currently written probably belongs in common?


return cl.__dict__[json_dict["__class__"]]().set_params(**json_dict["params"])

def _estimator_from_json(class_name, params):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a from_dict is disguise?

Comment thread chainladder/methods/base.py Outdated
class MethodBase(BaseEstimator, EstimatorIO, Common):
_estimator_type = "chainladder"

def _validate_params(self, X: Triangle | None = None) -> None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can leave this for a future PR if there's nothing to implement

Comment thread chainladder/development/base.py Outdated


class DevelopmentBase(BaseEstimator, TransformerMixin, EstimatorIO, Common):
def _validate_params(self, X: Triangle | None = None) -> None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the pattern that I was obssessed with is validate_X in MethodBase, i.e. it doesn't even have to be explicitly called by the subclasses. but Development doesn't call super().fit(). So do we still need to declare an empty method here? sorry for the extra work

Average and SigmaInterpolation are module-level Literal aliases.
Development.__init__ annotates average and sigma_interpolation with them, and
_check_param_values reads the allowed values with typing.get_args, so the
annotation, the check and the error message share one list.

Developed with an AI assistant (Árbol, running Muse).
_assumption_values resolves an assumption the same way _validate_assumption
does and is not specific to development estimators, so it moves from
DevelopmentBase to Common. Behavior is unchanged.

Developed with an AI assistant (Árbol, running Muse).
No method estimator checks its parameters yet, so MethodBase does not declare
the hook. read_json now calls _check_param_values on any estimator that
defines it; scikit-learn 1.7.2 defines no method of that name.

Developed with an AI assistant (Árbol, running Muse).
Development.fit does not call DevelopmentBase.fit, so an empty hook on the base
class added nothing. The hook now lives only on Development, and its docstring
says how fit and read_json call it.

Developed with an AI assistant (Árbol, running Muse).
The three read_json tests added for casact#1569 move from test_development.py to a
new chainladder/development/tests/test_io.py, unchanged, so later I/O tests
for development estimators have one place to go.

Developed with an AI assistant (Árbol, running Muse).
@scks1

scks1 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Pushed, one commit per change so any can be dropped:

  • Name: the hook is now _check_param_values. scikit-learn 1.7.2 doesn't use it; _check_params was out because scikit-learn does.
  • n_periods=0 fits as -1 again, with a test that ldf_ and sigma_ match. I'll open an issue on deprecating it.
  • average: 1.0, 1, 2.0, 0.0 and 1.5 all raise a KeyError on main, so the docstring lists only the four names. You were right that it once worked: fit passed numbers through as weight exponents from v0.7.7 until 59b7f9e (first in v0.8.15).
  • read_json class checks: follow-up issue, as you said.

On the inline comments:

  • Literal: Average and SigmaInterpolation are module-level Literal aliases, used in __init__'s annotations and read with get_args in the check and its error message.
  • Lengths: on main nothing checks them. On raa (9 link ratios), 8 average values or 8 n_periods values fail with a numpy broadcast error, 10 n_periods or 12 drop_high values raise a KeyError, while 10 average values and 3 drop_high or drop_above values fit without a word. I think they belong in this method, since fit passes the triangle, but in a follow-up PR.
  • test_io.py: yes, started in chainladder/development/tests/test_io.py with the three read_json tests.
  • _assumption_values moved to Common, next to _validate_assumption.
  • from_dict: close to it. chainladder has no from_dict today. The helper rebuilds an estimator from the dict to_json writes, so it could become an EstimatorIO.from_dict classmethod that read_json calls. That adds public API, so I'd leave the private helper here and propose it separately if you want it.
  • MethodBase and DevelopmentBase: both empty hooks are gone. The check lives only on Development, and read_json calls it on any estimator that has it.

(This comment was written with Árbol, an AI assistant, under my direction.)

@henrydingliu
henrydingliu merged commit c113fd9 into casact:main Nov 16, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEAT] Check estimator parameter values before fit, including estimators loaded with read_json

2 participants