-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ENH] Tweedie Distribution #428
base: main
Are you sure you want to change the base?
Conversation
@fkiraly I have tried to implement pdf of the compound poisson-gamma case when power is in (1,2). But I am still unsure what can be done to handle the case when power is in (2,3) or when it is p<0. Need some thought on whether this would be a good idea to distinguish them based on the power parameters in the way I have done or is there any better way. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm, this is incomplete - only covers p between 1 and 2, no?
May I suggest calling this CompoundPoissonGamma
instead, and later using it as one of the Tweedie ED components in a delegator, as suggested here?
#423
Yes I will be splitting it into Compound poisson gamma and then calling it in the tweedie distribution. However I still face the problem of integrating the Also facing the same issue for finding the |
Brute force integration and such are typically last resort defaults. I would suggest to check the approximation formulae along the lines of those linked in this comment: #429 (comment) |
it's implicit in the constants |
I see then how do I generate these crj from the power parameter alone. And also any thoughts on how this infinite series summation can be made efficient. Is there any way this expression will simplify to something without a summation? |
I think you need to map them, following through the correspondence between Tweedie and CPG parameters. I would recommend you start implementing CPG in pure CPG parameters, and the mapping is done in the delegator. |
I don't think so - the usual procedure is to prescribe an epsilon and abort once the change becomes less than that epsilon - this is sound as long as you know the magnitudes are decreasing, and you have some theory that links magnitude of individual terms to magnitude of the total remainder. |
Can we not just keep the pdf = poisson.pdf(x) * gamma.pdf(x) where gamma will have the For now i have implemented the |
I am not able to get the |
@fkiraly I have implemented the cdf as per the Witheres Nadaraja paper. But I think there is some small issue as the function is not converging to a sum within [0,1] when the iterations( |
Sure! What would help me if you write a snippet of code, either assuming the branch or just numpy imports, that computes the sum. |
I have no context here, what is the |
Just following through the references, my interpretation: the To avoid confusion, there are g- and h-polynomials, and also f-polynomials, and Withers claims there is a mistake in some original explicit formulae for the low order g-polynomials in the papers. The trail of references is:
This seems to become the definition of a rabbit hole... I would suggest trying to implement fast numpy based functions for the various polynomials or expressions, in cases they cannot be obtained from For expressions such as the standard normal |
I am not quite sure I see an implementation of I am not able to find any pre-existing method in scipy to interface the computation of I've also tried using |
This is the code that was used to test
Output
Regarding the implementation specifics is present in the comment. |
ok, I see, this is typical for "implementation from paper" - it is not unusual that it looks like things have been correctly implemented, but they then do not behave as expected. My primary suggestion for continuing with this is to proceed with "bottom up validation", i.e., first test correctness of the lower down functions like crj and gr. What I would do, in sequence:
Debug wherever manually computed values do not agree with machine computed ones. If we did this, and pdf, pmf values fail to match the constraints still, then there is supsicion that there is an error in the math, in the paper. But until we can conclude this, we need to do the tedious manual verification above. Of course we can decide to work on sth else if this feels like too much of a rabbit hole. |
This comment is the output for the final output of cdf. Now for the individual values itself for example
when r=5 and j=3 so the above array contains and the value
Now doing this for all values of j from 1 to r where r=5 gives
Now these are smaller for small values of r but when r is larger the hermite polynomial also becomes bigger
and the value of this iteration given by
and doing this for all values of
|
could you kindly do this bottom-up, with some values calculated manually, and saying explicitly which object in the papers this corresponds to? E.g., sth like manual computation for
|
So the whole stack output for say
The eqn 2) gives the Hermite polynomial
The eqn 3) computes the
Here |
can we schedule a walkthrough for the Monday tech session, or whichever next is convenient? I think this needs discussion with questions and explanations. |
Yeah I think Monday tech session should work as per my current schedule. |
So I have implemented the permutation of the partitions that we spotted. The cdf values did come down by a lot to the order 10^0. But it still has some values outside the [0,1] range. Not sure how to tackle that. The current cdf output
The change handles it this way
where the array is the partition array when r=3 and j=2. So sum is 3 and number of terms that add upto 3 is 2. The code doing that is
|
hm, this seems correct, if a bit inefficient (and likely subject to combinatorial explosion for larger The next oddity I am spotting are the negative values. I am suspecting pre-factors in capital and lower case phi - can you point me to the exact python code that you have for these two? |
Which pre-factors in capital are we talking about. The lower case
|
The capital phi represents the normal cdf but I have not used it anywhere in my code since I am implementing the equation (3.8) which is the 4th equation here in the paper for the cdf which is an approximation of equation (3.5). |
Slightly confused - 3.8 is an equation for the pdf, not cdf. How are you using it in implementing the cdf? |
Waitasec I am now noticing this:
Equation 3.8 is the equation for pdf, not cdf! The "dot" above the Equation 3.5 is the equation for cdf. |
Ahhh I see thats a blunder. But all the functions created are still useful nevertheless. I shall make the change and commit it. |
So now these are the outputs
for the following code
|
The reason why the value is going a little out of bound for x=3,x=4 etc maybe due to the minor division errors that keep on aggregating while calculating . It is outputting cdf=1.0 for large x like 20,100,200 etc. |
Nice! My gut feeling says that we have caught all logical inaccuracies, and the remainder is - as you say - probably indeed numerical issues. Specifically, I think the large coefficients are contributive here, the naive calculation will lead to severe error propagation. I would suggest to add tests for the individual functions in play. The typical next steps would be to use numerical "tricks of the trade" to bring down the error, and at least ensure properties such as monotonicity for the cdf. |
it seems to have problems in case 3 as well. The oscillations in the 1st case do not look like numerical artefacts, but still might be - we should investigate. Could you print the coefficients maybe, to see how large they get? And if you do any divisions, how large numerators/denominators are? |
does this equation converge to something monotonic if you take enough terms? |
@fkiraly I have followed through the references as mentioned in this <a href="https://github.com/sktime/skpro/pull/428#issuecomment-2252710366>comment and I dont think I see anything about the I could only find some formula for first 4 Could you please have a look at it again and give a more explicit form for the |
do you have a link to Fisher/Cornish 1960, or is that a physical library-only reference? |
The screenshot of the first 4 Regarding the Cornish Fisher 1960 expansions it can be found here. I think it can be accessed only through an institute account. |
that sounds like the wrong paper? Similar name, but the author is someone named Ulyanov from Moscov State U. |
Yea I wasn't able to find any original one as such. |
Hm, feels like we are a little stuck then. Should we park this? Would be a bit of a shame. |
Reference Issues/PRs
related to #423
What does this implement/fix? Explain your changes.
Implements a Tweedie distribution in skpro by working the compound poisson gamma case when power parameter 1<p<2. Calls the other pre-existing distributions in other interger cases
Does your contribution introduce a new dependency? If yes, which one?
No
What should a reviewer concentrate their feedback on?
If this idea would be suitable or is there something that might hinder the further extension as
skpro
does not handle case when 2<power<3.Did you add any tests for the change?
no
PR checklist
For all contributions
How to: add yourself to the all-contributors file in the
skpro
root directory (not theCONTRIBUTORS.md
). Common badges:code
- fixing a bug, or adding code logic.doc
- writing or improving documentation or docstrings.bug
- reporting or diagnosing a bug (get this pluscode
if you also fixed the bug in the PR).maintenance
- CI, test framework, release.See here for full badge reference
For new estimators
docs/source/api_reference/taskname.rst
, follow the pattern.Examples
section.python_dependencies
tag and ensureddependency isolation, see the estimator dependencies guide.