Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#638 closed defect (fixed)

overflow in Fisher

Reported by: Peter Owned by: Peter
Priority: major Milestone: yat 0.6.3
Component: statistics Version: 0.6.2
Keywords: Cc:


I get some strange results using the Fisher class with large numbers.

The problem is code like


where a,b,c and d are all unsigned int. In my case a is ~60,000,000 and d is ~1,000 so 'a*d' is ~60,000,000,000 which is larger than the upper limit (232) for unsigned int on my machine. We should probably do the calculation with floating points to avoid such problems.

Change History (4)

comment:1 Changed 11 years ago by Peter

Status: newassigned

comment:2 Changed 11 years ago by Peter

(In [2314]) refs #638. adding a test for Fisher with large values

comment:3 Changed 11 years ago by Peter

Resolution: fixed
Status: assignedclosed

(In [2315]) prefer casting to double first and then multiply rather than the opposite. fixes #638

comment:4 Changed 11 years ago by Peter

(In [2319]) fixed more problems with overflow in Fisher. refs #638

Note: See TracTickets for help on using tickets.