Opened 12 years ago

Closed 12 years ago

Last modified 12 years ago

#638 closed defect (fixed)

overflow in Fisher

Reported by: Peter Owned by: Peter
Priority: major Milestone: yat 0.6.3
Component: statistics Version: 0.6.2
Keywords: Cc:


I get some strange results using the Fisher class with large numbers.

The problem is code like


where a,b,c and d are all unsigned int. In my case a is ~60,000,000 and d is ~1,000 so 'a*d' is ~60,000,000,000 which is larger than the upper limit (232) for unsigned int on my machine. We should probably do the calculation with floating points to avoid such problems.

Change History (4)

comment:1 Changed 12 years ago by Peter

Status: newassigned

comment:2 Changed 12 years ago by Peter

(In [2314]) refs #638. adding a test for Fisher with large values

comment:3 Changed 12 years ago by Peter

Resolution: fixed
Status: assignedclosed

(In [2315]) prefer casting to double first and then multiply rather than the opposite. fixes #638

comment:4 Changed 12 years ago by Peter

(In [2319]) fixed more problems with overflow in Fisher. refs #638

Note: See TracTickets for help on using tickets.