Opened 13 years ago

Closed 13 years ago

#457 closed defect (fixed)

kolmogov_smirnov_test fails on Mac OS 10.4

Reported by: Peter Owned by: Peter
Priority: major Milestone: yat 0.5
Component: test Version: trunk
Keywords: Cc:

Description

Gives this error message

Error: unexpected large deviation between p_values
permutation p-value: 0.1544
analytical approximation: 0.210552
expected deviation to be smaller than 0.0181978
Test failed.

Change History (12)

comment:1 Changed 13 years ago by Peter

(In [1608]) refs #457 - correcting error margin (not fixing the problem though)

comment:2 Changed 13 years ago by Peter

Owner: changed from Jari Häkkinen to Peter
Status: newassigned

comment:3 Changed 13 years ago by Peter

On my Linux machine I get this result:

permutation p-value: 0.2097
analytical approximation: 0.210552

so there is something going on in the permutation test.

comment:4 Changed 13 years ago by Peter

(In [1612]) fixing bug in KolmogorovSmirnov::reset. May refs #457 but not in an obvious way.

comment:5 Changed 13 years ago by Jari Häkkinen

On my MacBook? Pro running latest OSX 10.5 I get the same as the description

Error: unexpected large deviation between p_values
permutation p-value: 0.1544
analytical approximation: 0.210552
expected deviation to be smaller than 0.0181978
testing reset
Test failed.

comment:6 Changed 13 years ago by Peter

(In [1613]) refs #457 - turning off the failing test temporarily until the issue is resolved (make check should not fail in repo)

comment:7 Changed 13 years ago by Peter

A hot candidate here would be the random generator. I tried with different seeds and the permutation p-value fluctuated between 0.15 and 0.165.

This implies that the problem is not as simple as being unlucky. If the problem lies in RNG, it more severe than that.

comment:8 Changed 13 years ago by Peter

I tried replacing random::random_shuffle with std::random_shuffle, i.e., using the default random generator rather than gsl_rng through RNG class. The permutation is still around 0.16. I guess this means that the problem is not really related to generation of random numbers.

comment:9 Changed 13 years ago by Peter

(In [1617]) refs #457 - adding a test with two samples and avoid shadowing variables

comment:10 Changed 13 years ago by Peter

If I modify the line in KolmogorovSmirnov.cc

if (ks.score()>=score())

to

if (ks.score()>=score()-0.0001)

the problem disappears.

Smells some rounding error, but need to look into what it actually means.

comment:11 Changed 13 years ago by Peter

The problem arises in how the KS score is calculated:

a/b - c/d

for example in the test, we have 20.0/100.0 - 5.0/100.0 which in theory becomes 0.15 but due to rounding errors in the division the score might get slightly smaller or larger. I think it should be enough to allow for 10*numeric_limits<double>().epsilon() error.

comment:12 Changed 13 years ago by Peter

Resolution: fixed
Status: assignedclosed

(In [1626]) closes #457 - kolmogorov_smirnov_test fails on Mac OS 10.4

Note: See TracTickets for help on using tickets.