kolmogov_smirnov_test fails on Mac OS 10.4

[Peter]

Gives this error message

Error: unexpected large deviation between p_values
permutation p-value: 0.1544
analytical approximation: 0.210552
expected deviation to be smaller than 0.0181978
Test failed.

[Peter]

(In [1608]) refs #457 - correcting error margin (not fixing the problem though)

[Peter]

On my Linux machine I get this result:

permutation p-value: 0.2097
analytical approximation: 0.210552

so there is something going on in the permutation test.

[Peter]

(In [1612]) fixing bug in KolmogorovSmirnov::reset. May refs #457 but not in an obvious way.

[Jari Häkkinen]

On my MacBook? Pro running latest OSX 10.5 I get the same as the description

Error: unexpected large deviation between p_values
permutation p-value: 0.1544
analytical approximation: 0.210552
expected deviation to be smaller than 0.0181978
testing reset
Test failed.

[Peter]

(In [1613]) refs #457 - turning off the failing test temporarily until the issue is resolved (make check should not fail in repo)

[Peter]

A hot candidate here would be the random generator. I tried with different seeds and the permutation p-value fluctuated between 0.15 and 0.165.

This implies that the problem is not as simple as being unlucky. If the problem lies in RNG, it more severe than that.

[Peter]

I tried replacing random::random_shuffle with std::random_shuffle, i.e., using the default random generator rather than gsl_rng through RNG class. The permutation is still around 0.16. I guess this means that the problem is not really related to generation of random numbers.

[Peter]

(In [1617]) refs #457 - adding a test with two samples and avoid shadowing variables

[Peter]

If I modify the line in KolmogorovSmirnov.cc

if (ks.score()>=score())

to

if (ks.score()>=score()-0.0001)

the problem disappears.

Smells some rounding error, but need to look into what it actually means.

[Peter]

The problem arises in how the KS score is calculated:

a/b - c/d

for example in the test, we have 20.0/100.0 - 5.0/100.0 which in theory becomes 0.15 but due to rounding errors in the division the score might get slightly smaller or larger. I think it should be enough to allow for 10*numeric_limits<double>().epsilon() error.

[Peter]

(In [1626]) closes #457 - kolmogorov_smirnov_test fails on Mac OS 10.4