Opened 12 years ago
Closed 12 years ago
#457 closed defect (fixed)
kolmogov_smirnov_test fails on Mac OS 10.4
Reported by: | Peter | Owned by: | Peter |
---|---|---|---|
Priority: | major | Milestone: | yat 0.5 |
Component: | test | Version: | trunk |
Keywords: | Cc: |
Description
Gives this error message
Error: unexpected large deviation between p_values permutation p-value: 0.1544 analytical approximation: 0.210552 expected deviation to be smaller than 0.0181978 Test failed.
Change History (12)
comment:1 Changed 12 years ago by
comment:2 Changed 12 years ago by
Owner: | changed from Jari Häkkinen to Peter |
---|---|
Status: | new → assigned |
comment:3 Changed 12 years ago by
On my Linux machine I get this result:
permutation p-value: 0.2097 analytical approximation: 0.210552
so there is something going on in the permutation test.
comment:4 Changed 12 years ago by
comment:5 Changed 12 years ago by
On my MacBook? Pro running latest OSX 10.5 I get the same as the description
Error: unexpected large deviation between p_values permutation p-value: 0.1544 analytical approximation: 0.210552 expected deviation to be smaller than 0.0181978 testing reset Test failed.
comment:6 Changed 12 years ago by
comment:7 Changed 12 years ago by
A hot candidate here would be the random generator. I tried with different seeds and the permutation p-value fluctuated between 0.15 and 0.165.
This implies that the problem is not as simple as being unlucky. If the problem lies in RNG, it more severe than that.
comment:8 Changed 12 years ago by
I tried replacing random::random_shuffle
with std::random_shuffle
, i.e., using the default random generator rather than gsl_rng through RNG class. The permutation is still around 0.16. I guess this means that the problem is not really related to generation of random numbers.
comment:9 Changed 12 years ago by
comment:10 Changed 12 years ago by
If I modify the line in KolmogorovSmirnov.cc
if (ks.score()>=score())
to
if (ks.score()>=score()-0.0001)
the problem disappears.
Smells some rounding error, but need to look into what it actually means.
comment:11 Changed 12 years ago by
The problem arises in how the KS score is calculated:
a/b - c/d
for example in the test, we have 20.0/100.0 - 5.0/100.0
which in theory becomes 0.15 but due to rounding errors in the division the score might get slightly smaller or larger. I think it should be enough to allow for 10*numeric_limits<double>().epsilon()
error.
comment:12 Changed 12 years ago by
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
(In [1608]) refs #457 - correcting error margin (not fixing the problem though)