Opened 15 years ago

Closed 10 years ago

Last modified 10 years ago

#211 closed defect (fixed)

weighted ROC p-value does not equal permutation test p-value?

Reported by: Peter Owned by: Peter
Priority: minor Milestone: yat 0.9
Component: statistics Version: trunk
Keywords: Cc:


The title might not be true but it is not straight forward to work out.

When having binary weights i.e. all non-zero weight are equal, then this should not be a problem. Work out what happens when this is not the case? Calculating the p-value exactly and approximately are two different stories, and need to be examined separately.

In both cases examine whether p-value is equal to permutation p-value. In the approximative case it is of course not identical but check if some correction should be done. In the exact case check if the p-value equals permutation p-value, if not try to modify implementation.

Change History (7)

comment:1 Changed 11 years ago by Peter

Milestone: yat 0.x+yat 0.9
Status: newassigned

comment:2 Changed 10 years ago by Peter

Just to clarify: when we say permute in a weighted context, it means weight and class label stay together and data values are permuted. This imnplies that sum w over one class is const wrt permutations and that missing values (w=0) are ignored.

comment:3 Changed 10 years ago by Peter

(In [2708]) tests for weighted ROC p-value. refs #211

comment:4 Changed 10 years ago by Peter

(In [2709]) refs #211. Changed how number of samples, n, is calculated in large sample approximation p-value calculation. Before the sum_w was used and now sum(w)*sum(w)/sum(ww) is used.

comment:5 Changed 10 years ago by Peter

(In [2712]) correct test. refs #211

comment:6 Changed 10 years ago by Peter

Resolution: fixed
Status: assignedclosed

(In [2718]) implement ROC p-value handling weights and ties. closes #211

comment:7 Changed 10 years ago by Peter

(In [2722]) fix implementation of ROC::remove. refs #211

Note: See TracTickets for help on using tickets.