testMultiplePairwise
Multiple Comparison Statistical Test (Friedman + Pairwise Shaffer PostHoc)
Usage
testMultiplePairwise(e, output, rankOrder = "max", alpha = 0.05)
Arguments
 e
 Input experiment
 output
 The output for which the tet will be performed.
 rankOrder
 The optimization strategy, can be either maximizing "max" or minimizing "min" the target output variable.
 alpha
 The significance level used for the whole testing procedure.
Value

an testMultiplePairwise object
Description
This function perfoms a multiple comparison statistical test for the given experiment. First of all it performs a Friedman Test over all methods. In the case this test is rejected, meaning that significant differences are present among the methods a posthoc test is then executed. For that, each pair of methods are compared between each other, and finally a Shaffer familywise error correction is applied to the resulting pvalues.
Examples
# First we create an experiment from the wekaExperiment problem and prepare # it to apply the test: experiment < expCreate(wekaExperiment, name="test", parameter="fold") experiment < expReduce(experiment, "fold", mean) experiment < expSubset(experiment, list(featureSelection = "yes")) experiment < expInstantiate(experiment, removeUnary=TRUE) # Then we perform a testMultiplePairwise test procedure test < testMultiplePairwise(experiment, "accuracy", "max") summary(test) Friedman test, objetive maximize output variable accuracy. Obtained pvalue: 3.3072e04 Chi squared with 3 degrees of freedom statistic: 18.6000 Test rejected: pvalue: 3.3072e04 < 0.0500  Pairwise post hoc test for output accuracy Adjust method: Shaffer pvalues: J48 NaiveBayes OneR RandomForest 0.4061836 0.06483817 0.0002465873 J48 NA 0.40618358 0.0148973333 NaiveBayes NA NA 0.2690580653 