Package weka.classifiers

Class Evaluation

java.lang.Object
weka.classifiers.Evaluation
All Implemented Interfaces:
Serializable, RevisionHandler, Summarizable
Direct Known Subclasses:
AggregateableEvaluation
public class Evaluation extends Object implements Serializable, Summarizable, RevisionHandler
Class for evaluating machine learning models. Delegates to the actual implementation in weka.classifiers.evaluation.Evaluation.

-------------------------------------------------------------------

General options when evaluating a learning scheme from the command-line:

-t filename
Name of the file with the training data. (required)

-T filename
Name of the file with the test data. If missing a cross-validation is performed.

-c index
Index of the class attribute (1, 2, ...; default: last).

-x number
The number of folds for the cross-validation (default: 10).

-no-cv
No cross validation. If no test file is provided, no evaluation is done.

-split-percentage percentage
Sets the percentage for the train/test set split, e.g., 66.

-preserve-order
Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s').

-s seed
Random number seed for the cross-validation and percentage split (default: 1).

-m filename
The name of a file containing a cost matrix.

-l filename
Loads classifier from the given file. In case the filename ends with ".xml", a PMML file is loaded or, if that fails, options are loaded from XML.

-d filename
Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model.

-v
Outputs no statistics for the training data.

-o
Outputs statistics only, not the classifier.

-i
Outputs information-retrieval statistics per class.

-k
Outputs information-theoretic statistics.

-classifications "weka.classifiers.evaluation.output.prediction.AbstractOutput + options"
Uses the specified class for generating the classification output. E.g.: weka.classifiers.evaluation.output.prediction.PlainText or : weka.classifiers.evaluation.output.prediction.CSV -p range
Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired.

Deprecated: use "-classifications ..." instead.

-distribution
Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes).

Deprecated: use "-classifications ..." instead.

-no-predictions
Turns off the collection of predictions in order to conserve memory.

-r
Outputs cumulative margin distribution (and nothing else).

-g
Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else).

-xml filename | xml-string
Retrieves the options from the XML-data instead of the command line.

-threshold-file file
The file to save the threshold data to. The format is determined by the extensions, e.g., '.arff' for ARFF format or '.csv' for CSV.

-threshold-label label
The class label to determine the threshold data for (default is the first label)

-------------------------------------------------------------------

Example usage as the main of a classifier (called FunkyClassifier): 

 public static void main(String [] args) {
   runClassifier(new FunkyClassifier(), args);
 }

------------------------------------------------------------------

Example usage from within an application: 

 Instances trainInstances = ... instances got from somewhere
 Instances testInstances = ... instances got from somewhere
 Classifier scheme = ... scheme got from somewhere
 
 Evaluation evaluation = new Evaluation(trainInstances);
 evaluation.evaluateModel(scheme, testInstances);
 System.out.println(evaluation.toSummaryString());
Version:
$Revision: 14449 $
Author:
Eibe Frank (eibe@cs.waikato.ac.nz), Len Trigg (trigg@cs.waikato.ac.nz)
See Also:

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String[]
    BUILT_IN_EVAL_METRICS
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    Evaluation(Instances data)
     
    Evaluation(Instances data, CostMatrix costMatrix)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    double
    areaUnderPRC(int classIndex)
    Returns the area under precision-recall curve (AUPRC) for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method.
    double
    areaUnderROC(int classIndex)
    Returns the area under ROC for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method.
    final double
    avgCost()
    Gets the average cost, that is, total cost of misclassifications (incorrect plus unclassified) over the total number of instances.
    double[][]
    confusionMatrix()
    Returns a copy of the confusion matrix.
    final double
    correct()
    Gets the number of instances correctly classified (that is, for which a correct prediction was made).
    final double
    correlationCoefficient()
    Returns the correlation coefficient if the class is numeric.
    final double
    coverageOfTestCasesByPredictedRegions()
    Gets the coverage of the test cases by the predicted regions at the confidence level specified when evaluation was performed.
    void
    crossValidateModel(String classifierString, Instances data, int numFolds, String[] options, Random random)
    Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.
    void
    crossValidateModel(Classifier classifier, Instances data, int numFolds, Random random)
    Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.
    void
    crossValidateModel(Classifier classifier, Instances data, int numFolds, Random random, Object... forPredictionsPrinting)
    Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.
    boolean
    equals(Object obj)
    Tests whether the current evaluation object is equal to another evaluation object.
    final double
    errorRate()
    Returns the estimated error rate or the root mean squared error (if the class is numeric).
    static String
    evaluateModel(String classifierString, String[] options)
    Evaluates a classifier with the options given in an array of strings.
    static String
    evaluateModel(Classifier classifier, String[] options)
    Evaluates a classifier with the options given in an array of strings.
    double[]
    evaluateModel(Classifier classifier, Instances data, Object... forPredictionsPrinting)
    Evaluates the classifier on a given set of instances.
    double
    evaluateModelOnce(double[] dist, Instance instance)
    Evaluates the supplied distribution on a single instance.
    void
    evaluateModelOnce(double prediction, Instance instance)
    Evaluates the supplied prediction on a single instance.
    double
    evaluateModelOnce(Classifier classifier, Instance instance)
    Evaluates the classifier on a single instance.
    double
    evaluateModelOnceAndRecordPrediction(double[] dist, Instance instance)
    Evaluates the supplied distribution on a single instance.
    double
    evaluateModelOnceAndRecordPrediction(Classifier classifier, Instance instance)
    Evaluates the classifier on a single instance and records the prediction.
    double
    evaluationForSingleInstance(double[] dist, Instance instance, boolean storePredictions)
    Evaluates the supplied distribution on a single instance.
    double
    falseNegativeRate(int classIndex)
    Calculate the false negative rate with respect to a particular class.
    double
    falsePositiveRate(int classIndex)
    Calculate the false positive rate with respect to a particular class.
    double
    fMeasure(int classIndex)
    Calculate the F-Measure with respect to a particular class.
    static List<String>
    getAllEvaluationMetricNames()
    Utility method to get a list of the names of all built-in and plugin evaluation metrics
    double[]
    getClassPriors()
    Get the current weighted class counts.
    boolean
    getDiscardPredictions()
    Returns whether predictions are not recorded at all, in order to conserve memory.
    Instances
    getHeader()
    Returns the header of the underlying dataset.
    List<String>
    getMetricsToDisplay()
    Get a list of the names of metrics to have appear in the output The default is to display all built in metrics and plugin metrics that haven't been globally disabled.
    AbstractEvaluationMetric
    getPluginMetric(String name)
    Get the named plugin evaluation metric
    List<AbstractEvaluationMetric>
    getPluginMetrics()
    Returns the list of plugin metrics in use (or null if there are none)
    String
    getRevision()
    Returns the revision string.
    final double
    incorrect()
    Gets the number of instances incorrectly classified (that is, for which an incorrect prediction was made).
    final double
    kappa()
    Returns value of kappa statistic if class is nominal.
    final double
    KBInformation()
    Return the total Kononenko & Bratko Information score in bits.
    final double
    KBMeanInformation()
    Return the Kononenko & Bratko Information score in bits per instance.
    final double
    KBRelativeInformation()
    Return the Kononenko & Bratko Relative Information score.
    static void
    main(String[] args)
    A test method for this class.
    double
    matthewsCorrelationCoefficient(int classIndex)
    Calculates the matthews correlation coefficient (sometimes called phi coefficient) for the supplied class
    final double
    meanAbsoluteError()
    Returns the mean absolute error.
    final double
    meanPriorAbsoluteError()
    Returns the mean absolute error of the prior.
    double
    numFalseNegatives(int classIndex)
    Calculate number of false negatives with respect to a particular class.
    double
    numFalsePositives(int classIndex)
    Calculate number of false positives with respect to a particular class.
    final double
    numInstances()
    Gets the number of test instances that had a known class value (actually the sum of the weights of test instances with known class value).
    double
    numTrueNegatives(int classIndex)
    Calculate the number of true negatives with respect to a particular class.
    double
    numTruePositives(int classIndex)
    Calculate the number of true positives with respect to a particular class.
    final double
    pctCorrect()
    Gets the percentage of instances correctly classified (that is, for which a correct prediction was made).
    final double
    pctIncorrect()
    Gets the percentage of instances incorrectly classified (that is, for which an incorrect prediction was made).
    final double
    pctUnclassified()
    Gets the percentage of instances not classified (that is, for which no prediction was made by the classifier).
    double
    precision(int classIndex)
    Calculate the precision with respect to a particular class.
    ArrayList<Prediction>
    predictions()
    Returns the predictions that have been collected.
    final double
    priorEntropy()
    Calculate the entropy of the prior distribution.
    double
    recall(int classIndex)
    Calculate the recall with respect to a particular class.
    final double
    relativeAbsoluteError()
    Returns the relative absolute error.
    final double
    rootMeanPriorSquaredError()
    Returns the root mean prior squared error.
    final double
    rootMeanSquaredError()
    Returns the root mean squared error.
    final double
    rootRelativeSquaredError()
    Returns the root relative squared error if the class is numeric.
    void
    setDiscardPredictions(boolean value)
    Sets whether to discard predictions, ie, not storing them for future reference via predictions() method in order to conserve memory.
    void
    setMetricsToDisplay(List<String> display)
    Set a list of the names of metrics to have appear in the output.
    void
    setPriors(Instances train)
    Sets the class prior probabilities.
    final double
    SFEntropyGain()
    Returns the total SF, which is the null model entropy minus the scheme entropy.
    final double
    SFMeanEntropyGain()
    Returns the SF per instance, which is the null model entropy minus the scheme entropy, per instance.
    final double
    SFMeanPriorEntropy()
    Returns the entropy per instance for the null model.
    final double
    SFMeanSchemeEntropy()
    Returns the entropy per instance for the scheme.
    final double
    SFPriorEntropy()
    Returns the total entropy for the null model.
    final double
    SFSchemeEntropy()
    Returns the total entropy for the scheme.
    final double
    sizeOfPredictedRegions()
    Gets the average size of the predicted regions, relative to the range of the target in the training data, at the confidence level specified when evaluation was performed.
    String
    toClassDetailsString()
    Generates a breakdown of the accuracy for each class (with default title), incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure.
    String
    toClassDetailsString(String title)
    Generates a breakdown of the accuracy for each class, incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure.
    String
    toCumulativeMarginDistributionString()
    Output the cumulative margin distribution as a string suitable for input for gnuplot or similar package.
    void
    toggleEvalMetrics(List<String> metricsToToggle)
    Toggle the output of the metrics specified in the supplied list.
    String
    toMatrixString()
    Calls toMatrixString() with a default title.
    String
    toMatrixString(String title)
    Outputs the performance statistics as a classification confusion matrix.
    String
    toSummaryString()
    Calls toSummaryString() with no title and no complexity stats.
    String
    toSummaryString(boolean printComplexityStatistics)
    Calls toSummaryString() with a default title.
    String
    toSummaryString(String title, boolean printComplexityStatistics)
    Outputs the performance statistics in summary form.
    final double
    totalCost()
    Gets the total cost, that is, the cost of each prediction times the weight of the instance, summed over all instances.
    double
    trueNegativeRate(int classIndex)
    Calculate the true negative rate with respect to a particular class.
    double
    truePositiveRate(int classIndex)
    Calculate the true positive rate with respect to a particular class.
    final double
    unclassified()
    Gets the number of instances not classified (that is, for which no prediction was made by the classifier).
    double
    unweightedMacroFmeasure()
    Unweighted macro-averaged F-measure.
    double
    unweightedMicroFmeasure()
    Unweighted micro-averaged F-measure.
    void
    updatePriors(Instance instance)
    Updates the class prior probabilities or the mean respectively (when incrementally training).
    void
    useNoPriors()
    disables the use of priors, e.g., in case of de-serialized schemes that have no access to the original training set, but are evaluated on a set set.
    double
    weightedAreaUnderPRC()
    Calculates the weighted (by class size) AUPRC.
    double
    weightedAreaUnderROC()
    Calculates the weighted (by class size) AUC.
    double
    weightedFalseNegativeRate()
    Calculates the weighted (by class size) false negative rate.
    double
    weightedFalsePositiveRate()
    Calculates the weighted (by class size) false positive rate.
    double
    weightedFMeasure()
    Calculates the macro weighted (by class size) average F-Measure.
    double
    weightedMatthewsCorrelation()
    Calculates the weighted (by class size) matthews correlation coefficient.
    double
    weightedPrecision()
    Calculates the weighted (by class size) precision.
    double
    weightedRecall()
    Calculates the weighted (by class size) recall.
    double
    weightedTrueNegativeRate()
    Calculates the weighted (by class size) true negative rate.
    double
    weightedTruePositiveRate()
    Calculates the weighted (by class size) true positive rate.
    static String
    wekaStaticWrapper(Sourcable classifier, String className)
    Wraps a static classifier in enough source to test using the weka class libraries.

    Methods inherited from class java.lang.Object

    getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • BUILT_IN_EVAL_METRICS

      public static final String[] BUILT_IN_EVAL_METRICS

  • Constructor Details

  • Method Details

    • getAllEvaluationMetricNames

      public static List<String> getAllEvaluationMetricNames()
      Utility method to get a list of the names of all built-in and plugin evaluation metrics
      Returns:
      the complete list of available evaluation metrics

    • getHeader

      public Instances getHeader()
      Returns the header of the underlying dataset.
      Returns:
      the header information

    • getPluginMetrics

      public List<AbstractEvaluationMetric> getPluginMetrics()
      Returns the list of plugin metrics in use (or null if there are none)
      Returns:
      the list of plugin metrics

    • getPluginMetric

      public AbstractEvaluationMetric getPluginMetric(String name)
      Get the named plugin evaluation metric
      Parameters:
      name - the name of the metric (as returned by AbstractEvaluationMetric.getName()) or the fully qualified class name of the metric to find
      Returns:
      the metric or null if the metric is not in the list of plugin metrics

    • setMetricsToDisplay

      public void setMetricsToDisplay(List<String> display)
      Set a list of the names of metrics to have appear in the output. The default is to display all built in metrics and plugin metrics that haven't been globally disabled.
      Parameters:
      display - a list of metric names to have appear in the output

    • getMetricsToDisplay

      public List<String> getMetricsToDisplay()
      Get a list of the names of metrics to have appear in the output The default is to display all built in metrics and plugin metrics that haven't been globally disabled.
      Returns:
      a list of metric names to have appear in the output

    • toggleEvalMetrics

      public void toggleEvalMetrics(List<String> metricsToToggle)
      Toggle the output of the metrics specified in the supplied list.
      Parameters:
      metricsToToggle - a list of metrics to toggle

    • setDiscardPredictions

      public void setDiscardPredictions(boolean value)
      Sets whether to discard predictions, ie, not storing them for future reference via predictions() method in order to conserve memory.
      Parameters:
      value - true if to discard the predictions
      See Also:

    • getDiscardPredictions

      public boolean getDiscardPredictions()
      Returns whether predictions are not recorded at all, in order to conserve memory.
      Returns:
      true if predictions are not recorded
      See Also:

    • areaUnderROC

      public double areaUnderROC(int classIndex)
      Returns the area under ROC for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method. Returns Utils.missingValue() if the area is not available.
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the area under the ROC curve or not a number

    • weightedAreaUnderROC

      public double weightedAreaUnderROC()
      Calculates the weighted (by class size) AUC.
      Returns:
      the weighted AUC.

    • areaUnderPRC

      public double areaUnderPRC(int classIndex)
      Returns the area under precision-recall curve (AUPRC) for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method. Returns Utils.missingValue() if the area is not available.
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the area under the precision-recall curve or not a number

    • weightedAreaUnderPRC

      public double weightedAreaUnderPRC()
      Calculates the weighted (by class size) AUPRC.
      Returns:
      the weighted AUPRC.

    • confusionMatrix

      public double[][] confusionMatrix()
      Returns a copy of the confusion matrix.
      Returns:
      a copy of the confusion matrix as a two-dimensional array

    • crossValidateModel

      public void crossValidateModel(Classifier classifier, Instances data, int numFolds, Random random) throws Exception
      Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances. Now performs a deep copy of the classifier before each call to buildClassifier() (just in case the classifier is not initialized properly).
      Parameters:
      classifier - the classifier with any options set.
      data - the data on which the cross-validation is to be performed
      numFolds - the number of folds for the cross-validation
      random - random number generator for randomization
      Throws:
      Exception - if a classifier could not be generated successfully or the class is not defined

    • crossValidateModel

      public void crossValidateModel(Classifier classifier, Instances data, int numFolds, Random random, Object... forPredictionsPrinting) throws Exception
      Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances. Now performs a deep copy of the classifier before each call to buildClassifier() (just in case the classifier is not initialized properly).
      Parameters:
      classifier - the classifier with any options set.
      data - the data on which the cross-validation is to be performed
      numFolds - the number of folds for the cross-validation
      random - random number generator for randomization
      forPredictionsPrinting - varargs parameter that, if supplied, is expected to hold a weka.classifiers.evaluation.output.prediction.AbstractOutput object
      Throws:
      Exception - if a classifier could not be generated successfully or the class is not defined

    • crossValidateModel

      public void crossValidateModel(String classifierString, Instances data, int numFolds, String[] options, Random random) throws Exception
      Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.
      Parameters:
      classifierString - a string naming the class of the classifier
      data - the data on which the cross-validation is to be performed
      numFolds - the number of folds for the cross-validation
      options - the options to the classifier. Any options
      random - the random number generator for randomizing the data accepted by the classifier will be removed from this array.
      Throws:
      Exception - if a classifier could not be generated successfully or the class is not defined

    • evaluateModel

      public static String evaluateModel(String classifierString, String[] options) throws Exception
      Evaluates a classifier with the options given in an array of strings.

      Valid options are:

      -t filename
      Name of the file with the training data. (required)

      -T filename
      Name of the file with the test data. If missing a cross-validation is performed.

      -c index
      Index of the class attribute (1, 2, ...; default: last).

      -x number
      The number of folds for the cross-validation (default: 10).

      -no-cv
      No cross validation. If no test file is provided, no evaluation is done.

      -split-percentage percentage
      Sets the percentage for the train/test set split, e.g., 66.

      -preserve-order
      Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s').

      -s seed
      Random number seed for the cross-validation and percentage split (default: 1).

      -m filename
      The name of a file containing a cost matrix.

      -l filename
      Loads classifier from the given file. In case the filename ends with ".xml",a PMML file is loaded or, if that fails, options are loaded from XML.

      -d filename
      Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model.

      -v
      Outputs no statistics for the training data.

      -o
      Outputs statistics only, not the classifier.

      -i
      Outputs detailed information-retrieval statistics per class.

      -k
      Outputs information-theoretic statistics.

      -classifications "weka.classifiers.evaluation.output.prediction.AbstractOutput + options"
      Uses the specified class for generating the classification output. E.g.: weka.classifiers.evaluation.output.prediction.PlainText or : weka.classifiers.evaluation.output.prediction.CSV -p range
      Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired.

      Deprecated: use "-classifications ..." instead.

      -distribution
      Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes).

      Deprecated: use "-classifications ..." instead.

      -no-predictions
      Turns off the collection of predictions in order to conserve memory.

      -r
      Outputs cumulative margin distribution (and nothing else).

      -g
      Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else).

      -xml filename | xml-string
      Retrieves the options from the XML-data instead of the command line.

      -threshold-file file
      The file to save the threshold data to. The format is determined by the extensions, e.g., '.arff' for ARFF format or '.csv' for CSV.

      -threshold-label label
      The class label to determine the threshold data for (default is the first label)

      Parameters:
      classifierString - class of machine learning classifier as a string
      options - the array of string containing the options
      Returns:
      a string describing the results
      Throws:
      Exception - if model could not be evaluated successfully

    • evaluateModel

      public static String evaluateModel(Classifier classifier, String[] options) throws Exception
      Evaluates a classifier with the options given in an array of strings.

      Valid options are:

      -t name of training file
      Name of the file with the training data. (required)

      -T name of test file
      Name of the file with the test data. If missing a cross-validation is performed.

      -c class index
      Index of the class attribute (1, 2, ...; default: last).

      -x number of folds
      The number of folds for the cross-validation (default: 10).

      -no-cv
      No cross validation. If no test file is provided, no evaluation is done.

      -split-percentage percentage
      Sets the percentage for the train/test set split, e.g., 66.

      -preserve-order
      Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s').

      -s seed
      Random number seed for the cross-validation and percentage split (default: 1).

      -m file with cost matrix
      The name of a file containing a cost matrix.

      -l filename
      Loads classifier from the given file. In case the filename ends with ".xml",a PMML file is loaded or, if that fails, options are loaded from XML.

      -d filename
      Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model.

      -v
      Outputs no statistics for the training data.

      -o
      Outputs statistics only, not the classifier.

      -i
      Outputs detailed information-retrieval statistics per class.

      -k
      Outputs information-theoretic statistics.

      -classifications "weka.classifiers.evaluation.output.prediction.AbstractOutput + options"
      Uses the specified class for generating the classification output. E.g.: weka.classifiers.evaluation.output.prediction.PlainText or : weka.classifiers.evaluation.output.prediction.CSV -p range
      Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired.

      Deprecated: use "-classifications ..." instead.

      -distribution
      Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes).

      Deprecated: use "-classifications ..." instead.

      -no-predictions
      Turns off the collection of predictions in order to conserve memory.

      -r
      Outputs cumulative margin distribution (and nothing else).

      -g
      Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else).

      -xml filename | xml-string
      Retrieves the options from the XML-data instead of the command line.

      Parameters:
      classifier - machine learning classifier
      options - the array of string containing the options
      Returns:
      a string describing the results
      Throws:
      Exception - if model could not be evaluated successfully

    • evaluateModel

      public double[] evaluateModel(Classifier classifier, Instances data, Object... forPredictionsPrinting) throws Exception
      Evaluates the classifier on a given set of instances. Note that the data must have exactly the same format (e.g. order of attributes) as the data used to train the classifier! Otherwise the results will generally be meaningless.
      Parameters:
      classifier - machine learning classifier
      data - set of test instances for evaluation
      forPredictionsPrinting - varargs parameter that, if supplied, is expected to hold a weka.classifiers.evaluation.output.prediction.AbstractOutput object
      Returns:
      the predictions
      Throws:
      Exception - if model could not be evaluated successfully

    • evaluationForSingleInstance

      public double evaluationForSingleInstance(double[] dist, Instance instance, boolean storePredictions) throws Exception
      Evaluates the supplied distribution on a single instance.
      Parameters:
      dist - the supplied distribution
      instance - the test instance to be classified
      storePredictions - whether to store predictions for nominal classifier
      Returns:
      the prediction
      Throws:
      Exception - if model could not be evaluated successfully

    • evaluateModelOnceAndRecordPrediction

      public double evaluateModelOnceAndRecordPrediction(Classifier classifier, Instance instance) throws Exception
      Evaluates the classifier on a single instance and records the prediction.
      Parameters:
      classifier - machine learning classifier
      instance - the test instance to be classified
      Returns:
      the prediction made by the clasifier
      Throws:
      Exception - if model could not be evaluated successfully or the data contains string attributes

    • evaluateModelOnce

      public double evaluateModelOnce(Classifier classifier, Instance instance) throws Exception
      Evaluates the classifier on a single instance.
      Parameters:
      classifier - machine learning classifier
      instance - the test instance to be classified
      Returns:
      the prediction made by the clasifier
      Throws:
      Exception - if model could not be evaluated successfully or the data contains string attributes

    • evaluateModelOnce

      public double evaluateModelOnce(double[] dist, Instance instance) throws Exception
      Evaluates the supplied distribution on a single instance.
      Parameters:
      dist - the supplied distribution
      instance - the test instance to be classified
      Returns:
      the prediction
      Throws:
      Exception - if model could not be evaluated successfully

    • evaluateModelOnceAndRecordPrediction

      public double evaluateModelOnceAndRecordPrediction(double[] dist, Instance instance) throws Exception
      Evaluates the supplied distribution on a single instance.
      Parameters:
      dist - the supplied distribution
      instance - the test instance to be classified
      Returns:
      the prediction
      Throws:
      Exception - if model could not be evaluated successfully

    • evaluateModelOnce

      public void evaluateModelOnce(double prediction, Instance instance) throws Exception
      Evaluates the supplied prediction on a single instance.
      Parameters:
      prediction - the supplied prediction
      instance - the test instance to be classified
      Throws:
      Exception - if model could not be evaluated successfully

    • predictions

      public ArrayList<Prediction> predictions()
      Returns the predictions that have been collected.
      Returns:
      a reference to the FastVector containing the predictions that have been collected. This should be null if no predictions have been collected.

    • wekaStaticWrapper

      public static String wekaStaticWrapper(Sourcable classifier, String className) throws Exception
      Wraps a static classifier in enough source to test using the weka class libraries.
      Parameters:
      classifier - a Sourcable Classifier
      className - the name to give to the source code class
      Returns:
      the source for a static classifier that can be tested with weka libraries.
      Throws:
      Exception - if code-generation fails

    • numInstances

      public final double numInstances()
      Gets the number of test instances that had a known class value (actually the sum of the weights of test instances with known class value).
      Returns:
      the number of test instances with known class

    • coverageOfTestCasesByPredictedRegions

      public final double coverageOfTestCasesByPredictedRegions()
      Gets the coverage of the test cases by the predicted regions at the confidence level specified when evaluation was performed.
      Returns:
      the coverage of the test cases by the predicted regions

    • sizeOfPredictedRegions

      public final double sizeOfPredictedRegions()
      Gets the average size of the predicted regions, relative to the range of the target in the training data, at the confidence level specified when evaluation was performed.
      Returns:
      the average size of the predicted regions

    • incorrect

      public final double incorrect()
      Gets the number of instances incorrectly classified (that is, for which an incorrect prediction was made). (Actually the sum of the weights of these instances)
      Returns:
      the number of incorrectly classified instances

    • pctIncorrect

      public final double pctIncorrect()
      Gets the percentage of instances incorrectly classified (that is, for which an incorrect prediction was made).
      Returns:
      the percent of incorrectly classified instances (between 0 and 100)

    • totalCost

      public final double totalCost()
      Gets the total cost, that is, the cost of each prediction times the weight of the instance, summed over all instances.
      Returns:
      the total cost

    • avgCost

      public final double avgCost()
      Gets the average cost, that is, total cost of misclassifications (incorrect plus unclassified) over the total number of instances.
      Returns:
      the average cost.

    • correct

      public final double correct()
      Gets the number of instances correctly classified (that is, for which a correct prediction was made). (Actually the sum of the weights of these instances)
      Returns:
      the number of correctly classified instances

    • pctCorrect

      public final double pctCorrect()
      Gets the percentage of instances correctly classified (that is, for which a correct prediction was made).
      Returns:
      the percent of correctly classified instances (between 0 and 100)

    • unclassified

      public final double unclassified()
      Gets the number of instances not classified (that is, for which no prediction was made by the classifier). (Actually the sum of the weights of these instances)
      Returns:
      the number of unclassified instances

    • pctUnclassified

      public final double pctUnclassified()
      Gets the percentage of instances not classified (that is, for which no prediction was made by the classifier).
      Returns:
      the percent of unclassified instances (between 0 and 100)

    • errorRate

      public final double errorRate()
      Returns the estimated error rate or the root mean squared error (if the class is numeric). If a cost matrix was given this error rate gives the average cost.
      Returns:
      the estimated error rate (between 0 and 1, or between 0 and maximum cost)

    • kappa

      public final double kappa()
      Returns value of kappa statistic if class is nominal.
      Returns:
      the value of the kappa statistic

    • getRevision

      public String getRevision()
      Description copied from interface: RevisionHandler
      Returns the revision string.
      Specified by:
      getRevision in interface RevisionHandler
      Returns:
      the revision

    • correlationCoefficient

      public final double correlationCoefficient() throws Exception
      Returns the correlation coefficient if the class is numeric.
      Returns:
      the correlation coefficient
      Throws:
      Exception - if class is not numeric

    • meanAbsoluteError

      public final double meanAbsoluteError()
      Returns the mean absolute error. Refers to the error of the predicted values for numeric classes, and the error of the predicted probability distribution for nominal classes.
      Returns:
      the mean absolute error

    • meanPriorAbsoluteError

      public final double meanPriorAbsoluteError()
      Returns the mean absolute error of the prior.
      Returns:
      the mean absolute error

    • relativeAbsoluteError

      public final double relativeAbsoluteError() throws Exception
      Returns the relative absolute error.
      Returns:
      the relative absolute error
      Throws:
      Exception - if it can't be computed

    • rootMeanSquaredError

      public final double rootMeanSquaredError()
      Returns the root mean squared error.
      Returns:
      the root mean squared error

    • rootMeanPriorSquaredError

      public final double rootMeanPriorSquaredError()
      Returns the root mean prior squared error.
      Returns:
      the root mean prior squared error

    • rootRelativeSquaredError

      public final double rootRelativeSquaredError()
      Returns the root relative squared error if the class is numeric.
      Returns:
      the root relative squared error

    • priorEntropy

      public final double priorEntropy() throws Exception
      Calculate the entropy of the prior distribution.
      Returns:
      the entropy of the prior distribution
      Throws:
      Exception - if the class is not nominal

    • KBInformation

      public final double KBInformation() throws Exception
      Return the total Kononenko & Bratko Information score in bits.
      Returns:
      the K&B information score
      Throws:
      Exception - if the class is not nominal

    • KBMeanInformation

      public final double KBMeanInformation() throws Exception
      Return the Kononenko & Bratko Information score in bits per instance.
      Returns:
      the K&B information score
      Throws:
      Exception - if the class is not nominal

    • KBRelativeInformation

      public final double KBRelativeInformation() throws Exception
      Return the Kononenko & Bratko Relative Information score.
      Returns:
      the K&B relative information score
      Throws:
      Exception - if the class is not nominal

    • SFPriorEntropy

      public final double SFPriorEntropy()
      Returns the total entropy for the null model.
      Returns:
      the total null model entropy

    • SFMeanPriorEntropy

      public final double SFMeanPriorEntropy()
      Returns the entropy per instance for the null model.
      Returns:
      the null model entropy per instance

    • SFSchemeEntropy

      public final double SFSchemeEntropy()
      Returns the total entropy for the scheme.
      Returns:
      the total scheme entropy

    • SFMeanSchemeEntropy

      public final double SFMeanSchemeEntropy()
      Returns the entropy per instance for the scheme.
      Returns:
      the scheme entropy per instance

    • SFEntropyGain

      public final double SFEntropyGain()
      Returns the total SF, which is the null model entropy minus the scheme entropy.
      Returns:
      the total SF

    • SFMeanEntropyGain

      public final double SFMeanEntropyGain()
      Returns the SF per instance, which is the null model entropy minus the scheme entropy, per instance.
      Returns:
      the SF per instance

    • toCumulativeMarginDistributionString

      public String toCumulativeMarginDistributionString() throws Exception
      Output the cumulative margin distribution as a string suitable for input for gnuplot or similar package.
      Returns:
      the cumulative margin distribution
      Throws:
      Exception - if the class attribute is nominal

    • toSummaryString

      public String toSummaryString()
      Calls toSummaryString() with no title and no complexity stats.
      Specified by:
      toSummaryString in interface Summarizable
      Returns:
      a summary description of the classifier evaluation

    • toSummaryString

      public String toSummaryString(boolean printComplexityStatistics)
      Calls toSummaryString() with a default title.
      Parameters:
      printComplexityStatistics - if true, complexity statistics are returned as well
      Returns:
      the summary string

    • toSummaryString

      public String toSummaryString(String title, boolean printComplexityStatistics)
      Outputs the performance statistics in summary form. Lists number (and percentage) of instances classified correctly, incorrectly and unclassified. Outputs the total number of instances classified, and the number of instances (if any) that had no class value provided.
      Parameters:
      title - the title for the statistics
      printComplexityStatistics - if true, complexity statistics are returned as well
      Returns:
      the summary as a String

    • toMatrixString

      public String toMatrixString() throws Exception
      Calls toMatrixString() with a default title.
      Returns:
      the confusion matrix as a string
      Throws:
      Exception - if the class is numeric

    • toMatrixString

      public String toMatrixString(String title) throws Exception
      Outputs the performance statistics as a classification confusion matrix. For each class value, shows the distribution of predicted class values.
      Parameters:
      title - the title for the confusion matrix
      Returns:
      the confusion matrix as a String
      Throws:
      Exception - if the class is numeric

    • toClassDetailsString

      public String toClassDetailsString() throws Exception
      Generates a breakdown of the accuracy for each class (with default title), incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure. Should be useful for ROC curves, recall/precision curves.
      Returns:
      the statistics presented as a string
      Throws:
      Exception - if class is not nominal

    • toClassDetailsString

      public String toClassDetailsString(String title) throws Exception
      Generates a breakdown of the accuracy for each class, incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure. Should be useful for ROC curves, recall/precision curves.
      Parameters:
      title - the title to prepend the stats string with
      Returns:
      the statistics presented as a string
      Throws:
      Exception - if class is not nominal

    • numTruePositives

      public double numTruePositives(int classIndex)
      Calculate the number of true positives with respect to a particular class. This is defined as 

       correctly classified positives
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the true positive rate

    • truePositiveRate

      public double truePositiveRate(int classIndex)
      Calculate the true positive rate with respect to a particular class. This is defined as 

       correctly classified positives
 ------------------------------
       total positives
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the true positive rate

    • weightedTruePositiveRate

      public double weightedTruePositiveRate()
      Calculates the weighted (by class size) true positive rate.
      Returns:
      the weighted true positive rate.

    • numTrueNegatives

      public double numTrueNegatives(int classIndex)
      Calculate the number of true negatives with respect to a particular class. This is defined as 

       correctly classified negatives
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the true positive rate

    • trueNegativeRate

      public double trueNegativeRate(int classIndex)
      Calculate the true negative rate with respect to a particular class. This is defined as 

       correctly classified negatives
 ------------------------------
       total negatives
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the true positive rate

    • weightedTrueNegativeRate

      public double weightedTrueNegativeRate()
      Calculates the weighted (by class size) true negative rate.
      Returns:
      the weighted true negative rate.

    • numFalsePositives

      public double numFalsePositives(int classIndex)
      Calculate number of false positives with respect to a particular class. This is defined as 

       incorrectly classified negatives
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the false positive rate

    • falsePositiveRate

      public double falsePositiveRate(int classIndex)
      Calculate the false positive rate with respect to a particular class. This is defined as 

       incorrectly classified negatives
 --------------------------------
        total negatives
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the false positive rate

    • weightedFalsePositiveRate

      public double weightedFalsePositiveRate()
      Calculates the weighted (by class size) false positive rate.
      Returns:
      the weighted false positive rate.

    • numFalseNegatives

      public double numFalseNegatives(int classIndex)
      Calculate number of false negatives with respect to a particular class. This is defined as 

       incorrectly classified positives
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the false positive rate

    • falseNegativeRate

      public double falseNegativeRate(int classIndex)
      Calculate the false negative rate with respect to a particular class. This is defined as 

       incorrectly classified positives
 --------------------------------
        total positives
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the false positive rate

    • weightedFalseNegativeRate

      public double weightedFalseNegativeRate()
      Calculates the weighted (by class size) false negative rate.
      Returns:
      the weighted false negative rate.

    • matthewsCorrelationCoefficient

      public double matthewsCorrelationCoefficient(int classIndex)
      Calculates the matthews correlation coefficient (sometimes called phi coefficient) for the supplied class
      Parameters:
      classIndex - the index of the class to compute the matthews correlation coefficient for
      Returns:
      the mathews correlation coefficient

    • weightedMatthewsCorrelation

      public double weightedMatthewsCorrelation()
      Calculates the weighted (by class size) matthews correlation coefficient.
      Returns:
      the weighted matthews correlation coefficient.

    • recall

      public double recall(int classIndex)
      Calculate the recall with respect to a particular class. This is defined as 

       correctly classified positives
 ------------------------------
       total positives

      (Which is also the same as the truePositiveRate.)

      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the recall

    • weightedRecall

      public double weightedRecall()
      Calculates the weighted (by class size) recall.
      Returns:
      the weighted recall.

    • precision

      public double precision(int classIndex)
      Calculate the precision with respect to a particular class. This is defined as 

       correctly classified positives
 ------------------------------
  total predicted as positive
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the precision

    • weightedPrecision

      public double weightedPrecision()
      Calculates the weighted (by class size) precision.
      Returns:
      the weighted precision.

    • fMeasure

      public double fMeasure(int classIndex)
      Calculate the F-Measure with respect to a particular class. This is defined as 

       2 * recall * precision
 ----------------------
   recall + precision
      Parameters:
      classIndex - the index of the class to consider as "positive"
      Returns:
      the F-Measure

    • weightedFMeasure

      public double weightedFMeasure()
      Calculates the macro weighted (by class size) average F-Measure.
      Returns:
      the weighted F-Measure.

    • unweightedMacroFmeasure

      public double unweightedMacroFmeasure()
      Unweighted macro-averaged F-measure. If some classes not present in the test set, they're just skipped (since recall is undefined there anyway) .
      Returns:
      unweighted macro-averaged F-measure.

    • unweightedMicroFmeasure

      public double unweightedMicroFmeasure()
      Unweighted micro-averaged F-measure. If some classes not present in the test set, they have no effect. Note: if the test set is *single-label*, then this is the same as accuracy.
      Returns:
      unweighted micro-averaged F-measure.

    • setPriors

      public void setPriors(Instances train) throws Exception
      Sets the class prior probabilities.
      Parameters:
      train - the training instances used to determine the prior probabilities
      Throws:
      Exception - if the class attribute of the instances is not set

    • getClassPriors

      public double[] getClassPriors()
      Get the current weighted class counts.
      Returns:
      the weighted class counts

    • updatePriors

      public void updatePriors(Instance instance) throws Exception
      Updates the class prior probabilities or the mean respectively (when incrementally training).
      Parameters:
      instance - the new training instance seen
      Throws:
      Exception - if the class of the instance is not set

    • useNoPriors

      public void useNoPriors()
      disables the use of priors, e.g., in case of de-serialized schemes that have no access to the original training set, but are evaluated on a set set.

    • equals

      public boolean equals(Object obj)
      Tests whether the current evaluation object is equal to another evaluation object.
      Overrides:
      equals in class Object
      Parameters:
      obj - the object to compare against
      Returns:
      true if the two objects are equal

    • main

      public static void main(String[] args)
      A test method for this class. Just extracts the first command line argument as a classifier class name and calls evaluateModel.
      Parameters:
      args - an array of command line arguments, the first of which must be the class name of a classifier.