Package weka.knowledgeflow.steps

Class MakeResourceIntensive

java.lang.Object
weka.knowledgeflow.steps.BaseStep
weka.knowledgeflow.steps.MakeResourceIntensive
All Implemented Interfaces:
Serializable, BaseStepExtender, Step
@KFStep(name="MakeResourceIntensive", category="Flow", toolTipText="Makes downstream connected steps resource intensive (or not). This shifts processing of such steps between the main step executor<br>service and the high resource executor service or vice versa.", iconPath="weka/gui/knowledgeflow/icons/DiamondPlain.gif") public class MakeResourceIntensive extends BaseStep
A Step that makes downstream steps that are directly connected to this step resource intensive (or not). This overrides whatever the downstream step may (or may not) have declared in it's KFStep class annotation with regards to whether it is resource intensive (cpu or memory). The Knowledge Flow execution environment uses two executor services - a primary one to execute batch processing for steps; and a secondary one for executing StepTasks (which are assumed to be resource intensive by default) or for executing batch processing for a Step when it declares itself to be resource intensive. This secondary executor service uses a limited (typically <= num cpu cores) number of threads. Steps that involve potentially intensive (cpu/memory) processing should declare themselves resource intensive so that less taxing steps (and the UI) get cpu cycles. E.g. the Classifier Step is resource intensive so that processing cross-validation folds in parallel for a large data set or computationally intensive classifier does not blow out memory or bog the system down.
Version:
$Revision: $
Author:
Mark Hall (mhall{[at]}pentaho{[dot]}com)
See Also:

  • Constructor Details

    • MakeResourceIntensive

      public MakeResourceIntensive()

  • Method Details

    • setMakeResourceIntensive

      @OptionMetadata(displayName="Make downstream step(s) high resource", description="<html>Makes downstream connected steps resource intensive (or not)<br>This shifts processing of such steps between the main step executor service and the high resource executor service or vice versa.</html>") public void setMakeResourceIntensive(boolean resourceIntensive)
      Set whether downstream steps are to be made resource intensive or not
      Parameters:
      resourceIntensive - true if the downstream connected steps are to be made resource intensive

    • getMakeResourceIntensive

      public boolean getMakeResourceIntensive()
      Get whether downstream steps are to be made resource intensive
      Returns:
      true if downstream connected steps are to be made resource intensive

    • stepInit

      public void stepInit() throws WekaException
      Initialize the step
      Throws:
      WekaException - if a problem occurs

    • getIncomingConnectionTypes

      public List<String> getIncomingConnectionTypes()
      Get a list of incoming connection types that this step can accept. Ideally (and if appropriate), this should take into account the state of the step and any existing incoming connections. E.g. a step might be able to accept one (and only one) incoming batch data connection.
      Returns:
      a list of incoming connections that this step can accept given its current state

    • getOutgoingConnectionTypes

      public List<String> getOutgoingConnectionTypes()
      Get a list of outgoing connection types that this step can produce. Ideally (and if appropriate), this should take into account the state of the step and the incoming connections. E.g. depending on what incoming connection is present, a step might be able to produce a trainingSet output, a testSet output or neither, but not both.
      Returns:
      a list of outgoing connections that this step can produce

    • processIncoming

      public void processIncoming(Data data) throws WekaException
      Process incoming data
      Specified by:
      processIncoming in interface BaseStepExtender
      Specified by:
      processIncoming in interface Step
      Overrides:
      processIncoming in class BaseStep
      Parameters:
      data - the data to process
      Throws:
      WekaException