Filter

../../_images/filterNode.png

The Clario Filter node allows you to perform many filtering tasks on an input data stream using one or more rule expressions.

Some common tasks Filter can handle are:

  • Keep/drop rows
  • Split a rowset into one or more streams
  • Using a set of rules to apply a key code to each row
  • Doing all of the above at the same time

Concepts

Each incoming row is tested against the set of rules in order:

  • If a row tests true against a rule, it is sent to both the “Filtered Rows” and rule-specific output streams
  • If a row does not test true against any rule, it is sent to the “Not Filtered Rows” output stream
  • If a row errors while testing against a rule, it is sent to the “Error Rows” output stream

Rules can be built by using the Code Editor or by using the Expression Builder and its intrinsic Functions, Operators, and Constants.

Configuration

The Filter node has only one configuration tab:

../../_images/filter_configurationTab.png

Filter requires an incoming stream to be connected and at least one rule expression to be defined.

Manage Rules

To create a new rule, click [New] to open the Expression Builder or click [Edit Code] to open the Code Editor.

Select a rule in the list to display its expression code.

To edit a rule, double-click on the rule to open the Expression Builder or click [Edit Code] to open the Code Editor.

Rules can be removed by selecting one or more from the the list and pressing [Delete].

Drag and drop items in the list to reorder them.

Rule Labels Each rule has a user-definable label. It is good practice to label each rule clearly and succinctly because the label is used for:

  • Output Stream naming
  • Results tables
  • Optionally “key coding” (see “Create Filter Attribute” section)

Unique ClarioScript/Expression Builder Features

In ClarioScript, Filter rules are written in the form :FILTER “filter label” = [boolean filter expression]

Create Filter Attribute

Filter can optionally create a new attribute on each outgoing row, which receives the label of the rule expression that filtered the row.

Normal attribute naming rules apply including forcing the attribute to be unique so it doesn’t clash with an existing attribute name on the incoming row stream.

Power Feature

To avoid having long label names in the outgoing attribute a pipe ‘|’ character can be used to split the label into a code and description.

For example, if the following two filters are defined:

:FILTER "F1 | Description 1" = ...
:FILTER "F2 | Description 2" = ...

then the Filter Attribute will receive “F1” or “F2” instead of the whole label. Spaces before the pipe character are trimmed when extracting each code.

Results

Filter node results have one or two tabs, depending on whether the node produced any warnings.

Summary Tab

../../_images/filter_resultsSummaryTab.png

Summary

The Summary table displays

  • Rows Processed - Count of all rows from the incoming data stream
  • Rows Filtered - Count of all rows that were filtered by the set of rules
  • Rows Not Filtered - Count of rows not filtered by any rule
  • Error Rows - Rows where an error occurred against one or more of the rules

Filter Details

The Filter Details table displays

  • Rule - Rule label
  • Rows Filtered - Count of rows filtered by rule
  • Error Rows - Count of rows in error
  • % of Rows Processed - Percent of rows filtered by the rule out of Rows Processed

Warnings Tab

../../_images/filter_resultsWarningsTab.png

Filter node results will only be available if there is a warning. Filter warnings occur when one or more rule errors occur. Select a rule in the Warning Summary table to explore detailed examples of what produced the warnings in the Detailed Examples table, including the operation and arguments involved.

Output Streams

Filter provides multiple output streams:

  • If a row tests true against a rule, it is sent to both the “Filtered Rows” and rule-specific output streams
  • If a row does not test true against any rule, it is sent to the “Not Filtered Rows” output stream
  • If a row errors while testing against a rule, it is sent to the “Error Rows” output stream

Not Filtered Rows

Rows that did not test true against any rule

Filtered Rows

Rows that tested true against a rule

Error Rows

Rows where an error occurred against one of the rules

Rule Expression Rows

Each defined rule has an associated output stream with a name that matches the rule’s label:

For example if you defined two filters:

:FILTER "Description 1" = ...
:FILTER "Description 2" = ...

Then two output streams will be created:

"Description 1" Rows  // receives all rows that evaluate to true against the "Description 1" expression
"Description 2" Rows  // receives all rows that evaluate to true against the "Description 2" expression