Filters

Filter functions can be used to restrict a data pipeline to only a subset of a data source, which resembles the WHERE clause of a SQL query. For instance, a data pipeline may need to process only records with a valid email address stored in a certain attribute and, to this end, apply the Match Regular Expression filter.

Additionally, filter functions can be used to restrict a transformation function to only a subset of all values of a certain attribute.


Contain Substring
String

Selects only values that contain a given substring.


End With Substring
String

Selects only values that end with a given substring.


Equal Value
All Data Types

Selects only values that equal a given value.


Greater Than Value
Date Double Float Int Long Time Timestamp

Selects only values that are greater than a given value.


Less Than Value
Date Double Float Int Long Time Timestamp

Selects only values that are less than a given value.


Match Regular Expression
String

Selects only values that match a given regular expression. By default, this filter evaluates to true if only a sub string of a value matches the regular expression. Please use ^ as prefix and $ as suffix for your regular expression, if you want to select only values that completely match the expression. Note that this filter performs a case-sensitive matching.


Missing Value
All Data Types

Selects only NULL values.


Not Equal Value
All Data Types

Selects only values that do not equal a given value.


Present Value
All Data Types

Selects only values that are not NULL.


Start With Substring
String

Selects only values that start with a given substring.