How to filter

The Filter class applies a config-based row filter to a Polars DataFrame or LazyFrame and returns a result of the same type. It reuses the same check expression notation as Validator — command, check_case, expressions, subject, arg_values — so the same operators and composition patterns apply.

A filter config requires a name and a filter key. An optional select key limits the returned columns.

Simple filter

The simplest form is a single check expression referencing one column.

import polars as pl
from dataguard import Filter

# Simple filter — keep rows where status == 'active'
config = {
    'name': 'active records',
    'filter': {
        'command': 'is_equal_to',
        'subject': ['status'],
        'arg_values': ['active'],
    },
}

df = pl.DataFrame({
    'id':     [1,        2,          3,        4],
    'age':    [25,       17,         34,       42],
    'status': ['active', 'inactive', 'active', 'inactive'],
})

filt = Filter.config_from_mapping(config)
result = filt.apply(df)
print(result)

shape: (2, 3)
┌─────┬─────┬────────┐
│ id  ┆ age ┆ status │
│ --- ┆ --- ┆ ---    │
│ i64 ┆ i64 ┆ str    │
╞═════╪═════╪════════╡
│ 1   ┆ 25  ┆ active │
│ 3   ┆ 34  ┆ active │
└─────┴─────┴────────┘

Conjunction

Use check_case: conjunction to keep rows that satisfy all expressions.

# Conjunction — age >= 18 AND country in [BE, BR]
config_conj = {
    'name': 'adult be/br',
    'filter': {
        'check_case': 'conjunction',
        'expressions': [
            {
                'command': 'is_greater_than_or_equal_to',
                'subject': ['age'],
                'arg_values': [18],
            },
            {
                'command': 'is_in',
                'subject': ['country'],
                'arg_values': ['BE', 'BR'],
            },
        ],
    },
}

df_conj = pl.DataFrame({
    'age':     [10, 20, 25, 30],
    'country': ['BE', 'BE', 'US', 'BR'],
})

filt_conj = Filter.config_from_mapping(config_conj)
result_conj = filt_conj.apply(df_conj)
print(result_conj)

shape: (2, 2)
┌─────┬─────────┐
│ age ┆ country │
│ --- ┆ ---     │
│ i64 ┆ str     │
╞═════╪═════════╡
│ 20  ┆ BE      │
│ 30  ┆ BR      │
└─────┴─────────┘

Disjunction

Use check_case: disjunction to keep rows that satisfy at least one expression.

# Disjunction — age < 18 OR status == 'VIP'
config_disj = {
    'name': 'minor or vip',
    'filter': {
        'check_case': 'disjunction',
        'expressions': [
            {
                'command': 'is_less_than',
                'subject': ['age'],
                'arg_values': [18],
            },
            {
                'command': 'is_equal_to',
                'subject': ['status'],
                'arg_values': ['VIP'],
            },
        ],
    },
}

df_disj = pl.DataFrame({
    'age':    [10,  25,    35,   15],
    'status': ['OK', 'VIP', 'OK', 'OK'],
})

filt_disj = Filter.config_from_mapping(config_disj)
result_disj = filt_disj.apply(df_disj)
print(result_disj)

shape: (3, 2)
┌─────┬────────┐
│ age ┆ status │
│ --- ┆ ---    │
│ i64 ┆ str    │
╞═════╪════════╡
│ 10  ┆ OK     │
│ 25  ┆ VIP    │
│ 15  ┆ OK     │
└─────┴────────┘

Nested expressions

Conjunctions and disjunctions can be nested freely to express compound conditions.

# Nested — (age >= 18 AND country in [BE, BR]) OR status == 'VIP'
config_nested = {
    'name': 'adult be/br or vip',
    'filter': {
        'check_case': 'disjunction',
        'expressions': [
            {
                'check_case': 'conjunction',
                'expressions': [
                    {
                        'command': 'is_greater_than_or_equal_to',
                        'subject': ['age'],
                        'arg_values': [18],
                    },
                    {
                        'command': 'is_in',
                        'subject': ['country'],
                        'arg_values': ['BE', 'BR'],
                    },
                ],
            },
            {
                'command': 'is_equal_to',
                'subject': ['status'],
                'arg_values': ['VIP'],
            },
        ],
    },
}

df_nested = pl.DataFrame({
    'age':     [10,   20,   25,   30,   15],
    'country': ['BE', 'BE', 'US', 'BR', 'US'],

shape: (4, 3)
┌─────┬─────────┬────────┐
│ age ┆ country ┆ status │
│ --- ┆ ---     ┆ ---    │
│ i64 ┆ str     ┆ str    │
╞═════╪═════════╪════════╡
│ 20  ┆ BE      ┆ OK     │
│ 25  ┆ US      ┆ VIP    │
│ 30  ┆ BR      ┆ OK     │
│ 15  ┆ US      ┆ VIP    │
└─────┴─────────┴────────┘

Column selection

Add a select key to return only the specified columns after filtering.

})

filt_nested = Filter.config_from_mapping(config_nested)
result_nested = filt_nested.apply(df_nested)
print(result_nested)

# Column selection — return only selected columns after filtering
config_select = {
    'name': 'active age only',
    'filter': {
        'command': 'is_equal_to',
        'subject': ['status'],
        'arg_values': ['active'],
    },
    'select': ['id', 'age'],

shape: (2, 2)
┌─────┬─────┐
│ id  ┆ age │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 25  │
│ 3   ┆ 34  │
└─────┴─────┘

Available commands

The command field accepts the same values as check expressions:

'is_equal_to'
'is_equal_to_or_both_missing'
'is_greater_than_or_equal_to'
'is_greater_than'
'is_less_than_or_equal_to'
'is_less_than'
'is_not_equal_to'
'is_not_equal_to_and_not_both_missing'
'is_unique'
'is_duplicated'
'is_in'
'is_null'
'is_not_null'

Configuration reference

name:   <string>
filter: <simple or complex check expression>
select: <empty or list of column names to return>

A simple filter expression:

command: <string — one of the commands above>
subject: <list with the column name to filter on>
arg_values: <empty or list of values>
arg_columns: <empty or list of column names>

A complex filter expression:

check_case: <conjunction | disjunction | condition>
expressions: [<2 or more simple or complex expressions>]