get_sundered_data(), target rows with a matching row in the comparison table pass; rows missing from the target can't be part of the sundered data. Sundering for this step is available with in-memory tables (not lazy or database tables) and isn't available in multiset mode.
## Preprocessing
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data table during interrogation. This function should take a table as input and return a modified table. This is useful for performing any necessary transformations or filtering on the data before the validation step is applied.
The preprocessing function can be any callable that takes a table as input and returns a modified table. For example, you could use a lambda function to filter the table based on certain criteria or to apply a transformation to the data. Note that the same preprocessing is **not** applied to the comparison table; only the target table is preprocessed. Regarding the lifetime of the transformed table, it only exists during the validation step and is not stored in the [Validate](Validate.md#pointblank.Validate) object or used in subsequent validation steps.
## Thresholds
The `thresholds=` parameter is used to set the failure-condition levels for the validation step. If they are set here at the step level, these thresholds will override any thresholds set at the global level in `Validate(thresholds=...)`.
There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values can either be set as a proportion failing of all test units (a value between `0` to `1`), or, the absolute number of failing test units (as integer that's `1` or greater).
Thresholds can be defined using one of these input schemes:
1. use the Thresholds class (the most direct way to create thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and 'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units for the 'warning' level only
If the number of failing test units exceeds set thresholds, the validation step will be marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be set, you're free to set any combination of them.
Aside from reporting failure conditions, thresholds can be used to determine the actions to take for each level of failure (using the `actions=` parameter).
## Cross-Backend Validation
The two tables can come from different backends (e.g., a Polars DataFrame and a Pandas DataFrame, or a database table and a CSV file). Tables on the same backend are compared in place (so two tables in the same database are compared inside the database). Tables on different backends are both converted to Polars (or to Pandas, if Polars isn't installed) before the comparison. Key columns whose types differ between the tables (e.g., integer keys that became floats in Pandas because of a missing value) are cast to a common type.
Values in columns whose types differ are compared numerically when both columns are numeric and as strings otherwise; note that such differences also fail the strict schema check (use `schema="common"` to only report them).
## Examples
For the examples here, we'll create two simple tables to demonstrate the [tbl_match()](Validate.tbl_match.md#pointblank.Validate.tbl_match) validation.
``` python
import pointblank as pb
import polars as pl
# Create the first table
tbl_1 = pl.DataFrame({
"a": [1, 2, 3, 4],
"b": ["w", "x", "y", "z"],
"c": [4.0, 5.0, 6.0, 7.0]
})
# Create an identical table
tbl_2 = pl.DataFrame({
"a": [1, 2, 3, 4],
"b": ["w", "x", "y", "z"],
"c": [4.0, 5.0, 6.0, 7.0]
})
pb.preview(tbl_1)
```
PolarsRows4Columns3
|
|||
|---|---|---|---|
a
Int64
|
b
String
|
c
Float64
|
|
| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 | 
tbl_match()
|
 |
✓ | 4 | 4 1.00 |
0 0.00 |
-- | -- | -- | -- |
| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C66 | 1 |
tbl_match()
|
|
✓ | 5 | 2 0.40 |
3 0.60 |
-- | -- | -- | CSV |
Report for Validation Step 1 ✗
ASSERTION tbl_match(keys='a')
tbl_compare (4 rows) → target (4 rows), aligned by key aDIFFERENT 3 of 5 rows fail (60.0%) match 2changed 1missing 1extra 1 |
|
|---|---|
| Column mismatches (1 of 2 columns) | |
| c | |
| Changed rows (1 row) | |
| 2 | c: 5 → 5.5 (+0.5) |
| Missing from target (1 row) | |
| 3 | b: y c: 6 |
| Extra in target (1 row) | |
| 5 | b: v c: 8 |