Incremental Validation:Faster Validation for
High-Volume Reporting
When regulatory reporting involves tens or hundreds of millions of records, validating the entire population in every reporting cycle can become a processing bottleneck.
Incremental Validation focuses processing on data that has changed and records impacted by those changes, helping accelerate validation while preserving the context needed for reliable regulatory reporting.
Explore the Case StudiesPL01 Population
100M+
Records
TAB01 Population
~70M
Rows per reporting cycle
Validation Approach
Incremental
Change- and impact-driven
PL01 — Pihak Lawan
Validate the Delta, Not the Entire Population
PL01 uses a delta reporting mechanism, where only changes during the reporting period are reported. Its data includes customers, debtors, issuers, guarantors, and other counterparties referenced across regulatory reports.
As the population grows beyond 100 million records, validating the entire dataset in every cycle can take approximately 12–24 hours, depending on the complexity of the validation rules.
With delta-based incremental validation, the system compares changes against the previous snapshot and focuses validation on new records, corrections, and deletions. In this illustrative scenario, only around 100,000 to 1 million records need to be processed per reporting period.
The key takeaway
A large data population does not necessarily require full-population validation in every cycle when the reporting process is based on deltas.
Illustrative processing results
Total population
100M+
Records in the reporting dataset
Before
12–24 hrs
Full-population validation
After
30 min–2 hrs
Delta-focused validation
Records processed per period
100K–1M
Illustrative processing results
Total population
~70M
Rows per reporting cycle
Before
12–18 hrs
End-to-end validation
After
1–2 hrs
Impact-based validation
Records processed per cycle
500K–3M
Focused on changed data and records impacted by reference changes
TAB01 — Tabungan
Validate Changed Data and Impacted References
A TAB01 reporting cycle can contain approximately 70 million rows. Before optimization, end-to-end validation may take around 12–18 hours because many validation rules depend on Pihak Lawan reference data.
Pihak Lawan data behaves like a slow-changing dimension: its changes are relatively small compared with the overall savings population. However, a change to a reference can affect many related TAB01 records.
Impact-based incremental validation identifies changes in relevant data, columns, and dimensions, then focuses validation on the associated records affected by those changes. In this illustrative scenario, around 500,000 to 3 million records are processed per cycle, reducing validation time to approximately 1–2 hours.
The key takeaway
Reference changes can affect dependent data even when the dependent records themselves have not changed. Validation must account for both the change and its impact.
From Full-Population to Change-Driven Validation
Different reporting characteristics call for different optimization strategies, united by one principle: validate what changed and understand what is impacted.
| Comparison | PL01 — Pihak Lawan | TAB01 — Tabungan |
|---|---|---|
| Optimization focus | Delta changes | Changes and impacted references |
| Total population | 100M+ records | ~70M rows |
| Records processed | ~100K–1M | ~500K–3M |
| Before | 12–24 hours | 12–18 hours |
| After | 30 min–2 hours | 1–2 hours |
Figures are illustrative estimates based on the scenarios described. Actual results depend on data characteristics, validation rules, infrastructure, and implementation.
Why Incremental Validation?
Shorter validation cycles help organizations use processing resources more effectively and give business teams more time to prepare accurate regulatory submissions.
Faster Validation
Reduce validation time by focusing on relevant changes and affected records.
Efficient Processing
Direct computing resources toward data that has changed or is actually impacted.
Earlier Issue Resolution
Give business users more time to review, investigate, and correct data issues before submission.
Scalable Reporting
Support growing reporting volumes without processing the entire population from scratch in every cycle.
Validate What Changed.Understand What Is Impacted.
By moving beyond repeated full-population validation, organizations can focus processing where it matters most and create more time for data quality, review, and regulatory readiness.