r/dataengineering • u/Chamelon_DE • 19h ago
Discussion AI can migrate a data warehouse now, but how do you actually trust the migrated data?
Was reading abt Devin’s new Databrick migration offering. Apparently they halped migrate a 142 opject data platform in 8 day.
Pretty interesting, but the part i’m more curious abt is data reconciliation rather than the actual code migration.
U can translate oracle sql to spark sql, get all the job running and even have matching row counts. But thing like null handlings, decimal precision or timezone conversion can quitely change the result without throwing any errors
Then there’s incremental load, late arriving data and old business rules buried somewhere in a stored procedure.
For people who have done large DWH migration, how do u actually validate that the new system behave the same as the old one?
Are row counts, checksum and record level diffs enough, or do u usually run both system in parallel for a while before switching over?
Curious how much of that validation you’d be comfortable handing over to an AI agent.
Source: devincentral.com/news/editorial-devin-launches-ai-driven-migration-tool-for-databricks