r/ETL • • 1h ago

Consultants / platform leads: how do you build monthly cost reports and next-year budget forecasts for clients or finance across Snowflake + Fivetran + dbt?

1 Upvotes

Spreadsheet? SELECT? Custom dashboards? How long does it take each month?


r/ETL • • 7h ago

Incremental Load to OBT

Thumbnail reddit.com
1 Upvotes

r/ETL • • 2d ago

CDC into a warehouse

3 Upvotes

Hi. What are people using for CDC in to a warehouse when they need changes to show up fast? Im comparing managed tools with a Debezium setup and trying to understand where the operational tradeoffs land.


r/ETL • • 2d ago

Real-world production reviews of Databricks Apps at scale? Looking at MDM & OLTP use cases.

3 Upvotes

We’re evaluating Databricks Apps for an internal MDM/transactional workflow serving thousands of users. We’ve done small POCs and love the native auth and Unity Catalog integration, but we want real-world production feedback before committing.

If you're running this in prod for large user bases:

Concurrency & Scaling: How does it handle hundreds of simultaneous users and concurrent writes? Any memory or session bottlenecks?

Backend Storage: Are you pairing it with Lakebase (Postgres) or an external OLTP DB for low-latency row-level writes, or hitting Delta directly?

Framework & Cost: Did you stick with Streamlit/Dash or move to FastAPI + React for state management, and how are always-on compute costs looking?

Looking for honest lessons learned or gotchas before we scale up!


r/ETL • • 2d ago

Databricks 5 Minute Features: Multiple Job Triggers

Thumbnail
youtube.com
1 Upvotes

Multiple Job Triggers is one of those features that on the surface seem really small - but it makes a great difference in your orchestration setup.

Check it out!

Documentation can be found here

https://docs.databricks.com/aws/en/jobs/triggers#add-multiple-triggers-to-a-job


r/ETL • • 2d ago

hot take - are we paying for real time data nobody actually uses?

Thumbnail reddit.com
1 Upvotes

r/ETL • • 3d ago

For teams running ClickHouse, how are you getting mutable operational data into it?

4 Upvotes

The use case is operational data from SQL Server landing in ClickHouse for analytics that needs to stay relatively current. The data is not append-only: there are updates, deletes, schema changes, and occasional backfills. The freshness target might be a few seconds or a few minutes, depending on the use case.

The obvious options seem to be:

  • Scheduled incremental loads
  • Debezium/Kafka/Connect
  • ClickPipes or another native ingestion option
  • A managed CDC platform like Striim for ClickHouse

For people who have built this, what did you choose and why?

I’m especially interested in:

  • How you handled the initial load plus CDC handoff
  • Updates/deletes and late or out-of-order records in ClickHouse
  • Schema changes and recovery/replay
  • The operational overhead you underestimated
  • Whether you would make the same architecture choice again

Would also be curious to hear from anyone who started with Kafka or a managed platform and later moved away from it.


r/ETL • • 4d ago

I got tired of spinning up environments just to move and query data, so I built an open-source local desktop ETL platform

Thumbnail
gallery
18 Upvotes

I know this sub gets flooded with weekend vibe-coded wrappers, so to be upfront: I’ve been building this for almost a year. 

I had a simple goal: build a privacy focused, no learning curve desktop ETL tool that lets analysts and developers move, transform, and query data without having to set up infrastructure or write code really.

Current state and struggle:

Pipeline generation and data analysis work reliably with frontier cloud models (scoring 50/50 in a SQL benchmark we recently published, GPT-6 Luna). Smaller local models still struggle on multi-table queries, so I’m currently building out a semantic layer with verified queries, explicit column types, table relationships, and simplified SQL output schemas, to cut down hallucinations. Any other suggestions to improve performance on locally hosted model?

So far what it can do:

Zero Environment Setup: Install on mac, Windows, or Linux 

80+ Source & Destination Connectors: Move data across relational DBs, cloud warehouses, SaaS APIs, local files, and Vector DBs in the same pipeline.

Prompt or write code (SQL, JS, Python): Execute transformations locally or on target database.

Safety**:** Includes automated code validation, schema drift notifications, and self-correcting logic to pipelines. All DML changes require approval before execution. All credentials are encrypted at rest locally and never get passed to AI.

Embedded Analytics: Query your data locally right inside the app

Next release will have MCP server connector

I’d love your honest feedback for what should change or be added.

https://github.com/Ternlux/ternlux/


r/ETL • • 3d ago

Incremental loads with updated_at > last_watermark can silently skip rows. A tiny simulation of why, and the overlap-plus-upsert fix

1 Upvotes

A common incremental extract: remember the max updated_at you loaded, and next run pull rows with updated_at > watermark. It works until a row becomes visible to readers after rows with later timestamps, for example when a long transaction commits late and keeps its earlier updated_at.

A minimal simulation (made-up data, integer times). Each row is (id, updated_at, visible_at):

a: updated_at=100, visible at 101
b: updated_at=105, visible at 130   <- slow commit
c: updated_at=110, visible at 111
d: updated_at=140, visible at 141
Pipeline runs at t=120 and t=150.

Naive watermark (no overlap):

  • Run at 120: sees a and c (b isn’t visible yet). Watermark becomes 110.
  • Run at 150: asks for updated_at > 110. b has updated_at = 105, so it’s never selected. Only d is loaded.
  • Result: loaded a, c, d. Missing: b, with no error anywhere.

Overlap window plus upsert by key (30 time units back):

  • Run at 120: loads a and c, watermark 110.
  • Run at 150: asks for updated_at > 110 - 30 = 80, so it re-reads a, b, c, d.
  • With an upsert by primary key, the target ends with a, b, c, d and no duplicates.

Overlap without a dedupe step: the same overlap with plain inserts loaded a and c twice. So the overlap needs an idempotent load (upsert / merge by key, or delete-and-reload by partition).

Ways to deal with it (and what each costs):

  • Overlap window + idempotent upsert. Simple and robust against late commits, but you re-process some rows each run, and the right window size depends on how late your data can arrive.
  • Use a monotonic change marker instead of a timestamp, such as a log sequence or change-data-capture stream, if your source offers one. This avoids the timestamp-visibility problem, but it depends on the source system and adds operational complexity.
  • Periodic full reconciliation, such as comparing counts or checksums per partition between source and target, to catch what incremental logic missed. It costs compute, but it checks the thing that matters.

Limits of this post: the simulation uses fictional rows and integer times to show the principle. Real sources differ in clock skew, isolation levels, time zones, and whether updated_at is set by the application or by the database, so check how yours behaves. Hard deletes are another gap: a watermark query never sees a row that was deleted.

How do you handle late-arriving rows in incremental loads: overlap windows, CDC, or reconciliation jobs? And how did you find out your first incremental load was missing data?


r/ETL • • 3d ago

Databricks otel logs

1 Upvotes

I’m sort of new to Databricks and trying to setup a o tel logs for a clients Databricks app. The service principal has access to otel log tables that got created while configuring. I’m getting following error “cannot enable app telemetry table’s storage could not be accessed from serverless compute. Verify that network connectivity NCC is configured and that apps service principal has access to the table’s storage. Note that storage behind a private end point is not yet supported”

Main reason to use otel logs is so that we can store the logs and query it for usage analysis. Would really appreciate any feedback or advice.
Happy to provide more details
Thanks


r/ETL • • 4d ago

Question for Databricks engineers and data architects:

Thumbnail reddit.com
1 Upvotes

r/ETL • • 4d ago

Why do so many skilled people still spend hours copying data between spreadsheets?

Thumbnail reddit.com
0 Upvotes

r/ETL • • 4d ago

[DISCUSSION] A payment fails at 2:15 pm and support finds out overnight. When is batch ETL still the right call?

Post image
2 Upvotes

r/ETL • • 4d ago

Legacy Infrastructure will fail under agent load.

Thumbnail
datpaq.com
1 Upvotes

r/ETL • • 5d ago

Hive to Iceberg migration?

6 Upvotes

We were recently talking to few big enterprises who are on Hive, but due to natural correlation to Apache Iceberg (as both ultimately store the data in parquets only and ACID compliant+Query-support from every engine out there), they wanted to move. we are thinking to create a tool to migrate from Hive to Iceberg (without people having to actually copy the data).

How many people feel this is a valid concern and if there are any experiments/product stories from people who have already achieved it?

Also thumbs up if you feel this is genuinely required and need a solution


r/ETL • • 5d ago

Is legacy data migration still a major pain point for enterprises? What if we could automate most of it?

3 Upvotes

&#x200B;

I've been exploring a problem that I recently encountered while working on enterprise data migration, and I'm curious to hear how others approach it.

A lot of organizations are moving from legacy platforms to modern systems. However, the actual migration process can be surprisingly complicated.

Some challenges I've noticed:

\- Inconsistent folder structures across different years or departments.

\- Different naming conventions for documents.

\- Complex relationships between records and their attachments.

\- Missing files, duplicate documents, and incorrectly classified data.

\- Platform-specific migration requirements.

\- A significant amount of manual effort to validate everything.

In many cases, the challenge isn't simply transferring files. It's understanding the existing data structure and preserving the relationships and organization during migration.

I'm exploring the idea of building an intelligent, configurable data migration tool that could:

  1. Analyze and understand an existing folder/document structure.
  2. Automatically identify and classify records using naming patterns and metadata.
  3. Extract relevant documents and their attachments.
  4. Map the existing structure to a target platform.
  5. Validate migrated data and generate detailed reports.

Ideally, it would require minimal manual configuration rather than developers writing custom extraction scripts for every migration project.

A few questions for people working in this space:

  1. How common are these problems in real-world enterprise migrations?
  2. Are existing tools like ShareGate, AvePoint, or custom ETL pipelines already solving this effectively?
  3. Would an intelligent, configurable migration engine provide meaningful value?
  4. What would be the hardest part to automate reliably?

I'm not trying to reinvent something that already exists. I'd genuinely like to understand the gaps in current solutions before investing time in building it.

Would love to hear experiences from people who've worked on SharePoint migrations, enterprise document management, data engineering, or digital transformation.


r/ETL • • 5d ago

How to Stop AI Agents From Running Up Your Snowflake Bill

Thumbnail
lakeops.dev
2 Upvotes

r/ETL • • 5d ago

Is legacy data migration still a major pain point for enterprises? What if we could automate most of it?

0 Upvotes

&#x200B;

I've been exploring a problem that I recently encountered while working on enterprise data migration, and I'm curious to hear how others approach it.

A lot of organizations are moving from legacy platforms to modern systems. However, the actual migration process can be surprisingly complicated.

Some challenges I've noticed:

\- Inconsistent folder structures across different years or departments.

\- Different naming conventions for documents.

\- Complex relationships between records and their attachments.

\- Missing files, duplicate documents, and incorrectly classified data.

\- Platform-specific migration requirements.

\- A significant amount of manual effort to validate everything.

In many cases, the challenge isn't simply transferring files. It's understanding the existing data structure and preserving the relationships and organization during migration.

I'm exploring the idea of building an intelligent, configurable data migration tool that could:

  1. Analyze and understand an existing folder/document structure.
  2. Automatically identify and classify records using naming patterns and metadata.
  3. Extract relevant documents and their attachments.
  4. Map the existing structure to a target platform.
  5. Validate migrated data and generate detailed reports.

Ideally, it would require minimal manual configuration rather than developers writing custom extraction scripts for every migration project.

A few questions for people working in this space:

  1. How common are these problems in real-world enterprise migrations?
  2. Are existing tools like ShareGate, AvePoint, or custom ETL pipelines already solving this effectively?
  3. Would an intelligent, configurable migration engine provide meaningful value?
  4. What would be the hardest part to automate reliably?

I'm not trying to reinvent something that already exists. I'd genuinely like to understand the gaps in current solutions before investing time in building it.

Would love to hear experiences from people who've worked on SharePoint migrations, enterprise document management, data engineering, or digital transformation.


r/ETL • • 6d ago

Looking for real-world Saga examples to implement advanced transactions in HQL

5 Upvotes

TL;DR: I'm looking for good real-world examples of Saga implementations, especially how they handle compensating actions, retries, idempotency, and recovery across multiple systems. I'm developing HQL, a query language in Rust, and considering adding Saga support. The HQL examples below are mostly there to provide context.

Over the past few months I've been developing HQL, a strongly typed functional query language written in Rust.

The idea is something like SQL, but more of a Swiss Army knife for data, connecting PostgreSQL, Kafka, Spark, Delta Lake, and other systems through the same programming model.

I now have something that works relatively well as version 0.7.1 for basic OLAP workloads, including transformations, aggregations, streams, and data contracts.

For example, here's a real Kafka -> PostgreSQL pipeline that validates incoming events and routes invalid records into quarantine:

tap("order_events_raw")
| validate(EventData)
| fork {
    clean =
        oks()
        | map(e => Event(
            id = e.id,
            kind = e.kind,
            customer = e.customer,
            amount = e.amount,
            at = e.at))
        | sink("order_clean"),

    quarantine =
        errs()
        | map(e => Rejected(
            id = match e.value.id { some(Int n) => n, _ => 0 },
            message = e.value,
            rules = e.violations | map(v => v.rule)))
        | sink("order_quarantine"),
}

Complete example: https://github.com/ewiger/hql-demo/blob/main/kafka-postgres/quarantine.hql

For OLTP, I recently had to introduce a transaction mechanism into the language. Here's a snippet from a working PostgreSQL ledger example:

with transaction(attempts = 3, isolation = Serializable) tx on ledger {
    sender = source(accounts) | filter(a => a.id == 1) | one
    tx.rollback("insufficient balance") given sender.balance < amount

    debited =
        update accounts
        | filter(a => a.id == 1)
        | set(a => {balance: a.balance - amount})

    credited =
        update accounts
        | filter(a => a.id == 2)
        | set(a => {balance: a.balance + amount})

    tx.rollback("no such account") given debited != 1 or credited != 1
}

Complete example: https://github.com/ewiger/hql-demo/blob/main/ledger/transfer.hql

The transaction scope makes the block atomic over its participants. isolation = Serializable only selects the isolation level.

A transaction scope, with transaction(…) tx on {…}, makes its block atomic over its participants: tables of one PostgreSQL database or one SQLite file, or one Delta table. They are read with source and changed with insert, update and delete. isolation = Serializable sets the strongest isolation: concurrent scopes that would conflict are refused, and attempts retries them. Writes outside the participants, such as a sink to a topic or file, are not part of the transaction.

In general, HQL transaction scopes rely on the ACID guarantees of the underlying storage engine (PostgreSQL, SQLite, or Delta Lake). isolation = Serializable specifies the isolation level, not the entire ACID contract. Currently, a transaction scope is limited to a single transactional participant/target; it does not provide atomic commits across independent databases, Kafka, or external services. That's the problem I'm exploring with Sagas.

My next idea is to add a Saga-style scope alongside with transaction.

So now I'm looking into how Saga patterns could fit into HQL, coordinating operations across multiple databases, Kafka, and external services.

I'm particularly interested in good real-world examples of Saga implementations, especially how to model compensating actions, retries, idempotency, and recovery. Hence, if you've implemented Sagas in production or seen an elegant approach in another language, I'd be interested in hearing your thoughts.

PS

Disclaimer: This is all still very much WIP. I also use Claude Code extensively for tests and the Rust backend of the language. Even if you're allergic to AI-assisted projects but know a lot about Sagas, I'd be super grateful for your input.

PPS

This blog post looks great:

https://hosseinnejati.medium.com/the-saga-pattern-managing-distributed-transactions-ae8236ccd4c4

However, I've also read a lot of literature warning against Sagas. They're hard to implement, a lot of things can break, and distributed systems are already extremely complex in general.

But I feel super tempted to dive into this topic. If you really, really know a good real-world example, or an attempt to build such a design, I'd be super grateful if you could share it!


r/ETL • • 6d ago

Company job history and hiring momentum

Thumbnail
foorilla.com
2 Upvotes

foorilla.com v1.14.24 | foorilla API v1.12

Every company now comes with its job history: how many jobs it has open, how many it published and how many expired, week by week and month by month, across every job source we track, not only the jobs listed on foorilla. A company's Insight tab shows its open jobs against a quarter earlier, the latest four weeks of new and expired jobs, the median stated annual pay of its open jobs, and how its open jobs split by experience, work mode and country. You can download the whole history as CSV.

Company lists show each company's open jobs and their 13-week change, and sort by open jobs or hiring momentum. In Insight, the Hiring board adds open jobs across the market week by week, the industries gaining or losing open jobs, and the companies hiring fastest; Movers shows the companies rising and cooling.

API v1.12 adds this without breaking changes. /api/v1/hiring/company/st/ includes each company's history_open, history_open_before and history_open_change; company lists filter with open_min and sort with order. The new /api/v1/hiring/company/history/?company=ID returns a company's weekly and monthly history. See the API documentation for details.


r/ETL • • 6d ago

Data Warehousing: Talking Book Spoiler

Thumbnail gallery
1 Upvotes

Fresh off the Good Strat presses!


r/ETL • • 6d ago

Suggestions for my open-source lakehouse

7 Upvotes

I was tasked to design and setup an enterprise grade open-source Data Lakehouse on on-prem RedHat OpenShift.

Reason for on-prem: Sensitive data

Reason for open-source: Cost effective & feasible to have every component on-prem without data ever leaving the closed LAN

These are the components I choose:

RustFS

Apache NiFi

Apache Airflow

Apache Spark

Trino

PostgreSQL

Planning to use OAuth2, OIDC or Kerberos for user and component to component authentication

The project is still in design and POC stage. So I am open for suggestions and improvements.

Looking forward for valuable suggestions. Thanks in advance!


r/ETL • • 7d ago

I built a browser-local tool that turns a Snowflake catalogue into review-gated governance SQL, looking for feedback!!

Thumbnail
reddit.com
1 Upvotes

r/ETL • • 9d ago

Braidplane - Open Source (Apache 2.0) and self-hostable platform for declarative and contract-based data integration

3 Upvotes

Hello everyone!

Around Februrary this year I've started working on my side project, which is now known as Braidplane. It's an open source platform for data integration; it's self-hostable and the core is fully open (Apache 2.0). At this moment it's in Alpha.

The goal of this project is to provide safe and reviewable data integrations for engineering teams. From the beginning this project was built upon data contracts (inline and ODCS), e2e Braidplane Proof engine for reviewing data pipeline updates and declarative pipelines (YAML, JSON, cue).

As an example, a pipeline can read data from a database and a file, transform and compare the records, and write the results to another system. The pipeline definition stays in Git, so changes can be reviewed before deployment.

In the current form Braidplane core includes backend, frontend, CLI and executor. MCP support is coming soon.

About me: my name is Lukas. I've been working as backend engineer for the last 10 years. I've spent a lot of time around different products and APIs. I've built the Braidplane's framework myself - based on my experience and good practices I've found around the other OSS projects. Later on I've added AI assisted programming around the hardest/most time-consuming parts, like Connectors integrations, tests, documentation, frontend, issues management.

The Braidplane Core repository is available here: https://github.com/Braidplane/braidplane-core

Examples repo: https://github.com/Braidplane/examples/blob/main/recipes/opaque-file-archive/opaque-file-archive.pipeline.json

Cheers