r/dataengineering • • 20d ago

Discussion Monthly General Discussion - Nov 2026

4 Upvotes

This thread is a place where you can share things that might not warrant their own thread. It is automatically posted each month and you can find previous threads in the collection.

Examples:

  • What are you working on this month?
  • What was something you accomplished?
  • What was something you learned recently?
  • What is something frustrating you currently?

As always, sub rules apply. Please be respectful and stay curious.

Community Links:


r/dataengineering • • Sep 01 '26

Career Quarterly Salary Discussion - Sep 2026

41 Upvotes

This is a recurring thread that happens quarterly and was created to help increase transparency around salary and compensation for Data Engineering where everybody can disclose and discuss their salaries within the industry across the world.

Submit your salary here

You can view and analyze all of the data on our DE salary page and get involved with this open-source project here.

If you'd like to share publicly as well you can comment on this thread using the template below but it will not be reflected in the dataset:

  1. Current title
  2. Years of experience (YOE)
  3. Location
  4. Base salary & currency (dollars, euro, pesos, etc.)
  5. Bonuses/Equity (optional)
  6. Industry (optional)
  7. Tech stack (optional)

r/dataengineering • • 22h ago

Blog From 15 year industry veteran - How real-time ETA pipelines work in food delivery/logistics (Online vs. Offline Feature Store architecture + cost tradeoffs)

30 Upvotes

After 15 years in product engineering and working around logistics/ML systems, one pattern I keep seeing teams underestimate is feature freshness in real-time ETA prediction (like Zomato / Swiggy / DoorDash).

The tutorials you see in YT start at model.fit(). But calculating whether an order takes 22 mins or 38 mins is 80% data engineering and feature pipeline orchestration, and 20% ML.

Here’s how the pipeline actually breaks down in production when you separate ordinary business events from ML serving:

  1. Business Events First (Zero ML Yet)
    Every order emits lifecycle events through API Gateway/Lambda into raw S3 event logs.
  2. Background Feature Aggregation
    A Spark job groups these logs.
  3. The Dual Feature Store Split (The Part Teams Get Wrong)

Offline Store (S3/Parquet): Holds historical time-travel snapshots for model training without leakage.
Online Store (Low-Latency KV / SageMaker Online Store): Serves single-digit-millisecond lookups for live features right at checkout.

4. Live Inference Orchestration
This fetches pre-computed features from the Online Feature Store, combines them with live request context (distance, traffic), hits the model endpoint, and returns the ETA.

(Note: If you want to see the full 38-minute whiteboard & sequence diagram walkthrough of this on AWS SageMaker Feature Store + Spark, I broke it down here: https://www.youtube.com/watch?v=v_4vzmv436s - happy to debate feature freshness SLAs or Online/Offline sync tradeoffs in the comments.)


r/dataengineering • • 1d ago

Discussion Checking naming conventions and data types across all Unity Catalog objects after a migration?

0 Upvotes

I'm a data engineer working on migrations to Databricks. After moving from legacy systems, table and column names often don't follow our standards or match the source-to-target mapping. Examples:

  • Legacy CUST_NM became custname instead of customer_name
  • Date columns stored as STRING
  • The same column name with different data types in different tables

I know about governed tags and Data Classification for sensitive data, but I haven't found a good way to check naming and type standards across every catalog and schema.
Questions for the community:

  1. How do you check naming and type standards across all Unity Catalog objects?
  2. Do you compare migrated tables against a source-to-target mapping sheet? How?
  3. Is there a tool you'd recommend, or is it mostly custom scripts or jobs?

r/dataengineering • • 19h ago

Discussion AI can migrate a data warehouse now, but how do you actually trust the migrated data?

0 Upvotes

Was reading abt Devin’s new Databrick migration offering. Apparently they halped migrate a 142 opject data platform in 8 day.

Pretty interesting, but the part i’m more curious abt is data reconciliation rather than the actual code migration.

U can translate oracle sql to spark sql, get all the job running and even have matching row counts. But thing like null handlings, decimal precision or timezone conversion can quitely change the result without throwing any errors

Then there’s incremental load, late arriving data and old business rules buried somewhere in a stored procedure.

For people who have done large DWH migration, how do u actually validate that the new system behave the same as the old one?

Are row counts, checksum and record level diffs enough, or do u usually run both system in parallel for a while before switching over?

Curious how much of that validation you’d be comfortable handing over to an AI agent.

Source: devincentral.com/news/editorial-devin-launches-ai-driven-migration-tool-for-databricks


r/dataengineering • • 1d ago

Discussion How do you enforce naming conventions on Snowflake objects created outside dbt?

1 Upvotes

Our dbt models follow naming and type rules, but a lot of objects are created outside dbt: Fivetran tables, manual tables, old views, and objects from migrations. Over time we get mixed naming (CUST_NM vs customer_name), dates stored as VARCHAR, and the same column with different types in different tables.

How does your team handle this?

  • Do you use a tool for it?
  • Custom scripts on INFORMATION_SCHEMA?
  • Or do you just live with it?

Also curious if anyone does the same on Databricks / Unity Catalog.


r/dataengineering • • 2d ago

Blog Scaling is easy if money is free. I built a free game where your system design has to survive traffic AND stay under budget

12 Upvotes

I kept reading about load balancers, caches and sharding, then forgetting all of it a week later. Diagrams never fail and never send you a bill, so you never find out why any of it matters.

So I built Bugpit. You build a system from blocks, send pretend traffic at it, and watch what breaks. Every design also shows what it would cost each month.

One level I like: 20,000 requests a second, 95% of them reads, and a $1,800/month budget. The obvious fix is more database copies, but that goes over budget. The cheaper fix is a cache in front of the database, and then you can cut your app servers down to 5 because they have far less to do.

What you can do:

  • Build a system from blocks: servers, load balancers, caches, databases, queues
  • Send traffic at it and watch overloaded parts turn red
  • See the monthly bill for every design, and hit budget goals
  • Crash a server on purpose and see if your design survives
  • Follow one request from start to finish to find where the time goes
  • Play real app levels for Netflix, Instagram and more
  • Survive levels based on real outages
  • Try a new puzzle every day, with its own budget
  • Answer short questions after each level, so the idea sticks
  • Get a step-by-step guide if you’re stuck, with new words explained as you meet them

49 levels so far, no code needed, runs in the browser.

To be upfront: it’s a simulation, not a real load test. The real app levels are lessons built on numbers these companies have made public, not copies of their systems, and I’m not connected to any of them.

Try it: Bugpit.vercel.app

I’d really like feedback on two things:

  1. Where did you get stuck or lost?
  2. If you work on real systems, which numbers, costs or results look wrong to you?

r/dataengineering • • 3d ago

Help Which of these two courses should I choose to transition from SQL only DEV to a DE?

30 Upvotes

Idk any Python, have never worked with Python. But now for DE, I want to learn so which one should I do?

  1. Data Engineering for Beginners: Learn SQL, Python & Spark by Durga Viswanatha Gadiraju / ITVersity
  2. The Complete Python Bootcamp From Zero to Hero in Python by Jose Portilla, Pierian

r/dataengineering • • 4d ago

Discussion Inherited a PB scale data mess. Looking for tech & people strategies.

56 Upvotes

Hello good folks of /r/dataengineering,

I need opinion &/ guidance on how you would handle such a situation.

Inherited a large data data platform PB level, with a lot of pipelines, tables and tools. Built mostly on OSS and K8s stack.

  1. Company grew very fast (think non-tech startup, public now), BigQuery is the primary warehouse, with a mix of batch and streaming.
  2. Engineers came and went, lot of the platform is not maintained and people do not wish to touch what works. Eg. There are teams that still use Airflow 1.10.
  3. There are a lot of pipelines and data sets are that are no longer in use, they're running on schedule and cost $$.
  4. Schedules are all over the place. I found a pipeline that runs daily, powering a report that only gets sent out every two weeks.

I was planning on mapping everything first then use that data to make culling decisions, but there is internal resistance from business. Folks are not even ready to spend time to add metadata on their own datasets.

What do you folks recommend I do? Any suggestions on both technical and people strategies would be greatly appreciated.

Please don't plug your tool here.


r/dataengineering • • 4d ago

Blog BigQuery Query Optimization: Myths vs. What Actually Saves Money

Post image
27 Upvotes

Interesting framework on how to think about optimizing SQL queries in BigQuery with partition pruning, caching, materialized views, data modeling... Here is the article: https://www.alvin.ai/blog/bigquery-query-optimization

Good to see that data modeling still plays a role, and not just rewriting SQL at query time. I feel like having most SQL code AI-generated today, by coding agents or during self-service analytics, makes the SQL optimization problem more complex than good old ETL codebases that didn't change that often.


r/dataengineering • • 4d ago

Discussion Airbyte has identified a security incident that they believe is part of a broader, global cyberattack.

Thumbnail
linkedin.com
134 Upvotes

Oof


r/dataengineering • • 3d ago

Personal Project Showcase Reaktiv Workbook – A spreadsheet where every cell is Python

4 Upvotes

Hey guys!

I am the maintainer of reaktiv, a Python Signals Reactivity state management package (inspired by Angular and SolidJS). After the positive resonance of the Python community with my reaktiv package, I like to go one step further to reimagine how Python notebooks (Jupyter Notebook, Marimo, …) could look like, if Signals gets used as the core mechanism for reactive data pipelines!

My mental model when working with Signals has always been the spreadsheet model. I am working in the signal processing field and use numpy, scipy everyday in combination with reaktiv. The combination has been so wonderful that I thought to myself, I should try to create a new Python package around this.

So here we are!

Reaktiv Workbook allows data exploration via a spreadsheet UI where every cell represents a signals primitive! Say hello again to Signal, Computed, Effect, Linked and Resource!

With the spreadsheet UI, we are modeling the data flow and business model and can see each piece of data affects which cell. Each cell can be embedded in a Markdown notes, to make the notes interactive with plots and live data. Great for documentation and little apps! Also the notes are always in sync with the spreadsheet.

There are also many little goodies here and there that many will appreciate: Cells can be copied and reused, MCP support, workbook fileformat is just a plain Python file, ...

In the last 2-3 months I tested many of my ideas and I think I found a good balance with this release.

Hopefully some data folks appreciate what I have done here. :)

You can interactively try Reaktiv Workbook in the browser!


r/dataengineering • • 4d ago

Help Pitchbook Share to Internal schema pipeline

5 Upvotes

Hello fellow data engineers, I need to build a new dashboard and backend data pipelines to show the data of the company that user search for…

So our company has subscribed to pitchbook snowflake share which provides different types of data and we need to fetch it from there and dump it into internal snowflake schemas.. and the frontend dashboard where user can search for the company and get information…

The data is most likely about company profile, investors, funding, news….

The backend pipelines we need to build using airflow jobs…


r/dataengineering • • 4d ago

Help Coming from ADO – dbt Cloud + GitHub CI triggering is a pain. How do you handle this?

11 Upvotes

We're migrating from Azure DevOps to GitHub and honestly, the dbt Cloud integration feels like pain. In ADO it just worked out of the box.

With GitHub, connecting via Deploy/Access Key gives us zero webhooks, so no automatic PR triggering. Meanwhile, the custom GitHub App option in dbt is still in Preview, which is a tough sell to our IT team.

How are you guys handling PR CI in stricter orgs? Triggering dbt Cloud jobs via GitHub Actions ?

Anyone using the beta custom Github App? How is it ?


r/dataengineering • • 4d ago

Discussion Secondary data warehouse app?

8 Upvotes

We are looking for a simple secondary data warehouse...

We are not IT professionals, and we do not want to hire an external expert for every single task.
We need an application designed for individuals with a general understanding of IT.

We intend to use this secondary data store to cross-reference data from various sources, perform simple calculations, and send cleaned data to secondary applications.

Requirements:
- 90% of our data is in MS SQL.
- We need simple, direct integration with MS SQL (IP, port, password), the ability to select tables to transfer to the secondary data store, and the option to set a refresh frequency...

The secondary data warehouse must include:
- The ability to create simple queries, apply filters, and establish relationships between tables...
- The ability to easily integrate with external applications (CRM, BI, AI agents, reporting, etc.)...

What do you recommend?


r/dataengineering • • 4d ago

Open Source What Is Apache IoTDB? An Introduction to the Open-Source Time-Series Database

Post image
0 Upvotes

Apache IoTDB is an open-source, IoT-native time-series database designed to efficiently manage massive volumes of time-series data generated by industrial devices, sensors, and IoT systems.

But what exactly is a time-series database, and why do IoT applications need one?

In many industrial environments, machines and sensors continuously generate data, including temperature, pressure, humidity, vibration, energy consumption, and equipment status. Each data point is associated with a timestamp, creating a continuous stream of time-series data. Traditional databases can store this information, but managing high-frequency data from thousands or even millions of devices introduces additional challenges.

This is where Apache IoTDB comes in.

Apache IoTDB provides specialized capabilities for time-series data collection, storage, management, and analysis. Its architecture is designed around the characteristics of IoT data, including high write throughput, large data volumes, complex device structures, and frequent out-of-order data arrivals.

Some of its key capabilities include:

  • Efficient storage: Specialized storage structures and compression techniques help reduce the storage footprint of time-series data.
  • Flexible data modeling: It supports both tree-based and table-based data models.
  • Time-series queries: Users can retrieve, aggregate, and analyze historical and real-time data.
  • Flexible deployment: It can be deployed in different environments, including edge and cloud scenarios.
  • Ecosystem integration: It supports integration with tools and platforms such as Grafana, Spark, and other data applications.

Apache IoTDB is particularly relevant to Industrial IoT (IIoT), where equipment monitoring, production data collection, and operational analytics depend on reliable access to time-series information.

In short, Apache IoTDB is more than a general-purpose database. It is a data management system built specifically around the needs of IoT and industrial time-series applications.

Official website: https://iotdb.apache.org/

GitHub: https://github.com/apache/iotdb


r/dataengineering • • 5d ago

Help DBT/Redshift to Powerbi Semantic Model Description Sync

4 Upvotes

I want to be able to use the PBI mcp so users can ask natural language questions about reports/questions in general, but to make it work well you need the semantic models to have column level descriptions. I’m using Redshift/DBT, so my column descriptions live in yaml files. To sync the descriptions to the semantic models via CI/CD, it seems like I would need to have the pbib files in Git and use something like fabric-cicd, but that would require analysts to use git for any changes they make to the semantic model (measure changes is the biggest thing). Is there some option to make this easy that I’m missing?


r/dataengineering • • 5d ago

Help GCP Professional Data Engineer Certification

12 Upvotes

Has anyone given the GCP Professional Data Engineer Certificatio recently? Looking for advice on preparation - i have followed some of the Google skills learning path and completed the Pierian Udemy course, but I still feel wildly unprepared for the exam.

Also since Google is supposed to have revised the exam pattern recently, I'm not sure which practice tests i can trust to give me a fair idea of my preparedness. Im currently doing the practice tests from Dan Sullivan on Udemy.

Any advice would be greatly appreciated!!


r/dataengineering • • 6d ago

Personal Project Showcase Open source: what's stuck in a Kafka flow, counted in orders and amounts instead of offsets

2 Upvotes

I am building this tool called "In-flight" as part of testing my architecture boards builder tool.

When a Kafka consumer fails on an event, it lands in a dead-letter queue. Kafka tooling shows a count; the business question is which orders and customers and how much money. "In-Flight" answers that without anyone writing a query.

It samples each topic and guesses what the fields mean: this one's the amount, in cents; that one's the customer. Each guess comes with a question, you answer a handful, and you see what each sampled event would be worth before you switch it on.

Code (Apache-2.0): https://github.com/provenmap/in-flight

Here is its architecture: https://provenmap.com/boards/paws/in-flight-the-money-moving-through-your-kafka


r/dataengineering • • 7d ago

Discussion Ingestion of sensitive data fields

54 Upvotes

Hello,

In some industries, we have to deal with sensitive data fields, whether it's PII (personally identifiable information) or other types of confidential data (e.g. investment orders, tax classes, medical records, etc.).

So far, in the European companies I've worked for, I've seen different approaches to this problem, despite all being subject to the same regulations. I've also tried reading some online content to get different points of view, but the approaches seem to vary quite a bit. These are the main options I've found or come across:

  1. Import everything into the datalake/bronze layer, then only provide access to the datamart/silver/gold layers, which contain only anonymized or aggregated data.
  2. Skip ingesting sensitive data fields unless they are strictly needed for downstream transformations.
  3. Mask or hash sensitive data values at ingestion.

My take on these options is:

- Option 1 feels like the most natural approach because it fully replicates the source data. However, it also brings the highest compliance risks, as sensitive data is stored in multiple locations, increasing the attack surface and the number of places where access needs to be controlled.

- Option 2 is the strictest and potentially the most compliant approach, but it means our datalake/bronze layer is no longer a full replication of its data sources. If some of the excluded fields are needed later, we might have lost access to historical data. The same applies to option 3 when masking is used.

- With option 3, hashing preserves the ability to perform joins and unique/distinct operations. However, apparently, some regulators still consider hashed PII to be PII if the hashing function is injective (i.e. each output value maps to a single input value). Even though the original PII values are no longer available, we can still link an individual across multiple records.

Overall, I think the right approach also depends on whether the people working with the data have a way to temporarily access the source systems directly. If they do, there may be no need to store sensitive data in the datalake/bronze layer. If they don't, keeping it there provides a way to access it when needed without having to configure privileged access to the source systems.

At my current company, there is quite a bit of disagreement on this topic. My manager is in favor of option 1, based on what he has seen in his previous experience, while the security/risk/data protection teams lean more towards option 2, arguing that sensitive data should only be stored where there is a valid business purpose and with strict access permissions.

If you've worked in an industry with regulations around sensitive data, what approach did your company take? I guess it is very different from one company and country to another but I'm curious about what approach you use(d) and if there are other alternatives to these 3 options.

Thanks!


r/dataengineering • • 7d ago

Personal Project Showcase Found a data model I drew 35 years ago today. First job as a data modeller and we'd challenge each other to model odd topics against our EDM. Printed on overhead projector film using ABC Flowcharter (RIP)

Post image
139 Upvotes

r/dataengineering • • 7d ago

Discussion Landing to Raw to Bronze or straight from Landing to Bronze?

52 Upvotes

tl;dr: I am currently working on building my personal data lake and medallion + platinum pipeline and need help deciding whether to have an Landing zone to Raw to Bronze steps for ingesting data or go directly to Bronze from the Landing zone.

Context:

I am working on a quantfied_self project where i am collecting a bunch of data on myself. Things like (everything is timestamped):
- Location at all points in the day,
- Every single workout set
- Major actions i do during a day like brushing teeth, reading a book, meals, work, socialising etc etc etc
- Financial transactions
- Hourly health metrics like heart rate, Hrv, steps, energy spent, calorie intake etc etc etc
- And a lot of other data. Again everything is timestamped so can be watermarked easily.

The plan:

The plan is to build my own local data platform where i upload / send the data and it runs PySpark transformations on the data through a medallion architecture and i can use the gold/platinum datasets for visualisations and getting insights into my behaviour patterns.

What I currently have:
- An old laptop that i have converted to a ubuntu server (with GUI) to be used for storage and compute.
- Samba is setup and I have automated data transfer to the exposed “incoming” folder using automations that can run automatically or on demand.
- The repo is set up and a cron job inside the repo runs the pipelines on schedule on the server.
- Current architecture: Data collection (csv or json) -> auto save via automations to “/incoming/health” or “/incoming/financial” or “/incoming/gps” etc etc folders. It works and I am happy to proceed with the next step which is ingestion.

What i want to ask is:
- Should the next step be “/incoming/*” -> “/raw/health”, “raw/workout”, “/raw/financial” folders, which would be an exact replica of the incoming folder for redundancy purposes or
- Go straight to “/bronze/workouts” “/bronze/transactions” or “bronze/heart_rate” and skip raw since that will be one less step to manage??

For anyone thinking this is overkill for my requirements, i am doing it purely for learning purposes. I have been a data engineer for about a year now (but have 6ish years experience working with all sorts of data, plus personal projects) and wanted to learn something new with something that I own while avoiding generic off the shelf project ideas.

Any ideas appreciated. Thanks


r/dataengineering • • 7d ago

Open Source Open source entity resolution on Databricks

4 Upvotes

Hello All,

Zingg lead dev here! Excited to have our community edition available as an app on Databricks marketplace. Took a lot of effort from our tiny team, and we are glad we finally made it! Hoping it helps folks here build massive scale identity resolution systems, master data management, composable CDPs and agentic flows needing identity graphs.

https://marketplace.databricks.com/details/6e0e1411-65c8-44ae-bdfa-ab706770ac11/Zingg-Labs-Inc_Zingg-Entity-Resolution-and-MDM


r/dataengineering • • 7d ago

Open Source Tparquet - Parquet for time series. Tick data at about half their ZSTD size, lossless, and still Parquet.

Thumbnail
github.com
27 Upvotes

If you want to save some space on your market data hard drive vs parquet, we have a very small fork of parquet libraries/loaders that'll give you ~50% better compression (with pcodec), and we're also backing initiative to make it an official parquet compression codec.

It'll be actually a couple of bytes of change to switch over when it becomes officially supported.

All credit goes to Martin Loncaric, the author of pcodec, who's also the maintainer of the excellent reference implementation: https://github.com/pcodec/pcodec