How to Build Scalable Data Pipelines on AWS

Modern data teams rarely struggle with whether they have data. The real challenge is handling the volume, speed, and variety of that data without creating brittle systems, runaway cloud costs, or constant operational headaches.

That was the core theme of a technical talk by Santos Dulam, an engineering leader working with large-scale financial and retirement data. His focus was not on hype, but on a practical question many data professionals face: How do you design AWS data pipelines that scale, stay governable, and remain flexible enough for analytics and AI use cases?

For mid-level engineers looking to grow into data engineering or AI engineering, this talk offers a useful architectural mindset. It connects AWS services like S3, Glue, Kinesis, EMR, Athena, Redshift, Lake Formation, and Step Functions into a coherent operating model. More importantly, it explains why some decisions matter more than others.

This article distills those lessons into a reader-friendly framework, while adding context on tradeoffs, implementation strategy, and what this means for real-world career growth.

Why scalable pipelines are still hard

The speaker frames the problem around a familiar big-data triad: volume, velocity, and variety. That part is standard. What makes the talk more useful is the next step: he connects those pressures to the modern expectations placed on data platforms.

Today, a pipeline is rarely built just for one dashboard. It must often support:

Historical reporting
Operational analytics
Data governance and access control
Cost management
Downstream machine learning
Emerging AI and LLM use cases
Reliability expectations close to zero downtime

That combination is what makes pipeline design difficult. A system that works for nightly ingestion may fail under real-time demands. A platform optimized for raw storage may become painful for governed self-service analytics. And a setup that is fast to build can become expensive or unmanageable at scale.

The central idea from the talk is simple but powerful: design the pipeline as a set of decoupled layers rather than a tightly connected monolith.

The four-layer AWS pipeline blueprint

The architecture presented in the session follows a clear progression:

Ingestion
Storage
Transformation
Analytics and serving

This is not new in theory, but the practical value comes from how each layer is separated so compute, storage, and consumption can scale independently.

1. Ingestion: choose the pattern that matches the business need

The talk emphasizes a point many teams learn too late: not every use case needs streaming.

That matters because engineers often overdesign early architectures around real-time processing. In practice, Santos argues that batch ingestion can solve most business problems, and for many organizations, that is the right default.

Common ingestion approaches mentioned include:

Batch loads for scheduled extraction
AWS DMS for change data capture from databases
Kinesis for streaming or near-real-time events
MSK (Managed Kafka) for event-driven architectures
File drops into S3 with event-based triggers

His broader message is worth underlining: build for the actual latency requirement, not for what sounds technically impressive.

When batch is enough

Batch is often the better choice when:

Reports are refreshed daily or hourly
Historical trend analysis matters more than instant updates
Data volumes are high but response urgency is low
Cost control is important
Simpler operations are preferred

When streaming makes sense

Streaming becomes more defensible when:

Users need sub-second or near-real-time insights
Event reactions are time-sensitive
You are supporting use cases like live leaderboards, fraud detection, or emergency systems
Business value truly depends on immediate processing

For aspiring data engineers, this is a strong design lesson: good architecture starts with service-level expectations, not tool preferences.

Storage: why S3 remains the center of gravity

The storage layer in the talk is built around Amazon S3 as the data lake foundation. That is expected, but the underlying rationale matters.

In older Hadoop-style environments, scaling storage often meant scaling the cluster itself. That introduced friction, rebalancing delays, and expensive infrastructure overhead. By separating storage from compute, AWS makes it possible to:

Store large volumes cheaply
Process data with elastic services only when needed
Avoid overprovisioning
Reduce operational complexity

This decoupling is one of the most important architectural advantages of the cloud-native approach described in the session.

Think in data lifecycle stages, not just buckets

The talk also references a medallion-style lifecycle, though with AWS-centric implementation choices:

Raw/Bronze: source-aligned ingested data
Refined/Silver: cleaned, validated, standardized data
Curated/Gold: business-ready datasets for reporting, applications, and ML

This pattern matters because it supports both reliability and reuse.

A common mistake in early-career data work is transforming directly from source to final dashboard tables. That works briefly, then collapses under changing requirements. Layered zones create better separation between:

what the source system produced,
what the platform validated,
and what the business actually consumes.

That separation also makes testing, lineage, and debugging much easier.

Transformation: Glue for elasticity, EMR for heavier workloads

The transformation section is one of the most practical parts of the talk. The speaker describes a blended strategy using AWS Glue for most ETL work and EMR for larger-scale or more specialized processing.

Why Glue becomes the default

According to the session, Glue handled the majority of workloads because it offers:

Serverless execution
Elastic scaling
No cluster management
Cost efficiency for many scheduled jobs
Integration with Spark and the AWS ecosystem

He notes that Glue jobs can scale from small workloads to much larger ones without requiring engineers to manage infrastructure directly. For organizations running large numbers of transformations on varying schedules, this is a major operational win.

When EMR still matters

EMR enters the picture when teams need:

Larger-scale Spark workloads
Heavy memory usage
More control over cluster behavior
Transient clusters for research or batch-intensive workloads

The talk suggests EMR was used less frequently, especially for weekly research-oriented jobs, while Glue supported a much larger estate of production jobs.

That aligns with a broader cloud pattern: use serverless first unless control requirements justify managed clusters.

PySpark over Scala: a practical talent decision

One useful hiring and team-design insight from the talk is the preference for PySpark over Scala. The reason was not purely technical. It was organizational.

PySpark was chosen because:

It had matured enough to be production-viable
Python skills were easier to align across teams
Developers could stay in one language ecosystem for multiple tasks

This is a career-relevant point. For professionals moving into data engineering, Python plus SQL plus Spark concepts remains one of the strongest skill combinations to build.

Query and analytics: Athena vs. Redshift is a usage decision

One of the clearest decision frameworks in the talk concerns where to serve curated data.

Rather than pushing everything into a warehouse by default, the team uses a consumption-based strategy:

Athena for less frequently queried datasets sitting in S3
Redshift for datasets that are queried often and need more warehouse-style performance

This is a mature pattern because it avoids a common anti-pattern: loading every dataset into an expensive warehouse regardless of actual demand.

Athena works well when:

Data is queried occasionally
S3 is already the system of record
Dashboards can cache results
You want to minimize unnecessary warehouse usage

Redshift works well when:

Data is queried regularly
Business users need faster repeated access
Warehouse semantics are useful
Usage justifies the platform cost

The speaker also notes that dashboard tools can reduce repeated query costs through caching mechanisms. That point matters because teams often forget to account for BI behavior when modeling infrastructure spend.

The larger lesson: storage and serving choices should reflect query frequency, not habit.

Metadata and governance: the hidden foundation for AI readiness

A particularly valuable part of the talk is the emphasis on metadata. Many engineers think of metadata catalogs as compliance overhead. Here, it is presented as an enabler for future intelligence.

The team uses AWS Glue Data Catalog for metadata and schema management, including information about:

Tables
Columns
Partitions
Source-aligned descriptions

They also carry over descriptive metadata from source systems so column and table meaning is preserved.

Why does that matter?

Because once organizations begin experimenting with AI or LLM-based querying, semantic context becomes essential. If your catalog knows only table names and data types, AI systems have little business meaning to work with. If your metadata includes column definitions, descriptions, and relationships, those tools become much more useful.

This is one of the strongest forward-looking insights in the talk:
good metadata work done today reduces AI implementation friction tomorrow.

Schema evolution and file format strategy

The talk touches on schema evolution and format choices, both of which are highly relevant for scalable design.

Schema evolution

For growing datasets, schema changes are inevitable. The speaker points to support for adding columns dynamically in modern data lake patterns, especially with Spark-based pipelines.

That suggests an important engineering principle: assume your schemas will change. Designs that break on every source update do not scale organizationally, even if they scale technically.

Parquet as the primary format

Most data in the architecture is stored in Parquet, which makes sense because Parquet offers:

Columnar storage
Better compression
Efficient reads for analytics workloads
Strong compatibility across AWS analytics tools

The team also discusses partitioning heavily and chooses partitions based on how the data is read downstream, not just how it arrives.

That is exactly the right rule.

Bad partitioning strategies can create small-file issues, poor parallelism, and expensive scans. Good partitioning improves Spark execution, query efficiency, and cost control.

Iceberg as an emerging option

The talk briefly mentions experimentation with Apache Iceberg for update-heavy use cases. However, the speaker is careful not to oversell it, noting operational concerns like file management overhead.

That nuance is important. Iceberg can be a strong fit for mutable lakehouse patterns, but it is not automatically the right answer for every dataset. Professionals should see this as a reminder to evaluate table formats based on update patterns and operational tolerance, not trendiness.

Orchestration and observability: where reliability is won or lost

A pipeline is only as valuable as its ability to run consistently. The session treats orchestration and monitoring as first-class concerns, which is exactly right.

Orchestration choices

The main orchestration options mentioned are:

AWS Step Functions
S3 event triggers
Scheduled jobs through Glue
Some use of Airflow, though it was considered more expensive in this context

The preference for Step Functions reflects a common AWS-native strategy: keep orchestration within the cloud platform when possible, especially for event-driven or service-integrated workflows.

Observability practices

The team uses operational tooling such as:

CloudWatch for alerts and monitoring
SNS for notification routing
Integration with Jira for incident tracking
Glue job bookmarks to track already-processed files

This is where the talk becomes especially practical. Silent failures are called out as dangerous, and rightly so. Many pipelines do not fail loudly enough. A missed run, partial load, or broken transformation may not surface until a stakeholder notices bad numbers.

The operational maturity described here includes:

automatic alerting,
required acknowledgment,
ticket creation,
and explicit remediation workflows.

For engineers trying to advance their careers, this is a critical mindset shift: senior-level data engineering is not just about building pipelines; it is about operating them reliably.

Security and compliance: not optional in real-world pipelines

Because the underlying business domain involves financial and participant data, the talk pays careful attention to governance and compliance.

Key services and practices mentioned include:

Lake Formation for row- and column-level access control
KMS for encryption at rest and in transit
Macie for detecting sensitive data such as PII
Group-based access management integrated with enterprise identity controls

This matters beyond regulated industries. Even if your organization is not in finance or healthcare, the trend is clear: data engineering increasingly overlaps with privacy, access control, and auditability.

That means professionals who understand governance tooling are more valuable than those who only know transformation code.

Serving multiple consumers from one platform

One of the strongest architectural patterns in the talk is that the pipeline does not stop at dashboards.

Curated data is served to multiple destinations, including:

BI tools
Internal and external applications
Marketing systems
DynamoDB for application access
Redis for low-latency caching
OpenSearch for search and vector-style use cases
Downstream publishers and topics
ML and Bedrock-based AI workloads

This is a modern platform mindset. Data is not just for analysts anymore. It is also an application asset, an AI asset, and an operational asset.

For aspiring AI engineers, this is especially relevant: AI systems depend on well-organized data platforms far more than many newcomers realize.

What this architecture gets right

The talk presents a broad AWS-centric architecture, and several aspects stand out as especially strong.

It prioritizes decoupling

This is likely the most important design principle in the session. Decoupled ingestion, storage, transformation, and serving layers make systems easier to scale, debug, and replace incrementally.

It avoids unnecessary real-time complexity

The insistence that most business problems can be solved with batch is refreshingly pragmatic. Teams that learn this early avoid costly overengineering.

It treats metadata as strategic infrastructure

This is one of the most future-proof ideas discussed. Metadata is not just for governance. It is a prerequisite for trustworthy self-service and AI-assisted analytics.

It links architecture to cost discipline

The talk repeatedly ties design choices back to cost optimization through:

right-sized compute,
storage tiering,
partitioning,
file formats,
and selective use of warehouse resources.

That is exactly how mature cloud data platforms are managed.

Where teams should add caution

Even strong architectures come with caveats. A few areas deserve careful consideration.

Too many tools can create cognitive overhead

The architecture spans many AWS services. That flexibility is powerful, but for small teams it can become operationally heavy. Standardization matters. If every pipeline uses a different pattern, maintainability declines quickly.

Glue-first works well, but frameworks still matter

The talk mentions internal wrappers that allow SQL-heavy developers to work more productively with Glue. That is smart, but it also signals a larger truth: serverless tools alone do not remove the need for engineering standards, libraries, testing, and conventions.

Data lake success depends on discipline

Raw, refined, and curated zones sound elegant on slides. In practice, they require:

naming standards,
retention policies,
ownership rules,
schema controls,
and consumer agreements.

Without those, "data lake" can still become "data swamp."

Key Takeaways

Start with business latency needs before choosing batch or streaming. Most teams do not need real-time pipelines for most use cases.
Design pipelines as decoupled layers for ingestion, storage, transformation, and consumption to improve scalability and maintainability.
Use S3 as the core data lake layer and scale compute independently with Glue or EMR instead of tying storage to cluster infrastructure.
Adopt a raw-refined-curated data lifecycle so source preservation, standardization, and business-facing models stay cleanly separated.
Choose Athena or Redshift based on query frequency, not by defaulting every dataset into a warehouse.
Invest in metadata early. Rich table and column descriptions make governance better and future AI use cases far easier to support.
Prefer Parquet and deliberate partitioning strategies to improve performance and reduce query and processing costs.
Treat observability as part of the product, with alerts, incident routing, and explicit failure handling.
Use Lake Formation, KMS, and sensitive-data detection controls to build security and compliance into the platform from the start.
Build for multiple consumers, including dashboards, applications, and AI systems, rather than assuming analytics is the only endpoint.

A practical roadmap for engineers building similar systems

If you are trying to apply these lessons in your own work, a sensible progression would look like this:

Phase 1: Build a minimal but scalable foundation

Land source data in S3
Define raw and refined layers
Use Glue for scheduled transformations
Store outputs in Parquet
Query curated data with Athena

Phase 2: Add governance and operational maturity

Register datasets in Glue Data Catalog
Apply access policies with Lake Formation
Set up CloudWatch alerts and ticketing
Define retention and tiering policies in S3

Phase 3: Optimize for consumption

Move frequently queried datasets to Redshift only when justified
Improve partitioning strategies
Add application-serving stores like DynamoDB or Redis where needed
Tune job sizing and execution patterns

Phase 4: Prepare for AI-native analytics

Improve metadata descriptions and lineage
Standardize business-friendly curated datasets
Make semantic context available for AI-assisted querying
Add evaluation and guardrail processes for LLM-based workflows

This sequence reflects the deeper message of the talk: scalable pipelines are not built by stacking tools; they are built by making disciplined architectural choices layer by layer.

Final thoughts

The strongest takeaway from this AWS pipeline discussion is not any single service. It is the operating philosophy behind the architecture.

A scalable data platform should be:

modular,
observable,
cost-aware,
governable,
and ready for both analytics and AI.

That is what separates a collection of cloud services from a real data engineering system.

For professionals moving into data engineering or AI engineering, this is exactly the kind of thinking that matters. Knowing AWS tools is useful. Knowing how to combine them into a resilient platform with clear tradeoffs is what moves you into higher-impact roles.

As the speaker suggests, the goal is to move from raw data to decision-making faster. But the hidden requirement is just as important: do it in a way that your team can sustain, scale, and trust.

Source: "From Ingestion to Insights: Building Robust Data Pipelines in AWS by Santosh Durgam" - Kevin Feasel, YouTube, Jun 28, 2026 - https://www.youtube.com/watch?v=DccHTFJdjb8