Cloud Data Platform Engineering for Data Engineers Training Course

Course overview

Data engineers are increasingly expected to build more than individual pipelines: they must create reliable cloud data platforms that ingest operational data, manage changing schemas, support analytics and machine learning workloads, and remain secure and cost-controlled as usage grows. This requires sound decisions about storage layers, compute patterns, orchestration, transformation, observability, access control and infrastructure automation—not just writing SQL or Spark jobs. Poor platform design produces duplicated datasets, brittle schedules, uncontrolled cloud spend and uncertain data quality.

This two-week course develops the engineering practices required to design and operate a production-oriented cloud data platform. Participants work through lakehouse architecture, object storage and table design, batch and streaming ingestion, medallion data layers, distributed processing, dbt transformations, Apache Airflow orchestration, data testing, lineage, monitoring, IAM controls and Terraform-based infrastructure provisioning. They learn to select workload-appropriate compute, implement incremental pipelines, handle late-arriving data, define service-level objectives and investigate pipeline failures using operational evidence.

Instructor-led technical sessions are paired with guided labs using Databricks, Apache Airflow, dbt and Terraform. A running case study follows a growing digital business whose event, customer and transaction data must be made usable for reporting and data products. Teams build and defend a deployable platform design, including architecture diagrams, data contracts, pipeline code patterns, quality checks, orchestration design, monitoring requirements and an implementation backlog. Participants leave with a documented cloud data platform blueprint that can be adapted to their own organisation.

The course is best suited to practicing data engineers and technical specialists moving from pipeline delivery into platform engineering, modernisation or operational ownership roles.

Course objectives

By the end of this course, participants will be able to:

• Design a layered cloud lakehouse architecture for batch, streaming and analytical workloads
• Model object storage zones, table layouts and partitioning strategies for scalable data access
• Build incremental transformation pipelines with dbt models, tests and documentation
• Orchestrate dependency-aware workflows using Apache Airflow DAGs, retries and service-level checks
• Implement distributed processing patterns for schema evolution, late data and idempotent loads
• Provision repeatable platform infrastructure using Terraform modules and environment variables
• Define data observability controls including freshness, volume, quality and pipeline failure alerts
• Produce a cloud data platform blueprint with target architecture, operating model and delivery backlog

Benefits of attending

Target competencies

Who should attend

• Data Engineers — who need to design and operate resilient cloud pipelines rather than isolated data jobs
• Senior Data Engineers — who are taking ownership of platform standards, reliability and technical design
• Analytics Engineers — who need dependable transformation, testing and deployment practices across shared data models
• Data Platform Engineers — who build the infrastructure, orchestration and governance foundations used by data teams
• Cloud Engineers — who support data workloads and need to understand lakehouse architecture, data access and workload patterns
• Technical Data Leads — who must assess platform trade-offs and plan a practical modernisation roadmap

Requirements and prerequisites

Participants should have practical experience writing SQL and building or maintaining data pipelines. They should understand relational tables, joins, basic data modelling, files such as CSV or Parquet, and the purpose of APIs or scheduled jobs. Familiarity with Python, Git and a cloud provider console is strongly recommended, as labs include configuration and code review; participants do not need to be expert software developers. Prior use of Databricks, Apache Airflow, dbt or Terraform is not required. This is not a beginner course in SQL, cloud fundamentals or programming.

Training methodology

The course combines instructor-led architecture briefings with daily build-and-review labs based on a shared cloud data platform case. Participants configure storage and table patterns, develop dbt transformations, inspect Airflow DAG behaviour, define monitoring controls and review Terraform plans in small technical teams. Architecture trade-offs are debated using workload volumes, recovery targets, security constraints and cost assumptions rather than abstract examples. Each day closes with an applied exercise, and the final day converts the working design into a platform blueprint and sequenced implementation plan for workplace use.

Course outline

Day 1: Cloud data platform architecture

• Cloud data platform reference architectures
• Lakehouse, warehouse and data mesh operating patterns
• Workload classification for batch, streaming and serving
• Data product boundaries and ownership models
• Non-functional requirements for data platforms
• Platform control plane versus data plane
• Architecture decision records for data engineering

Workshop: Create a target-state architecture and decision record for a retailer moving from scheduled database extracts to a cloud data platform.

Day 2: Storage, tables and data layout

• Object storage zones and naming conventions
• Parquet file format and compression choices
• Delta Lake table transaction semantics
• Partitioning, clustering and file-size management
• Bronze, silver and gold data layer design
• Schema evolution and schema enforcement
• Data retention, archival and recovery requirements

Workshop: Design storage zones and create a table layout specification for event, customer and transaction datasets.

Day 3: Ingestion engineering patterns

• Batch extraction patterns and watermarking
• Change data capture concepts and ordering challenges
• API ingestion with rate-limit and retry handling
• Streaming event ingestion fundamentals
• Idempotency keys and duplicate prevention
• Late-arriving data and backfill procedures
• Data contracts for source-system integration

Workshop: Develop an ingestion design that handles API failures, duplicate events, schema changes and late transaction records.

Day 4: Distributed processing with Databricks

• Databricks workspace and compute configuration
• Spark execution concepts for data engineers
• Notebook, job and workflow execution patterns
• Incremental merge operations in Delta Lake
• Join strategies and shuffle reduction
• Handling skewed keys and uneven partitions
• Compute autoscaling and job cost controls

Workshop: Build and tune an incremental Databricks processing job that merges daily transaction updates into a curated table.

Day 5: Transformation engineering with dbt

• dbt project structure and model layering
• SQL models, references and dependency graphs
• Incremental models and snapshot strategies
• Generic and singular data tests
• Source definitions and freshness checks
• Documentation generation and model lineage
• Deployment environments and Git workflows

Workshop: Create a dbt transformation project with staging and mart models, source tests, freshness checks and generated documentation.

Day 6: Workflow orchestration and reliability

• Apache Airflow DAG design principles
• Task dependencies, scheduling and catchup behaviour
• Retries, timeouts and failure callbacks
• Backfills and safe rerun strategies
• Dataset-aware scheduling concepts
• Parameterisation and environment configuration
• Service-level objectives for pipeline operations

Workshop: Implement an Airflow DAG design for ingestion, transformation and publication, including retries, alerts and backfill controls.

Day 7: Data quality, observability and incident response

• Data quality dimensions and acceptance thresholds
• Freshness, volume and distribution monitoring
• Pipeline logs, metrics and traceable run history
• Lineage for impact analysis and root-cause investigation
• Alert routing and incident severity criteria
• Runbooks for failed data pipelines
• Post-incident reviews and corrective actions

Workshop: Create an observability specification and incident runbook for a failed executive reporting pipeline.

Day 8: Security, governance and cost engineering

• IAM roles and least-privilege access design
• Secrets management and credential rotation
• Row-level and column-level data access controls
• Sensitive data classification and masking patterns
• Audit logging and evidence retention
• Cloud cost allocation tags and budgets
• FinOps metrics for data platform workloads

Workshop: Produce an access-control matrix, sensitive-data handling design and cost-control checklist for the case platform.

Day 9: Infrastructure as code and platform delivery

• Terraform state, providers and resource lifecycle
• Reusable Terraform modules for data platforms
• Variables, outputs and environment promotion
• Infrastructure plan review and change control
• CI/CD pipeline stages for data assets
• Git branching and pull-request quality gates
• Platform release and rollback strategies

Workshop: Draft a Terraform module structure and deployment workflow for provisioning development, test and production data environments.

Day 10: Platform blueprint and implementation roadmap

• Platform capability maturity assessment
• Target architecture synthesis and trade-off review
• Migration sequencing for legacy pipelines
• Operational ownership and support model
• Platform KPIs and engineering scorecards
• Risk register and dependency management
• Ninety-day implementation roadmap development

Workshop: Present a cloud data platform blueprint, operating model and ninety-day implementation roadmap for peer and instructor review.

Tools & standards covered

Databricks, Apache Airflow, dbt, Terraform

A typical training day

08:30 – 10:30	First session
10:30 – 10:45	Refreshment break
10:45 – 12:30	Second session
12:30 – 13:30	Lunch and networking
13:30 – 15:00	Third session
15:00 – 15:15	Refreshment break
15:15 – 16:30	Workshop and daily review

Live online deliveries follow the same structure in the East Africa Time zone, with shorter screen blocks and longer breaks.

What the fee includes

How you can take this course

Certification

Participants who complete the full five days receive the Skillset Development Certificate of Completion, stating the course title, course code, dates and delivery format — suitable for professional-development records and employer reimbursement.

Frequently asked questions