Apache Spark Data Science for Large Scale Analytics Training Course

Course overview

Data science teams often prove a model or analytical method on a sampled dataset, then struggle to run the same work reliably across billions of records. Slow pandas workflows, poorly partitioned files, repeated joins, skewed keys and ungoverned feature pipelines can turn an apparently simple analysis into an expensive production bottleneck. This course equips practitioners to use Apache Spark for distributed exploration, feature engineering and machine learning workloads that must perform at organisational scale.

Participants build practical proficiency in PySpark, Spark SQL and Spark MLlib. They learn how Spark distributes data and computation; how to design DataFrame transformations and window calculations; how to read, write and optimise Parquet and Delta Lake tables; and how to diagnose inefficient jobs through execution plans and the Spark UI. The course also covers feature pipelines, model training and evaluation, MLflow experiment tracking, and the operational choices required to move a notebook workflow into a repeatable batch process.

Instructor-led demonstrations are followed by guided labs using a realistic large-scale customer analytics dataset. Participants progressively build a reusable churn-risk analytics pipeline: ingesting raw events, creating governed feature tables, training a classification model, comparing experiments and tuning Spark performance. They leave with documented PySpark notebooks, an ML pipeline design, optimisation findings and an implementation plan that can be adapted to a live data platform. A certificate of completion is awarded after the five-day course.

The course is designed for data professionals who already work with Python and SQL and now need to analyse, model or prepare data beyond the limits of a single machine. It also gives technical leads a concrete basis for setting coding, storage and performance practices for Spark-based data science work.

Course objectives

By the end of this course, participants will be able to:

• Build distributed data preparation workflows with PySpark DataFrames and Spark SQL
• Design partitioned Parquet and Delta Lake tables for efficient analytical access
• Apply joins, aggregations, window functions and pivot operations to large datasets
• Inspect Spark execution plans and Spark UI metrics to diagnose slow jobs
• Mitigate data skew, excessive shuffles and inefficient partitioning in Spark workloads
• Construct reproducible feature engineering and model training pipelines with Spark MLlib
• Track model parameters, metrics and artifacts using MLflow experiments
• Produce a documented Spark analytics pipeline and performance improvement plan

Benefits of attending

Target competencies

Who should attend

• Data Scientists — who need to train models and engineer features on datasets too large for local Python tools
• Data Engineers — who build Spark pipelines and need stronger analytical and machine learning workflow skills
• Machine Learning Engineers — who must operationalise distributed feature preparation and model evaluation
• Analytics Engineers — who create scalable transformation layers for reporting and advanced analytics
• Data Analysts — who use SQL and Python and need to work effectively with large distributed datasets
• Technical Data Science Leads — who define standards for Spark performance, experimentation and reusable pipelines

Requirements and prerequisites

Participants should be comfortable writing Python functions, working with lists and dictionaries, and using SQL SELECT statements, joins, GROUP BY and basic window functions. Experience with pandas or another tabular data library is useful because the course contrasts local and distributed processing. Participants should understand basic statistics and supervised machine learning concepts such as training data, features, labels and evaluation metrics. No prior Apache Spark, Databricks, cluster administration, Scala, Java or distributed-systems expertise is required. A working laptop able to access the supplied browser-based lab environment is needed.

Training methodology

The course combines focused instructor explanations with daily PySpark labs in a configured Spark environment. Participants inspect real query plans, run transformations against multi-file datasets, and compare inefficient and optimised job designs using Spark UI evidence. Short case discussions address storage layout, model reproducibility and handover between data science and engineering teams. Working in pairs or small groups, participants develop a churn-risk pipeline across the week. The final workshop converts their technical findings into a documented implementation and performance plan for a business workload.

Course outline

Day 1: Spark foundations for distributed data science

• Apache Spark architecture: driver, executors, clusters and lazy evaluation
• PySpark session configuration and notebook workflow
• Resilient Distributed Datasets versus DataFrames and when each applies
• Schema definition, data types and null-handling strategies
• Reading CSV, JSON and Parquet files with explicit options
• DataFrame selections, filters, expressions and column transformations
• Spark SQL views and interoperability between SQL and PySpark

Workshop: Load and profile a multi-source customer events dataset, producing a typed PySpark DataFrame and an initial data-quality summary.

Day 2: Scalable transformation and storage design

• Aggregations, rollups, cubes and grouped calculations
• Window functions for customer history, ranking and time-based features
• Join strategies: broadcast, sort-merge and shuffle hash joins
• Union, pivot and complex nested-data transformations
• Partitioning, bucketing and file-size considerations
• Parquet compression, predicate pushdown and column pruning
• Delta Lake tables, ACID transactions and time travel

Workshop: Create a curated Delta Lake feature table by joining transactions, customer records and event data, then validate its storage layout.

Day 3: Spark performance engineering

• Catalyst optimiser, Tungsten execution and adaptive query execution
• Logical plans, physical plans and explain plan interpretation
• Spark UI stages, tasks, storage and SQL tabs
• Shuffle mechanics, spill behaviour and network data movement
• Data skew detection and salting techniques for skewed joins
• Repartition, coalesce, cache and persist decision rules
• Executor memory, cores and partition-count tuning principles

Workshop: Profile a deliberately slow customer aggregation job in Spark UI and produce an optimised version with before-and-after runtime evidence.

Day 4: Machine learning pipelines at scale

• Feature assembly, indexing, encoding and scaling with Spark MLlib
• Train-validation splits and cross-validation for distributed datasets
• Classification algorithms: logistic regression and random forests
• Pipeline stages, parameter grids and reusable model objects
• Evaluation metrics including AUC, precision, recall and confusion matrices
• Class imbalance handling and threshold selection
• MLflow runs, metrics, artifacts and model registration concepts

Workshop: Train and evaluate a Spark MLlib churn classifier, logging competing model runs and metrics in MLflow.

Day 5: Production-ready Spark data science workflows

• Notebook-to-job refactoring and parameterised PySpark applications
• Batch orchestration dependencies and idempotent pipeline design
• Data quality checks, schema evolution and expectation rules
• Feature table versioning and training-serving consistency
• Model scoring patterns for batch inference
• Access controls, sensitive data handling and audit considerations
• Cost, reliability and monitoring measures for Spark workloads

Workshop: Complete a capstone design workshop that documents the churn pipeline architecture, operational controls, performance decisions and deployment actions.

Tools & standards covered

Apache Spark, Databricks, Delta Lake, MLflow

A typical training day

Live online deliveries follow the same structure in the East Africa Time zone, with shorter screen blocks and longer breaks.

What the fee includes

How you can take this course

Certification

Participants who complete the full five days receive the Skillset Development Certificate of Completion, stating the course title, course code, dates and delivery format — suitable for professional-development records and employer reimbursement.

Frequently asked questions