Apache Spark Data Analytics with PySpark Training Course

Course overview

Teams often have data spread across transaction systems, log files, APIs and cloud storage, yet struggle to turn large or messy datasets into repeatable analysis. Analysts and engineers may be comfortable with Python or SQL on small files but encounter slow processing, memory limits, inconsistent joins and unreliable pipelines when data volumes increase. This five-day Apache Spark Data Analytics with PySpark course equips participants to use distributed processing deliberately: selecting appropriate transformations, managing partitions, validating results and producing analysis that can be rerun rather than rebuilt manually.

Participants learn the Spark execution model and use PySpark DataFrames and Spark SQL to ingest, clean, join, aggregate and analyse structured and semi-structured data. The course covers schemas, null handling, date and text functions, window functions, user-defined functions, Parquet storage, partitioning, caching, query plans and performance troubleshooting. Participants also build analytical pipelines that combine raw operational data with reference data, calculate business measures, identify data-quality exceptions and write curated datasets for downstream reporting or machine-learning work.

Instruction combines instructor demonstrations with guided coding labs in JupyterLab and a realistic data analytics case study. Each participant develops a PySpark analytics notebook containing reusable ingestion, transformation, validation and reporting steps, along with a short optimisation rationale based on Spark execution plans and partitioning choices. This tangible artefact gives both the participant and their manager a practical starting point for applying Spark to an internal reporting, customer, operational or digital-data use case.

The course is suited to data analysts, BI professionals, data engineers and Python or SQL practitioners who need to analyse data beyond the limits of desktop tools. It starts with core Spark concepts and progresses to intermediate analytical patterns, making it appropriate for professionals moving into distributed data processing as well as teams standardising existing PySpark work.

Course objectives

By the end of this course, participants will be able to:

• Explain Spark driver, executor, partition and shuffle behaviour when designing a PySpark workload
• Create schema-aware PySpark DataFrames from CSV, JSON and Parquet data sources
• Clean and standardise analytical datasets using PySpark column expressions, null handling and date functions
• Join, aggregate and filter distributed datasets with DataFrame APIs and Spark SQL
• Apply window functions to calculate rankings, running totals, period comparisons and deduplicated records
• Build a reusable PySpark analytics notebook with parameterised ingestion, transformation and validation steps
• Inspect Spark query plans and use partitioning, caching and broadcast joins to improve workload performance
• Publish curated Parquet datasets and a documented data-quality exception report for downstream use

Benefits of attending

Target competencies

Who should attend

• Data Analysts — who need to analyse larger datasets than Excel, pandas or desktop BI tools can reliably handle
• Data Engineers — who build and maintain PySpark transformations for data lakes, reporting layers and operational pipelines
• BI Developers — who prepare governed, performant datasets before semantic modelling and dashboard development
• Analytics Engineers — who need repeatable transformation logic, validation checks and curated analytical tables
• Python Developers — who are moving from single-machine data processing to distributed Spark workloads
• Data Science Practitioners — who need to prepare and explore high-volume feature and event data with PySpark

Requirements and prerequisites

Participants should be comfortable writing basic Python, including variables, functions, lists and conditional logic, and should understand tabular data concepts such as rows, columns, data types, filters and joins. Familiarity with SQL SELECT, WHERE, GROUP BY and JOIN statements is strongly recommended because Spark SQL is used throughout the course. Participants should have used a command line or notebook environment at least once. Prior Apache Spark, PySpark, distributed-systems or cloud-platform experience is not required. Complete programming beginners should first gain practical Python and SQL foundations before attending.

Training methodology

The five days alternate focused instructor-led explanations with live PySpark coding and structured lab work. Participants work in JupyterLab against realistic customer, product, transaction and event datasets, progressively building one analytical pipeline rather than isolated examples. The instructor reviews common errors such as schema drift, skewed joins and accidental driver-side processing, then guides participants through diagnosis using Spark plans and metrics. Small-group discussions compare transformation and storage choices. The final workshop converts the case-study notebook into an application plan for a participant-selected workplace dataset.

Course outline

Day 1: Spark foundations and DataFrame workflows

• Apache Spark architecture: driver, executors, clusters and lazy evaluation
• PySpark session setup and notebook-based development workflow
• Resilient distributed datasets versus Spark DataFrames
• DataFrame schemas, column types and explicit schema definition
• Reading CSV, JSON and Parquet files with PySpark
• Core DataFrame transformations and actions
• Spark SQL temporary views and introductory analytical queries

Workshop: Build an initial PySpark notebook that loads customer and transaction files, defines schemas and produces a first set of Spark SQL summary queries.

Day 2: Data preparation and quality control

• Column expressions with select, withColumn, when and otherwise
• Null detection, imputation and invalid-value treatment
• String, date, timestamp and numeric transformation functions
• Duplicate detection and record-level deduplication methods
• Standardising category values and reference-data lookups
• Data-quality rules and exception DataFrame design
• Writing reproducible transformation functions in PySpark

Workshop: Clean a flawed operational dataset and produce a curated DataFrame plus a separate exception report showing rejected or corrected records.

Day 3: Joins, aggregation and analytical patterns

• Join types, join keys and unmatched-record analysis
• GroupBy aggregations and conditional measures
• Pivoting and reshaping data for analytical outputs
• Window specifications using partitionBy and orderBy
• Ranking, running totals and moving-average calculations
• Lead and lag functions for period-over-period comparison
• User-defined functions and when built-in functions are preferable

Workshop: Create a customer-performance analysis that joins sales and reference datasets, calculates cohort measures and identifies top and declining accounts.

Day 4: Performance, storage and reliability

• Spark execution plans and the explain method
• Partitions, parallelism and the cost of data shuffles
• Repartitioning and coalescing for workload control
• Caching, persistence levels and cache lifecycle decisions
• Broadcast joins and strategies for small dimension tables
• Parquet columnar storage, compression and predicate pushdown
• Data skew symptoms and practical mitigation approaches

Workshop: Profile a deliberately slow PySpark job, interpret its execution plan and revise joins, partitions and storage choices to improve runtime.

Day 5: End-to-end Spark analytics delivery

• Structuring maintainable PySpark notebooks and reusable functions
• Parameterising file paths, dates and business rules
• Validation checks for row counts, schemas and reconciliation totals
• Publishing partitioned curated Parquet datasets
• Producing analytical extracts for BI and downstream modelling
• Documenting lineage, assumptions and transformation logic
• Workplace use-case planning and Spark solution design

Workshop: Complete an end-to-end case-study pipeline and present a PySpark notebook, curated Parquet output, validation results and workplace application plan.

Tools & standards covered

Apache Spark, PySpark, JupyterLab, Apache Parquet

A typical training day

Live online deliveries follow the same structure in the East Africa Time zone, with shorter screen blocks and longer breaks.

What the fee includes

How you can take this course

Certification

Participants who complete the full five days receive the Skillset Development Certificate of Completion, stating the course title, course code, dates and delivery format — suitable for professional-development records and employer reimbursement.

Frequently asked questions