Data Wrangling with Python and R Training Course

Participants can inspect tabular datasets and identify structural issues before transformation.

• pandas DataFrame indexing and R tibble semantics compared
• Inspecting shape, dtypes, classes, memory use, and cardinality
• Vectorised operations replace row-wise loops in both languages
• Boolean masks and dplyr filters express compound selection rules
• Column names standardised using snake_case naming conventions
• Type coercion failures isolated before downstream transformations

Participants can import common business data formats with controlled parsing rules and schemas.

• CSV parsing controlled with delimiters, quoting, headers, and encodings
• read_csv and pandas read_csv configured with explicit column types
• Excel workbooks imported from selected sheets, ranges, and headers
• JSON records normalised from nested objects into tabular fields
• Parquet files selected for typed, columnar analytical storage
• Malformed records quarantined with source-file and row-number evidence

Participants can build readable transformation pipelines that convert business rules into code.

• Tidy data principles map variables, observations, and values consistently
• Pipeline verbs filter, mutate, arrange, select, and summarise records
• Method chains retain readable transformation order and intermediate intent
• Reusable functions parameterise repeated cleaning rules and assumptions
• Grouped aggregations calculate counts, rates, quantiles, and weighted means
• Business rules translated into transparent transformation specifications

Participants can profile data-quality issues and apply defensible treatments for missing and anomalous values.

• Missingness profiles distinguish nulls, blanks, sentinels, and absent rows
• MCAR, MAR, and MNAR assumptions assessed before imputation
• Rule-based imputation separated from analytical estimation and prediction
• Forward fill, backward fill, and interpolation applied with safeguards
• Outliers flagged using IQR fences, z-scores, and domain thresholds
• Cleaning decisions documented in a data-quality issue log

Participants can standardise text fields and extract usable values from semi-structured descriptions.

• Regular expressions extract identifiers, amounts, dates, and coded fields
• stringr and pandas string methods standardise text values
• Unicode normalisation resolves accented characters, symbols, and compatibility forms
• Whitespace, punctuation, and case rules prepared for matching
• Named capture groups parse semi-structured operational descriptions
• Text transformations tested against edge cases and malformed inputs

Participants can parse, validate and derive reporting periods from date and timestamp data.

• ISO 8601 parsing distinguishes dates, times, durations, and intervals
• Timezone-aware timestamps converted without losing local business meaning
• lubridate and pandas datetime methods derive reporting periods
• Fiscal calendars map transactions to accounting weeks and quarters
• Duplicate timestamps and daylight-saving transitions handled explicitly
• Date validation tests identify impossible sequences and future records

Participants can combine datasets safely and identify unmatched or duplicated records.

• Primary keys, natural keys, and surrogate keys assessed for reliability
• One-to-one, one-to-many, and many-to-many joins diagnosed before merging
• Join cardinality checks prevent duplicated revenue, customer, or transaction totals
• Anti-joins identify unmatched records requiring remediation or exception handling
• Fuzzy matching combines blocking keys with similarity-score thresholds
• Master-data crosswalks preserve source identifiers and match confidence

Participants can reshape reporting extracts, survey data and nested records into usable tables.

• Pivot longer and pivot wider reshape reporting extracts predictably
• Multi-value columns separated using delimiters, positions, and regular expressions
• Nested JSON arrays exploded while retaining parent record identifiers
• Wide survey responses converted into analysis-ready respondent-level tables
• Hierarchical headers reconstructed into usable variable names
• Reshaping steps preserve record counts and aggregation grain

Participants can assemble an initial end-to-end pipeline for a cleaned operational dataset.

• Source-to-output pipeline assembles a cleaned operational dataset
• Data dictionary defines fields, units, permitted values, and ownership
• Reconciliation totals compare transformed outputs with source control totals
• Exception tables isolate rejected records and stated failure reasons
• Functions organise ingestion, cleaning, validation, and publication stages
• Peer review examines assumptions, readability, and reproducibility evidence

Participants can standardise categories and apply controlled mappings to inconsistent source values.

• Categorical levels harmonised across systems using controlled value mappings
• Reference tables replace hard-coded categories and business labels
• Frequency profiling reveals rare levels, misspellings, and category drift
• Regular expressions classify free-text records into reporting categories
• Rule precedence resolves overlapping classification conditions explicitly
• Mapping coverage measured with matched, unmatched, and ambiguous rates

Participants can design and automate data-quality checks that detect transformation failures.

• Validation rules specify completeness, uniqueness, validity, consistency, and timeliness
• Assertion-based checks test expected values before dataset publication
• Schema checks enforce expected columns, types, ranges, and nullability
• Referential integrity tests compare foreign keys with approved dimensions
• Unit tests protect transformations against regressions after source changes
• Quality metrics published as pass rates, exceptions, and trend counts

Participants can retrieve, flatten and control data from SQL queries and REST APIs.

• Parameterized SQL retrieves only required fields and filtered records
• Window functions calculate rankings, lags, and partitioned running totals
• REST API requests handle pagination, authentication, retries, and rate limits
• JSON responses flattened into typed tables with provenance fields
• Incremental extraction uses watermark timestamps and idempotent loading logic
• API failures logged with status codes, payload context, and retry outcomes

Participants can select efficient processing methods for datasets that strain desktop memory and runtime.

• Chunked CSV processing handles files exceeding available memory
• Arrow and Parquet reduce storage size and read times
• data.table and pandas operations profiled for bottlenecks
• Vectorisation, joins, and grouping choices benchmarked on representative data
• Memory copies identified when selecting, assigning, and concatenating tables
• Partitioned datasets support selective reads by date or business unit

Participants can automate recurring transformations and retain execution evidence for review.

• Parameterized scripts separate configuration values from transformation code
• Quarto reports combine code, outputs, narrative, and source references
• Command-line arguments support scheduled runs for recurring data feeds
• Logging records row counts, warnings, runtime, and transformation milestones
• Error handling stops publication when critical validation rules fail
• Git branches and pull requests support reviewed pipeline changes

Participants can exchange data and selected functions between Python and R workflows.

• reticulate calls Python libraries from R analytical workflows
• rpy2 exposes R functions within Python-driven processing scripts
• Parquet exchange preserves types better than comma-separated files
• Shared data contracts align field definitions across language boundaries
• Package environments isolated so Python and R dependencies remain reproducible
• Tool selection considers library maturity, team skills, and deployment constraints

Participants can apply practical controls for sensitive data, provenance and retention during wrangling.

• Personally identifiable information classified before extraction and transformation
• Data minimisation removes fields unnecessary for defined analytical purposes
• Hashing and tokenisation compared for pseudonymising stable identifiers
• Role-based access rules applied to raw and processed datasets
• Provenance records source, transformation, owner, and publication date
• Retention rules flag extracts requiring deletion or restricted storage

Participants can prepare documented datasets that analysts, modellers and reporting teams can use confidently.

• Analysis-ready tables define grain, measures, dimensions, and observation windows
• Feature engineering derives ratios, lags, cohorts, and rolling measures
• Denominator checks prevent misleading rates and percentage calculations
• Sampling frames assessed for duplicates, exclusions, and population coverage
• Visual profiling uses distributions, missingness plots, and relationship checks
• Hand-off datasets include metadata, quality results, and known limitations

Participants can define a controlled wrangling solution from requirements through acceptance criteria.

• Requirements translated into source inventory, transformation map, and acceptance criteria
• Stakeholder questions prioritised against available data and quality constraints
• Pipeline architecture selects ingestion, storage, validation, and output stages
• Review checklists test correctness, auditability, maintainability, and privacy
• Technical findings communicated through exception summaries and data dictionaries
• Capstone scope fixed using realistic source files and delivery criteria

Participants can build, review and present a complete documented pipeline that converts messy source data into a validated analysis-ready dataset.

• Complete pipeline ingests, cleans, validates, joins, and publishes source data
• Python or R implementation documents assumptions, dependencies, and execution steps
• Quality report quantifies exceptions, reconciliation results, and unresolved risks
• Peer review tests logic, reproducibility, performance, and data protection controls
• Presentation explains design choices, findings, limitations, and hand-off requirements
• Final package includes code, data dictionary, validation log, and output dataset