Python Automation for Research Training Course

Participants can configure and use a controlled local Python environment for research work.

• Python interpreter versions and virtual environments for isolated project dependencies
• JupyterLab notebooks organised for exploratory analysis and documented findings
• Package installation with pip and requirements files for repeatable setups
• Project folders created with predictable names and relative file paths
• Command-line execution of Python scripts using arguments and exit codes
• Error tracebacks read to locate failing functions, files, and data fields

Participants can write readable Python code for common research data tasks.

• Variables, strings, numbers, booleans, and None values in Python
• Lists, dictionaries, tuples, and sets selected for research data structures
• Conditional logic used to classify records and handle exceptions
• For loops and comprehensions applied to repeated file processing
• Functions written with parameters, return values, docstrings, and type hints
• Exceptions handled with try, except, else, and finally blocks

Participants can load, inspect, and store research datasets in appropriate formats.

• CSV, Excel, JSON, and Parquet formats compared for research workflows
• pandas DataFrames inspected using shape, columns, dtypes, and sample records
• Encoding, delimiters, headers, and date formats diagnosed during file loading
• Data dictionaries linked to fields, codes, units, and collection methods
• File paths generated with pathlib for portable project execution
• Clean datasets exported with controlled formats and explicit index settings

Participants can clean and transform messy tabular research data using pandas.

• Missing values profiled with isna, counts, and documented treatment rules
• Text fields standardised using strip, case conversion, and replacements
• Dates parsed, validated, and derived into reporting periods
• Numeric fields converted with coercion and invalid-value inspection
• Duplicate records identified using keys, timestamps, and business rules
• Transformation steps chained with readable pandas method sequences

Participants can join, reshape, and aggregate datasets while protecting analytical integrity.

• Join keys assessed for uniqueness, completeness, and inconsistent coding
• pandas merge operations selected for one-to-one and one-to-many relationships
• Concatenation used to combine monthly extracts with consistent schemas
• Wide and long data structures converted with melt and pivot
• Groupby aggregations built for indicators, respondent segments, and periods
• Post-join row counts checked against expected source and output totals

Participants can implement explicit checks that identify unreliable research data before analysis.

• Validation rules specified for ranges, categories, formats, and required fields
• Assertions used to stop pipelines when critical assumptions fail
• Frequency tables inspected for impossible values and coding drift
• Cross-field checks detect inconsistent dates, totals, and respondent attributes
• Quality exceptions written to review files with record identifiers
• Data quality logs distinguish warnings, failures, corrections, and exclusions

Participants can automate routine descriptive analysis for research datasets.

• Descriptive statistics generated for numeric measures and grouped populations
• Categorical distributions calculated with counts, proportions, and missing categories
• Survey indicators computed with denominators, filters, and suppression rules
• Time series summaries created by month, quarter, and reporting cycle
• Reusable analysis functions accept datasets, variables, and grouping parameters
• Results tables formatted with labels, rounding, and consistent ordering

Participants can produce consistent charts that communicate research findings accurately.

• Matplotlib chart selection matched to comparisons, distributions, and trends
• Figure dimensions, titles, labels, legends, and annotations standardised
• Accessible colour choices applied to categorical and sequential data
• Small multiples used to compare locations, groups, and time periods
• Chart functions parameterised for repeated indicator reporting
• Publication-ready images exported with controlled resolution and file names

Participants can extract, clean, and classify structured information from unstructured text.

• Regular expression syntax applied to identifiers, dates, codes, and citations
• Capture groups extract named components from inconsistent text fields
• Text cleaning sequences remove whitespace, punctuation, and markup artifacts
• Keyword dictionaries classify records using transparent rule-based methods
• Pattern matching results reviewed for false positives and omissions
• Extraction rules stored externally for maintenance and peer review

Participants can collect permitted public web data responsibly and reproducibly.

• HTTP requests sent with headers, parameters, timeouts, and status checks
• HTML parsed with Beautiful Soup to extract tables and page elements
• Pagination handled through predictable URLs, links, and page counters
• robots.txt, terms of use, and rate limits assessed before collection
• Retries and delays implemented for temporary request failures
• Raw web responses stored with retrieval dates and source URLs

Participants can retrieve and structure research data from documented APIs.

• API documentation interpreted for endpoints, parameters, authentication, and limits
• JSON responses normalised into pandas tables with nested fields
• Query parameters constructed for dates, geographies, indicators, and filters
• Pagination tokens processed to retrieve complete result sets
• API credentials stored outside source code using environment variables
• Response metadata retained for provenance, updates, and reproducibility

Participants can automate routine processing of spreadsheets and research document outputs.

• Excel workbooks read and written with pandas and openpyxl
• Multiple worksheet extracts processed using controlled sheet-name rules
• Cell formats, formulas, and summary tables applied programmatically
• Word report sections generated from validated tables and chart files
• Template placeholders populated with research periods, findings, and caveats
• Output folders structured for draft, reviewed, and approved publications

Participants can organise research automation projects for repeatable and reviewable results.

• FAIR principles applied to findability, access, interoperability, and reuse
• Configuration files separate changing parameters from Python source code
• README files document setup, inputs, outputs, assumptions, and run instructions
• Data provenance recorded from source acquisition through published outputs
• Random seeds fixed for reproducible sampling and simulation procedures
• Research decisions documented with dates, owners, and approval status

Participants can use Git to manage changes and review research automation code.

• Git repositories initialised with meaningful project structure and ignore rules
• Commits written to describe discrete analytical or automation changes
• Branches used to isolate new features and revised methods
• Diffs inspected before merging changes into the main branch
• Merge conflicts resolved in scripts, notebooks, and documentation files
• Sensitive files excluded using .gitignore and secure credential practices

Participants can test Python automation components and diagnose failures systematically.

• pytest tests written for transformations, validation rules, and helper functions
• Test fixtures provide small controlled datasets for repeatable checks
• Boundary cases tested for empty files, null values, and duplicate keys
• Python logging records processing stages, counts, warnings, and failures
• Debugging uses breakpoints, print inspection, and minimal reproducible examples
• Failure messages designed to identify affected files, fields, and corrective actions

Participants can prepare an automation workflow for routine execution and team handover.

• Scheduled runs planned around source availability, review points, and deadlines
• Task Scheduler and cron concepts compared for routine script execution
• Runbooks specify prerequisites, commands, expected outputs, and escalation routes
• Parameter changes controlled through dated configuration files and approvals
• Output checks assigned before publication or distribution of research findings
• Handover packs identify code owners, data custodians, and review responsibilities

Participants can build, review, and present a complete version-controlled Python research automation pipeline.

• Research question translated into pipeline inputs, transformations, outputs, and controls
• Raw data ingested from files, web sources, or documented APIs
• Cleaning, validation, and analysis functions assembled into reusable scripts
• Automated tables, charts, and report outputs generated from validated data
• Git history, tests, logs, and provenance documentation reviewed by peers
• Pipeline presentation explains design choices, limitations, handover, and maintenance