Machine Learning for Monitoring and Evaluation Training Course

Participants can write a decision-focused machine learning use-case statement.

• Decision canvas specifies users, actions, constraints and expected value
• Prediction target defined without confusing proxies and programme outcomes
• Baseline process mapped before proposing automated prediction or classification
• Use-case scoring compares feasibility, harm, urgency and data readiness
• Counterfactual questions separated from operational risk prediction questions
• Assumptions register records causal claims, exclusions and intended beneficiaries

Participants can assess whether M&E data are fit for model development.

• Data inventory maps surveys, registries, service records and partner reports
• Indicator reference sheets define units, frequencies, disaggregation and ownership
• Completeness, validity, timeliness and consistency tested using data-quality rules
• Unique identifiers reconciled across beneficiary, facility and activity datasets
• Missing-data mechanisms classified as MCAR, MAR and MNAR
• Data dictionary documents variables, coding schemes, lineage and permitted uses

Participants can prepare reproducible analytical datasets from common M&E sources.

• Analytical dataset constructed from documented extraction and transformation steps
• Categorical variables encoded using one-hot, ordinal and target encoding
• Numerical features scaled with standardisation, normalisation and robust scaling
• Outliers examined using boxplots, z-scores and domain plausibility checks
• Train, validation and test partitions created without temporal leakage
• Feature engineering converts programme events into measurable predictive signals

Participants can inspect data patterns and select defensible candidate features.

• Univariate distributions inspected through histograms, density plots and summary statistics
• Cross-tabulations reveal disparities across sex, age, location and disability
• Correlation matrices identify redundancy, multicollinearity and suspicious relationships
• Feature selection uses mutual information, LASSO and recursive elimination
• Temporal trends checked before treating historical patterns as stable predictors
• Proxy variables reviewed for hidden socioeconomic, ethnic or protection signals

Participants can build and assess regression models for continuous programme outcomes.

• Linear regression estimates continuous outcomes such as attendance and expenditure
• Regularised regression applies ridge and LASSO penalties to correlated features
• Random forest regression captures nonlinear relationships within administrative data
• Mean absolute error and RMSE compared against operational tolerances
• Residual plots identify systematic error across facilities and population groups
• Prediction intervals communicated without presenting estimates as causal effects

Participants can construct classification models and set operational decision thresholds.

• Binary classification predicts events such as dropout, stock-out or non-response
• Logistic regression establishes interpretable baseline classification models
• Decision trees and random forests compare rules against ensemble performance
• Precision, recall, specificity and F1 score interpreted for service decisions
• ROC and precision-recall curves assessed under class imbalance
• Thresholds selected using referral capacity, error costs and safeguarding requirements

Participants can use unsupervised methods to identify patterns requiring investigation.

• K-means clustering segments households, facilities or communities by measured characteristics
• Hierarchical clustering reveals nested patterns without predefined segment numbers
• Silhouette scores assess cluster separation and practical interpretability
• Principal component analysis reduces correlated indicators for exploratory visualisation
• Isolation forests flag unusual transactions, reports or service-delivery records
• Clusters treated as analytical hypotheses rather than beneficiary labels

Participants can design responsible text-analysis workflows for open-ended M&E data.

• Text cleaning removes duplicates, stop words and personally identifying information
• Tokenisation, stemming and lemmatisation prepare feedback for structured analysis
• TF-IDF represents recurring terms within complaints and beneficiary narratives
• Topic modelling identifies candidate themes for human validation
• Sentiment classification tested against locally reviewed language samples
• Human coding protocols verify context, sarcasm, dialect and sensitive disclosures

Participants can assess geospatial data for monitoring coverage and contextual risk.

• Spatial joins connect programme records with administrative boundaries and exposure data
• Raster and vector formats distinguished for satellite and facility datasets
• Distance calculations identify travel barriers to services and distribution points
• Spatial autocorrelation checked before interpreting geographically clustered predictions
• Remote-sensing proxies assessed against ground-truthed field observations
• Location privacy controls protect sensitive sites, households and movement patterns

Participants can test model reliability and diagnose errors before operational use.

• K-fold cross-validation estimates performance across repeated data partitions
• Temporal validation tests models against later programme periods
• Confusion matrices expose errors hidden by aggregate accuracy measures
• Calibration curves compare predicted probabilities with observed event rates
• Subgroup performance examined by sex, age, geography and vulnerability status
• Error cases reviewed with programme staff before deployment decisions

Participants can explain model outputs to programme, technical and governance stakeholders.

• Global feature importance identifies variables influencing overall model behaviour
• Partial dependence plots show average relationships between features and predictions
• SHAP values explain individual predictions using additive feature contributions
• Counterfactual explanations identify changes associated with altered prediction outcomes
• Model cards document purpose, data, metrics, limits and intended users
• Plain-language explanation separates predictive association from programme causation

Participants can assess equity risks and specify safeguards for machine learning use.

• Representation bias assessed through coverage gaps and missing population groups
• Label bias identified where historical decisions reflect unequal access
• Demographic parity and equal opportunity compared for classification outcomes
• Disaggregated error rates tested before approving targeting or referral models
• Human rights impact assessment identifies harms, remedies and accountability holders
• Appeal routes and human review designed for consequential decisions

Participants can apply data-protection controls to machine learning-enabled M&E activities.

• Purpose limitation restricts data use to documented monitoring objectives
• Data minimisation removes unnecessary identifiers, attributes and historical records
• Consent language distinguishes service delivery from secondary analytical use
• Pseudonymisation and aggregation reduce re-identification risks during analysis
• Data protection impact assessments document risks, mitigations and residual exposure
• Retention schedules define deletion, archival and access-review requirements

Participants can specify operational controls for maintaining deployed analytical models.

• Model registry records versions, owners, approval status and deployment dates
• Data drift monitored through feature distributions and population stability indices
• Concept drift detected when outcome relationships change over time
• Performance dashboards track calibration, subgroup errors and referral volumes
• Retraining triggers defined using performance thresholds and operational changes
• Incident procedures address harmful predictions, outages and suspected misuse

Participants can embed model outputs within accountable programme management processes.

• Standard operating procedures assign review, escalation and decision responsibilities
• Model outputs integrated with indicator dashboards and routine review meetings
• Referral workflows specify human verification before participant-facing actions
• Change-management plans address staff trust, skills and process adaptation
• Vendor requirements define documentation, audit access and data ownership
• Implementation roadmap sequences piloting, evaluation, scaling and retirement decisions

Participants can communicate model evidence, limitations and recommendations for action.

• Decision briefs state the operational question, evidence and recommended action
• Uncertainty visualisation presents ranges, calibration and plausible error consequences
• Audience-specific reporting differentiates technical, management, donor and community needs
• Data stories connect predictions with programme context and verified field evidence
• Limitations section documents exclusions, bias risks and non-causal interpretation
• Governance presentations record approval decisions, conditions and review dates

Participants can build, review and present a complete machine learning-enabled M&E system design for a defined programme decision.

• Use-case statement connects programme decisions, users, outcomes and constraints
• Analytical dataset plan specifies sources, quality checks and transformation logic
• Model approach justifies baseline, evaluation metrics and threshold choices
• Fairness and privacy assessment documents risks, safeguards and accountability
• Monitoring plan defines drift indicators, review cycles and retraining triggers
• Implementation roadmap presents pilots, resources, governance and success criteria