KDD Process for Data Science Projects Training Course

Course overview

Data science projects often stall because teams begin modelling before they have defined the knowledge to be discovered, selected defensible source data, or established how findings will be validated by the business. The Knowledge Discovery in Databases (KDD) process provides a disciplined route from operational data to usable evidence: selection, preprocessing, transformation, data mining, and interpretation/evaluation. This course helps experienced analysts and technical professionals use KDD to reduce rework, make modelling choices traceable, and produce findings that decision-makers can inspect and act upon.

Across five days, participants apply the full KDD process to a realistic business dataset. They learn to frame a discovery objective and success measures; profile and select source data; document missingness, duplicates, outliers, and quality rules; engineer and transform features; compare supervised and unsupervised data-mining techniques; and evaluate patterns for statistical, operational, and ethical validity. Practical work uses Python, Jupyter Notebook, pandas, and scikit-learn, alongside KDD artefacts including data-selection rationales, preprocessing logs, feature definitions, experiment records, and interpretation reports.

Teaching combines instructor-led method sessions with guided notebook labs, peer review, and a cumulative case study. Participants work in small teams to build an auditable KDD workflow for a customer, operations, or risk-oriented problem, making explicit decisions at every stage rather than treating data preparation as an informal prelude to modelling. They leave with a completed KDD project pack: a problem statement, data-quality assessment, transformation pipeline, evaluated model or pattern set, and an implementation-ready findings briefing that can be adapted for workplace use.

The course is designed for practitioners who already work with datasets and need a repeatable discovery method for projects where data quality, explainability, and business adoption matter as much as predictive performance.

Course objectives

By the end of this course, participants will be able to:

• Define a KDD discovery objective, target variable, decision context, and measurable success criteria
• Select and justify source datasets using coverage, granularity, lineage, and relevance criteria
• Profile data with pandas to identify missingness, duplicates, outliers, invalid values, and leakage risks
• Build documented preprocessing rules for cleaning, imputation, deduplication, and data-quality controls
• Create and evaluate transformed features using encoding, scaling, aggregation, and dimensionality-reduction techniques
• Apply classification, clustering, and association-rule methods to an appropriate KDD problem
• Evaluate discovered models and patterns using performance metrics, validation design, interpretability, and business utility
• Produce an auditable KDD project pack containing decision logs, notebook evidence, findings, and recommendations

Benefits of attending

Target competencies

Who should attend

• Data Scientists — who need a rigorous, traceable process for turning raw operational data into validated findings
• Senior Data Analysts — who move from reporting into predictive, segmentation, or pattern-discovery work
• Machine Learning Engineers — who need stronger data-selection, preprocessing, and evaluation discipline before deployment
• Business Intelligence Managers — who oversee analytical teams and need repeatable standards for evidence-based projects
• Data Engineers — who prepare analytical datasets and must understand the downstream requirements of KDD workflows
• Digital Transformation Leads — who sponsor data initiatives and need to assess whether proposed insights are reliable and usable

Requirements and prerequisites

Participants should be comfortable working with structured tabular data and should understand basic statistics, including distributions, correlation, sampling, and train/test splits. Prior experience writing simple Python scripts or using Jupyter notebooks is expected; attendees should be able to read pandas data-frame operations and follow code for filtering, grouping, and joining data. Familiarity with one predictive modelling method, such as linear regression, decision trees, or logistic regression, is useful. Deep learning, advanced calculus, production MLOps, database administration, and prior knowledge of the KDD process are not required.

Training methodology

The instructor introduces each KDD phase using a worked business case, then participants implement that phase in guided Jupyter Notebook labs. Exercises use pandas for data profiling and preparation, followed by scikit-learn workflows for modelling and evaluation. Teams review one another's data-selection rationale, cleaning rules, and interpretation claims against explicit evidence criteria. Short case discussions examine false discoveries, target leakage, weak validation, and misleading visualisations. The final session is an application-planning workshop in which each participant maps the KDD artefacts and controls to a live or planned workplace project.

Course outline

Day 1: Framing Knowledge Discovery and Selecting Data

• KDD process stages and hand-offs from data to knowledge
• Business problem framing for discovery-oriented projects
• Decision context, discovery questions, and success criteria
• Unit of analysis, target definition, and observation windows
• Data-source inventory, lineage, and ownership assessment
• Dataset selection using relevance, coverage, granularity, and timeliness
• Selection bias, sampling bias, and data leakage at project initiation

Workshop: Participants create a KDD project charter and source-data selection matrix for a customer attrition case, including assumptions and acceptance criteria.

Day 2: Preprocessing and Data Quality Control

• Exploratory data profiling with pandas DataFrame methods
• Missing-data mechanisms and imputation decision rules
• Duplicate records, entity resolution, and key-integrity checks
• Outlier detection using distributions, IQR rules, and domain thresholds
• Invalid values, inconsistent categories, and schema conformance
• Data-cleaning pipelines and reproducible preprocessing notebooks
• Preprocessing logs, quality metrics, and decision-trace documentation

Workshop: Participants profile a flawed operational dataset and produce a documented cleaning plan, quality dashboard, and reproducible preprocessing notebook.

Day 3: Transformation and Feature Construction

• Feature semantics and operational definition of variables
• Categorical encoding with one-hot and ordinal approaches
• Scaling, normalisation, and transformation for model suitability
• Aggregation across customer, transaction, and time dimensions
• Date-time features, recency-frequency measures, and rolling windows
• Feature selection using variance, correlation, and mutual information
• Dimensionality reduction with principal component analysis

Workshop: Participants build a feature dictionary and transformation pipeline that converts prepared transaction data into an analysis-ready modelling table.

Day 4: Data Mining Methods and Experiment Design

• Matching KDD questions to classification, clustering, and association rules
• Decision trees and random forests for interpretable classification
• K-means clustering and cluster-profile interpretation
• Association-rule mining with support, confidence, and lift
• Training, validation, test sets, and cross-validation design
• Baseline models, hyperparameter search, and experiment tracking
• Class imbalance, resampling, and threshold selection

Workshop: Participants run and compare classification, clustering, and association-rule experiments, then record an evidence-based method-selection decision.

Day 5: Interpretation, Evaluation, and Knowledge Deployment

• Performance metrics including precision, recall, F1, ROC-AUC, and silhouette score
• Business utility, error costs, and decision thresholds
• Model interpretability using feature importance and partial dependence
• Pattern validity, spurious correlations, and confirmation checks
• Fairness, privacy, and responsible interpretation of discovered knowledge
• Findings narratives, visual evidence, and stakeholder briefing structure
• KDD project closure, monitoring requirements, and next-step recommendations

Workshop: Participants complete and present a KDD findings pack containing validated results, limitations, recommended actions, and a workplace application plan.

Tools & standards covered

Python, Jupyter Notebook, pandas, scikit-learn

A typical training day

Live online deliveries follow the same structure in the East Africa Time zone, with shorter screen blocks and longer breaks.

What the fee includes

How you can take this course

Certification

Participants who complete the full five days receive the Skillset Development Certificate of Completion, stating the course title, course code, dates and delivery format — suitable for professional-development records and employer reimbursement.

Frequently asked questions