Basketball Data Pipeline

The Problem

Data analysis often begins with information spread across multiple files or sources. Before useful analysis can happen, the data must be collected, organized, cleaned, and converted into a consistent structure.

Without a repeatable process, these preparation steps can become difficult to reproduce, easy to break, and time-consuming to update when new data becomes available.

The Approach

I organized the work as a sequence of clearly defined stages: acquisition, cleaning, preprocessing, and feature engineering. Each stage produces a structured output that can be inspected and reused by the next part of the workflow.

The project combines player information from the NCAA website with performance statistics retrieved from the Yahoo Sports API. The raw data is preserved separately from cleaned and processed outputs so that each transformation can be reproduced.

The Solution

The resulting workflow collects player information and performance statistics, stores the original source files, and combines the data into a consistent dataset using Python and pandas.

The preparation stages identify missing, inconsistent, or invalid values; convert datatypes; standardize measurements; and create derived fields such as per-game statistics, conference labels, assist-to-turnover ratio, and fantasy points.

The final processed dataset is structured for reuse in exploratory analysis, visualization, and modeling without requiring the original preparation work to be repeated manually.

Project Structure

The repository organizes the workflow into numbered Jupyter notebooks, saved datasets, and model outputs. The first three notebooks cover the pipeline stages highlighted on this page: acquisition, cleaning and preprocessing, and feature engineering.

ncaa-basketball-stats/
├── data/
│   ├── 2024-04-07 NCAA WBB-Div1 Player Info.csv
│   ├── player_info.xlsx
│   ├── player_stats.xlsx
│   ├── player_data_raw.xlsx
│   ├── player_data_clean.xlsx
│   ├── player_data_engineered.xlsx
│   └── additional model input and output files
├── models/
│   └── saved model files
├── notebooks/
│   ├── 1_data_acquisition.ipynb
│   ├── 2_data_cleaning_preprocessing.ipynb
│   ├── 3_feature_engineering.ipynb
│   ├── 4_data_exploration.ipynb
│   ├── 5_data_visualization.ipynb
│   ├── 6_model_selection.ipynb
│   ├── 7_model_training.ipynb
│   ├── 8_model_evaluation.ipynb
│   └── iframe_figures/
├── README.md
├── LICENSE
└── .gitignore

The numbered prefixes on each notebook indicate the order of the workflow. The intermediate datasets also show how the data changes from raw inputs to cleaned and feature-engineered outputs before it is used in later analysis and modeling stages.

Tools Used

• Python for acquisition, transformation, and workflow logic
• JupyterLab for developing and documenting the workflow
• pandas for cleaning, joining, transforming, and exporting data
• requests for retrieving statistics from an API
• JSON for working with API responses
• openpyxl for reading and writing Excel files
• Git and GitHub for version control and project organization

Where the Workflow Leads

This pipeline provides the structured foundation for the later stages of the larger project, including exploratory analysis, visualization, linear regression, and model evaluation.

The same approach can be adapted to recurring business workflows that combine spreadsheet exports, API responses, database extracts, or data received from multiple teams.