Programming 6 Projects: Snakemake, Dask, MLOps

Below are short descriptions and goals of each project. For more detailed information, refer to README and report files in the respective project's folders.

Snakemake:

Trying Different Models:

The goal is to Refactor a script that tests different machine learning models, parallelizing the model training and evaluation using Snakemake. Compare the performance to a regular python pipeline.

Task Overview:

1. Refactor the Script
2. Design the Workflow
3. Implement the Workflow
4. Document the Workflow

Dask:

Lung Cancer Classification with Dask and XGBoost:

The goal is to use Dask for data manipulation and parallelized workflows. Integrate Dask with XGBoost for distributed machine learning. Compare the performance of Dask workflows with traditional pandas/numpy workflows.

Task Overview:

1. Load the data
2. Initial Data Preprocessing
3. Exploratory Data Analysis (EDA)
4. Further Preprocessing

MLOps:

Monitoring Platform:

The goal is to build a lightweight, end‑to‑end monitoring platform that tracks prediction quality, data integrity, service latency, and resource consumption for a simple model‑serving service (e.g., a scikit‑learn classifier exposed via a FastAPI endpoint).

Task Overview:

1. instrument code to emit metrics
2. store and visualise those metrics
3. detect issues e.g. prediction's low quality
4. document the design choices and trade‑offs