AI
The Reproducibility Problem in ML Workflows
A 2023 study found 300 machine learning papers affected by reproducibility failures tied to data leakage. Data leakage happens when a model sees the held-out test split during training. The model already knows the answers, so every metric it produces is meaningless. Beyond leakage, reproducibility is what lets teams compare runs at all. Without tracked inputs and outputs across runs, there is no way to confirm which parameter set produced the best model before pushing to production.
Five Failure Modes in Traditional ML Workflows
Jupyter notebooks vanish when a laptop restarts. Manual spreadsheet tracking accumulates human errors, a misread 0.5 entered as 0.05 ruins a run. Model files pile up as model_final_final_v2, with no way to trace which produced which result. Copy-pasting a model into a new environment triggers crashes with no version to roll back to. The fifth failure is the classic “it worked on dev” problem. Each issue traces to two root causes: environment and version drift, and missing lineage.
The MLOps Stack: Kubeflow, MLflow, Kaniko, and Argo
The solution is a fully open source stack. Kubeflow Pipelines orchestrates the pipeline and bundles Argo Workflows plus ML metadata. Argo Workflows runs each stage in a separate container, so no stage can read another stage’s data, which closes the leakage vector. MLflow tracks parameters, metrics, and registered model versions. MinIO stores versioned model artifacts (swappable for S3 in production). Kaniko builds containers in user space because Argo Workflows does not allow privileged containers. Argo CD monitors the Git repository and syncs any new desired state to the cluster automatically.
Demo: Running the Five-Stage Pipeline Live
The demo dataset is an open source fitness dataset: 20,000 records, 39 numeric features, unlabeled, with an expected anomaly rate of 5%. The pipeline trains an Isolation Forest model, registers it with MLflow (incrementing the version number each run), validates the anomaly rate against a 3 to 8% gate, builds the container image with Kaniko, then commits the new image tag to Gitea. Every package in every container is pinned; MLflow 2.17.2 is pinned explicitly. Kubeflow’s compare-runs view showed one run with contamination 0.5 producing a 50% anomaly rate, correctly blocked by the gate.
GitOps Closes the Audit Loop
After the pipeline commits to Gitea, the Argo CD application detects the diff and syncs it to the cluster. The Gitea commit log shows two entries: the initial lab startup commit, then a second commit authored by Kubeflow Pipelines that added a model version annotation and updated the image tag to the new registry URL. Argo CD confirmed sync success six minutes after the pipeline finished. Two pods came up running in the default namespace. A test query against the served model returned prediction equal to 1 for a healthy 35-year-old’s fitness record.
Five Rules for a Reproducible ML Pipeline
Mohamed closed with five concrete rules. Pin every package to an exact version so environment drift cannot happen between runs. Track full lineage with ML metadata so every input, artifact, and output is auditable. Use GitOps so no one can modify the cluster state outside the recorded desired state. Validate before shipping: if the anomaly rate falls outside the accepted range, raise an error and stop the pipeline before the build stage. Store all artifacts as immutable objects in versioned storage, whether Git or a versioned S3-compatible bucket, so nothing can be silently overwritten.
Q&A
What Kubernetes components does the cluster need beyond Kubeflow and Argo? Everything deployed is in the provided deploy script running against Kubernetes 1.34; nothing is hidden and all configuration is explicit. ▶ 01:07:00
Can external tools like a hosted MLflow or Harbor registry replace the in-cluster ones? Yes, each tool endpoint is a pipeline parameter; swapping in an external Harbor or hosted Argo instance requires only changing that parameter value. ▶ 01:07:00
Does hardware variation across nodes affect reproducibility? Yes. Taints, tolerations, and node selectors in Kubernetes can pin workloads to specific GPU nodes, or workloads can be separated across clusters by hardware tier. ML metadata also captures scheduling information as part of lineage. ▶ 01:09:22
How do you configure a Kubeflow pipeline stage to use a GPU for fine-tuning? Node selectors and taints/tolerations can be enforced in the pipeline Python definition; the exact DSL syntax was deferred to a follow-up conversation after the session. ▶ 01:13:09
Notable Quotes
300 is the amount of machine learning papers that detected data leakage affected by reproducibility. Nourhan Mohamed · ▶ 01:31
If the model peaks on that data, then our experiment is ruined. Nourhan Mohamed · ▶ 05:37
each package is pinned to a version. So if I run this today, if my friend run this tomorrow, MLflow is always going to be installed by version 2.17.2 Nourhan Mohamed · ▶ 41:00
It has an anomaly rate of 50% and this is outside 3 to 8. So this is not what we want. We want it to crash. Nourhan Mohamed · ▶ 50:54
immutable artifacts. Make sure that if you are using git, if you are using virgined S3 like storage, everything is going to be the same. It’s as if you’re writing with a pen, not a pencil. Nourhan Mohamed · ▶ 01:04:36
Key Takeaways
- Pin every package to an exact version to eliminate environment drift between pipeline runs.
- ML metadata embedded in Kubeflow Pipelines tracks full input-to-output lineage automatically.
- A validation gate that raises an error on out-of-range anomaly rates stops bad models before build.
About the Speaker(s)
Nourhan Mohamed is a Golden Kubestronaut and DevOps Lead at KodeKloud, where she leads the Labs and Curriculum teams. Her background spans Dell technical support, virtualization, containerization, and Kubernetes platform management at VMware before joining KodeKloud. She designs hands-on learning experiences used by thousands of practitioners across cloud-native and infrastructure automation topics.