My Tryst with Out of Memory (OOM) Error: Taming High-Volume ML Pipelines on Limited Hardware

 How I Fixed Memory Bloat in a Prefect-Orchestrated Workflow Without RAM Upgrades

The Breaking Point

There I was—wrapping up Module 3 of DataTalksClub’s #mlopszoomcamp. My mission: orchestrate an ML pipeline for a high-volume dataset using Prefect, with MLflow tracking experiments. Everything ran smoothly until...

The dreaded Out of Memory (OOM) error struck during data transformation. With millions of records and high-cardinality features, my 16GB RAM AWS EC2 instance was gasping. Here’s the twist: I refused to add RAM. Why? To simulate real-world constraints of local machines/laptops where hardware upgrades aren't always possible or where budget provisions for scaling up compute in cloud virtual machines are not approved readily.


The Hardware Gambit

  • Initial EC2 setup: 16GB RAM + 30GB storage (barely more than a typical laptop)

  • Deliberate constraint: Avoided RAM upgrade to mirror local-dev limitations

  • Storage expansion: Added 20GB storage (total 50GB) to create breathing room

  • Swap strategy: Allocated 30GB of this for swap file virtual memory


2 Battle-Tested Fixes + The Swap Revelation

  1. Surgical Data Loading
    Problem:     Loading all columns wasted memory
    Fix: Select only essential features upfront:
    df = pd.read_parquet('data.parquet', columns=['required_feature(s)', 'target'])

    
    2. Aggressive Memory Cleanup

          Problem: Lingering Dataframes choked RAM.

          Fix: Purge intermediates & no longer to be used structures after preprocessing but before vectorizing:    

           del df  # Execute before deriving encoding/vectorization   

        


      3. Swap File Optimization

         Phase 1: Proof of Concept

         Created 30GB swap file on expanded storage:

            sudo fallocate -l 30G /swapfile  
         sudo chmod 600 /swapfile  
          sudo mkswap /swapfile  
          sudo swapon /swapfile  
      
         Result: Pipeline ran successfully (without Prefect) using 16GB RAM + 13GB swap


        Phase 2: Right-Sizing

        Replaced with 16GB swap to conserve storage:

          sudo swapoff /swapfile && sudo rm /swapfile # Remove old

          free -h # check the new swap file created

    Final Test: Prefect-orchestrated workflow ran flawlessly with 16GB RAM + 16GB swap


Why This Matters for Local Development   

  1. Real-world simulation:

    • 30GB storage mimics limited laptop SSD space

    • Fixed RAM mirrors consumer hardware constraints

  2. The swap insight:
            

  3.  Resource tradeoffs:

  • Used 30% storage for swap to avoid 100% RAM cost

  • Achieved 32GB effective memory (16+16GB swap used) for heavy transformations and training

     

The Victory Lap

Results:

  • Prefect workflows executed end-to-end with OOM conquered

  • MLflow logged artifacts reliably despite memory pressure

  • Models trained on 3M+ records using storage as memory


Golden Insight:

"When RAM is finite, treat storage as your emergency oxygen tank – but measure exactly how much air you need before diving into production."



Call to Action:

Pushed your hardware to its limits? Share your creative workarounds below!


ResourceInitialOptimizedPurpose
RAM16GB16GBFixed constraint
Storage30GB50GBExpanded for swap
Swap FileNone30GB → 16GBRight-sized virtual memory


The revised version emphasizes our deliberate constraints, showcases our iterative problem-solving, and positions our solution as universally applicable to local development scenarios.






Comments

Post a Comment

Popular posts from this blog

My midterm project at MLZoomcamp led by Alexey Grigorov for DataTalksClub

 Predicting Patient No-Shows: A Data-Driven Approach Hospital no-shows significantly disrupt healthcare systems, wasting resources and delaying care for those in need. My midterm project for the MLZoomcamp , led by Alexey Grigorev and hosted by DataTalksClub, tackles this challenge using machine learning to predict no-show probabilities for appointments in Brazilian hospitals. Here's how I approached the problem: The Challenge The dataset, sourced from Kaggle, includes over 110,000 appointments and diverse features such as patient demographics, appointment details, and medical history. However, achieving reliable predictions is complex due to: Imbalanced Data : About 80% of appointments were attended, while 20% were no-shows. Dependence on Feature Engineering : Key predictors like patient history (previous/missed appointments) were engineered from the raw data. Bias Mitigation : Socioeconomic factors, such as neighborhood, required careful handling to ensure fairness. The Solution ...
Read more

My Capstone 1 Project at MLZoomcamp: Bird Species Classification with Deep Learning

 Classifying Bird Species: A Deep Learning Approach to Image Classification Bird species classification can contribute to various ecological and environmental studies, helping researchers identify patterns and protect endangered species. For my Capstone 1 project at MLZoomcamp led by Alexey Grigorev @DataTalks.Club, I took on the challenge of classifying bird species from a dataset of 25 Indian bird species, leveraging deep learning techniques for image classification. Here’s a breakdown of how I approached this problem: The Challenge The dataset, sourced from Kaggle, consists of over 22,600 images of 25 different bird species. The key challenges for this project included: Large Dataset: With more than  22,600  images, managing such a large dataset requires efficient    preprocessing and handling techniques. High Image Variability:  Different lighting conditions, angles, variations in bird image backgrounds, poses and image resolutions made it difficult fo...
Read more

Starting my Data Engineering journey with a foundational insight on Docker, Terraform and Google Cloud Platform

 The Data Engineering Zoomcamp 2025, led by Alexey Grigorev at DataTalksClub, offers an in-depth exploration of modern data engineering practices. The first module, "Containerization and Infrastructure as Code," serves as a foundational entry point into the course, equipping participants with essential skills for building and managing scalable data systems. Module 1: Containerization and Infrastructure as Code This module introduces participants to two pivotal concepts in data engineering: containerization and infrastructure as code (IaC). By leveraging these technologies, data engineers can create consistent, reproducible environments and automate the provisioning of infrastructure, leading to more efficient and reliable data pipelines. Key Topics Covered: Introduction to Google Cloud Platform (GCP): Participants are introduced to GCP, a leading cloud service provider offering a suite of tools and services for building and managing data systems. The course provides guidance ...
Read more