Listen to this Post
Introduction: Your AI Project Is Growing, but Your Storage Strategy Is Falling Apart
There is a familiar moment in almost every serious machine learning project.
The prototype works.
The data starts growing.
The experiments multiply.
A few gigabytes quietly become hundreds of gigabytes. Then come Parquet files, checkpoints, cleaned datasets, intermediate artifacts, logs, model outputs, and multiple engineers all trying to work on the same pipeline.
At first, the obvious answer seems simple: put everything in Git.
After all, developers already trust Git. It provides history, collaboration, branches, pull requests, and a workflow that almost every engineering team understands. Add Git LFS, and storing large files may appear to solve the problem.
But this is where many AI and machine learning architectures begin to break.
Git was designed to manage source code. It was never designed to become a constantly changing warehouse for hundreds of gigabytes of training data. As datasets evolve, files are regenerated, Parquet objects change, and training artifacts accumulate, the elegant developer workflow can slowly transform into a frustrating operational bottleneck.
Clones become slow. CI/CD pipelines become heavier. Storage consumption grows. Engineers begin learning obscure Git LFS flags just to avoid downloading files they never intended to use.
The problem is not that Git is bad.
The problem is architectural misuse.
Modern AI systems need different types of storage for different stages of the workflow. Source code, local scratch data, immutable datasets, mutable training artifacts, and cloud-scale objects do not all behave the same way. Treating them as if they do is one of the easiest ways to create unnecessary complexity.
The real question is no longer, “Where can I put this file?”
The real question is, “What kind of storage architecture matches the lifecycle of this data?”
The Original Problem: What Happens When 500GB Lands in Your ML Pipeline
Imagine preparing a machine learning model that will predict prices using gradient-boosted trees. Or perhaps the project is much larger, involving fine-tuning or training a large language model.
After cleaning, collecting, and transforming the data, you estimate that the training dataset has reached 500GB.
The training will happen in the cloud because most teams do not have an NVIDIA H200 sitting unused in the office. You decide to use a modern columnar format such as Parquet, which makes sense for large-scale analytical and machine learning workloads.
Now comes the storage decision.
Your code already lives in Git.
Several developers are helping clean and prepare the data.
The dataset will go through multiple iterations.
So the natural temptation is to place the dataset alongside the code and use Git LFS for the large files.
At first, everything appears manageable.
Then someone clones the repository.
Twenty minutes pass.
The repository is enormous.
The CI/CD pipeline begins downloading or processing data it does not necessarily need.
Slack messages start appearing.
Why is the build still running?
Why is cloning this repository taking forever?
“Why did my pipeline suddenly consume so much storage?”
The architecture has reached the point where a tool built for source control is being forced to behave like a large-scale machine learning storage platform.
That distinction matters.
The Core Architectural Mistake: Versioning Code Is Not the Same as Managing Data
Git is exceptional at handling source code.
Source files are usually relatively small. They are text-based. Developers need to inspect changes, compare branches, merge modifications, and preserve history.
Git was built around exactly this kind of workflow.
Machine learning data behaves differently.
A dataset may contain large binary files, Parquet tables, images, audio, embeddings, checkpoints, model weights, logs, and generated artifacts.
A small logical change may cause a large physical file to change.
If a Parquet file is regenerated after adding several rows or modifying metadata, the storage system may need to deal with a new version of a large object.
The human sees a tiny change.
The storage system may see a completely different large binary file.
That is where the economics of data movement become important.
In AI infrastructure, the cost is not only about storing the data.
It is also about moving it.
Every unnecessary download, upload, clone, synchronization, and replication becomes part of the infrastructure bill.
The First Storage Paradigm: Standard POSIX File Systems
Local Storage Remains the Fastest Place to Work
Traditional file storage is the environment developers know best.
You have files.
You have directories.
You use commands such as:
ls cp mv find du
The storage may be backed by NVMe drives, SSDs, spinning disks, network-attached storage, or larger storage arrays.
For active development, this model is extremely convenient.
The operating system understands it.
Applications understand it.
Tools expect it.
Latency is usually low, and data can move quickly between storage and memory.
This makes standard file storage particularly useful during active development and GPU training.
The Scratch Disk Advantage: Fast Storage Still Matters for AI
During a GPU training run, local or high-performance attached storage can be extremely valuable.
Training jobs may repeatedly read batches of data.
Checkpoints may be written frequently.
Temporary artifacts may appear and disappear.
Caching may become essential.
Trying to perform every operation directly against remote storage can introduce latency and unnecessary network activity.
A fast scratch disk allows the compute environment to operate efficiently.
For example:
df -h du -sh ./training-data mkdir -p /mnt/nvme/ml-cache cp -r ./dataset /mnt/nvme/ml-cache/
This approach is simple.
The problem is that local storage does not automatically solve collaboration.
The Collaboration Problem: Local Storage Does Not Scale Across Teams
A dataset stored on one
It is not necessarily convenient for everyone else.
How do other team members access the same data?
How do cloud training jobs retrieve it?
How do you manage permissions?
How do you maintain a shared catalog?
How do you synchronize changes?
The answer often becomes a collection of scripts, shared drives, cloud buckets, mounted volumes, and manual instructions.
That works for a while.
Then the organization grows.
The AI pipeline becomes larger.
Different teams begin producing different datasets.
Eventually, the storage architecture becomes something held together by shell scripts and institutional memory.
That is rarely a sustainable design.
The Second Storage Paradigm: Traditional Object Storage
Why S3 Became the Foundation of Modern Data Infrastructure
Object storage changed the way organizations think about large-scale data.
Services following the Amazon S3 model can store massive numbers of objects and scale far beyond what most traditional file systems can comfortably support.
This is why object storage has become a major foundation for data lakes and large analytical platforms.
A typical workflow may involve tools such as Apache Spark, Apache Arrow, Iceberg, or other systems reading and writing data through object storage.
The architecture is powerful.
The scale is enormous.
The API is widely supported.
But scale does not automatically mean seamless developer experience.
The Object Storage Problem: AI Data Can Become Operationally Fragmented
Traditional object storage is often separated from the developer workflow.
Code may live in one system.
Data may live in another.
Authentication may use different credentials.
Access policies may depend on IAM.
Networking may require VPC configuration.
Training infrastructure may exist in another environment entirely.
Each individual piece may be correct.
Together, however, they can create friction.
An engineer might need Git credentials for code, cloud credentials for data, separate permissions for compute, and networking access for training infrastructure.
The system works.
But it is not always simple.
For small teams and rapidly evolving AI projects, complexity can become an engineering tax.
The Mutability Challenge: AI Data Rarely Stays Still
Machine learning datasets are often works in progress.
Data is cleaned.
Rows are removed.
Features are added.
Labels are corrected.
Files are reorganized.
Embeddings are regenerated.
Intermediate outputs appear.
Training checkpoints evolve.
This is fundamentally different from publishing a final release of source code.
During active experimentation, the data is mutable.
It is changing constantly.
Storage systems that force large-scale replacement or repeated transfer of large objects can create unnecessary overhead.
The question becomes simple.
Why move 1GB when only a few kilobytes of useful information actually changed?
The Third Storage Paradigm: Git and Git LFS
Why Git Feels Like the Obvious Answer
Git has one enormous advantage.
Developers already use it.
It provides a familiar workflow:
git clone git status git add git commit git diff git push
The team understands branches.
The team understands pull requests.
The team understands history.
Keeping code and data together feels architecturally clean.
Git LFS extends this idea by replacing large file contents in the repository with pointer files while storing the actual objects separately.
For some workloads, this is perfectly reasonable.
The problem appears when Git is asked to manage a huge, constantly changing machine learning workspace.
The Binary Data Problem: Git Cannot Read Your Dataset Like Source Code
Git is extremely effective when a developer changes:
Run learning_rate = 0.01
into:
Run learning_rate = 0.02
The difference is clear.
Git can display it.
A human can review it.
A merge system can reason about it.
Large binary datasets are different.
A Parquet file is not a source file.
A model checkpoint is not a configuration file.
An embedding database is not a Markdown document.
When a large binary object changes, the workflow becomes much less informative.
A normal Git diff cannot provide the same meaningful review experience.
You may simply know that a large file changed.
That does not necessarily explain what changed inside the data.
The Hidden Cost of Git History
Every Large Change Can Become an Infrastructure Burden
Suppose a large file is modified.
Perhaps only a small amount of logical information changed.
But the resulting file is still large.
Over time, repeated iterations can create significant storage and transfer overhead.
The project is no longer just managing code.
It is carrying around the history of a large and evolving collection of data artifacts.
Engineers may attempt to improve the situation using options such as:
GIT_LFS_SKIP_SMUDGE=1
or selective fetch configurations.
They may use partial clones:
git clone --filter=blob:none <repository>
or shallow clones:
git clone --depth 1 <repository>
These techniques can be useful.
But there is an architectural warning sign when every new engineer needs a collection of special instructions before they can efficiently work with the repository.
The workflow is becoming harder because the tool is being stretched beyond its primary purpose.
The Real Problem: One Storage System Cannot Solve Every AI Problem
The biggest mistake is not using Git.
The biggest mistake is expecting one storage technology to be perfect for every stage of an AI pipeline.
An AI project contains multiple categories of information.
Source code has one lifecycle.
Configuration has another.
Raw data has another.
Intermediate datasets change frequently.
Training checkpoints evolve rapidly.
Final datasets may need strong versioning and reproducibility.
Temporary GPU artifacts may only need to exist for hours.
These objects should not automatically be treated the same way.
Storage architecture should reflect data lifecycle.
The Missing Layer: Storage Designed for Mutable ML Workflows
Why Hugging Face Buckets Target a Different Problem
Hugging Face Buckets are presented as a storage approach aimed at machine learning workflows where large collections of files change frequently and need to move efficiently between environments.
The key architectural idea is not to replace every storage technology.
Instead, it fills a specific gap.
Git remains useful for code.
Local storage remains useful for active development and scratch space.
A mutable bucket can be used for data that is actively changing during preparation and experimentation.
Once the dataset is finalized, it can be promoted into a more permanent, versioned distribution model.
This separation is important.
Mutable does not mean unorganized.
Versioned does not mean everything must be versioned at every second of its life.
S3 Compatibility: Reusing the Existing AI Ecosystem
A Familiar Interface Reduces Migration Friction
One of the strongest ideas behind an S3-compatible storage interface is interoperability.
Developers already have tools that understand the S3 model.
For example:
aws s3 ls
Apache Spark can interact with S3-compatible storage.
Many data engineering tools can do the same.
Applications built around object storage often do not need an entirely new conceptual model.
The article describes using the Hugging Face S3-compatible endpoint:
https://s3.hf.co
The objective is straightforward.
Instead of forcing engineers to learn a completely new storage SDK, the infrastructure attempts to integrate with tools that already support the S3 ecosystem.
That can significantly reduce friction during adoption.
Xet Deduplication: The Storage Optimization AI Workloads Need
AI Produces Repetition at an Enormous Scale
Machine learning workflows often create artifacts that are highly similar to previous artifacts.
Consider model checkpoints.
A checkpoint from epoch 10 may be extremely similar to epoch 9.
A dataset after cleaning may differ only slightly from the previous version.
A large Parquet collection may contain mostly unchanged information.
Traditional file-level storage often treats files as independent objects.
But a content-addressable system can potentially recognize repeated content and avoid storing or transferring identical blocks unnecessarily.
This is where block-level deduplication becomes powerful.
Why Chunk-Level Deduplication Changes the Economics
Imagine a 1GB dataset file.
You make a small modification.
Perhaps only a few rows were added.
A naïve workflow might upload the entire 1GB object again.
A chunk-aware workflow can potentially identify unchanged sections and transfer only the sections that differ.
The article describes Xet-based deduplication and content-defined chunking as mechanisms intended to reduce redundant transfers.
Conceptually, the workflow looks like this:
Old Dataset
|
v
Chunk Identification
|
+--> Unchanged Chunks: Reuse Existing Data | +--> Changed Chunks: Upload New Data
This approach becomes increasingly valuable as datasets grow.
At 10MB, inefficient transfer may be annoying.
At 500GB, it becomes an infrastructure strategy.
A Simple Synchronization Workflow
Moving Data Without Rebuilding the Entire Storage Process
The article demonstrates a simple synchronization command:
hf buckets sync . hf://buckets/prpatel/mybucket
The attraction of this approach is simplicity.
Developers can work locally.
The data can be synchronized to shared storage.
The storage backend can determine what content already exists and what needs to be transferred.
Instead of manually managing every changed file, the synchronization layer handles the process.
A practical workflow could look like:
mkdir ml-project cd ml-project
hf buckets sync ./data hf://buckets/organization/training-data
After preparing a dataset:
hf buckets sync ./processed-data hf://buckets/organization/processed-data
The storage system becomes part of the ML workflow rather than an obstacle surrounding it.
The Mutable Data Layer: Where AI Experiments Actually Live
Intermediate Data Needs a Different Philosophy
Not every dataset deserves permanent version history.
This is especially true for intermediate artifacts.
Consider:
Temporary feature datasets
Training checkpoints
Generated embeddings
Evaluation results
Agent traces
Experiment logs
Cached model outputs
Data preparation artifacts
These objects may change frequently.
Some may be deleted.
Some may be overwritten.
Some may only exist until the next experiment.
For these workloads, a mutable storage layer can be more practical than forcing every change into a permanent version history.
The Frozen Dataset: When Versioning Finally Makes Sense
Eventually, experimentation ends.
The dataset becomes stable.
The training run becomes reproducible.
The organization may want to preserve the exact data used for a model.
This is the moment when versioning becomes especially valuable.
The architecture can therefore separate two stages.
During development:
Mutable Storage
After finalization:
Versioned Dataset
This is a more natural representation of the machine learning lifecycle.
Data does not have to become permanent while it is still being actively reshaped.
The Golden Rules of AI Storage
Rule One: Keep Git for What Git Does Best
Git should remain the foundation for:
Source code
Infrastructure configuration
Documentation
Small configuration datasets
Experiment definitions
Deployment files
A repository should ideally remain fast enough that developers do not fear cloning it.
For example:
git clone https://example.com/ml-project.git cd ml-project
The experience should be immediate and predictable.
If cloning the repository feels like downloading a data lake, the architecture deserves another look.
Rule Two: Use Local Storage for Active Work
Fast local or attached storage remains extremely valuable.
Use it for:
Development
Data processing
Temporary caches
GPU scratch space
High-speed training access
For example:
rsync -av ./dataset/ /mnt/nvme/training-data/
After training:
rsync -av /mnt/nvme/checkpoints/ ./checkpoints/
Local storage is not obsolete.
It simply belongs to a different layer of the architecture.
Rule Three: Use Object Storage for Scale
Large shared datasets need storage that can grow with the organization.
Object storage is particularly useful for:
Large datasets
Distributed compute
Shared access
Data lake architectures
Analytical systems
Long-term storage
The challenge is selecting an implementation and workflow that minimizes unnecessary complexity.
S3 compatibility remains valuable because the ecosystem already understands it.
Rule Four: Separate Mutable and Immutable Data
This may be the most important rule.
Ask:
Is this data still changing?
If the answer is yes, optimize for efficient iteration.
Ask:
“Is this the final dataset used to produce a reproducible result?”
If the answer is yes, optimize for preservation and versioning.
Trying to use the same workflow for both can create unnecessary operational friction.
Deep Analysis
Architecture Inspection: Measure the Problem Before Rebuilding Everything
Before migrating storage, inspect the current repository and determine where the real bottleneck exists.
Check repository size:
du -sh .git
Inspect the largest files:
find . -type f -printf '%s %p ' | sort -nr | head -20
Check Git LFS objects:
git lfs ls-files
Measure the repository:
git count-objects -vH
These commands help determine whether the repository is becoming a data warehouse rather than a source repository.
Dataset Inspection: Understand What Is Actually Changing
A team should also determine whether large files are changing completely or whether most of their content remains stable.
Calculate file hashes:
sha256sum dataset/.parquet
Compare datasets:
diff -qr old_dataset new_dataset
For directory-level synchronization:
rsync -av --dry-run old_dataset/ new_dataset/
The goal is to understand the difference between logical change and physical change.
A dataset may contain only a few changed records while the storage workflow treats the entire object as new.
That difference can determine whether an AI platform spends minutes or hours moving data.
Pipeline Design: Separate Code, Data, and Compute
A clean AI architecture can be represented as:
Git Repository
|
| Code and Configuration
v
Data Preparation Pipeline
|
v
Mutable Data Storage
|
| Finalized Dataset
v
Versioned Dataset Registry
|
v
Cloud Training
|
v
Model Registry
Each layer has a different responsibility.
This is more maintainable than a single giant repository containing everything.
Training Workflow: Move Only What the Compute Needs
Before starting a training job, synchronize only the required dataset.
For example:
hf buckets sync hf://buckets/organization/training-data ./training-data
Prepare local scratch storage:
mkdir -p /mnt/nvme/ml-training
Move active data:
rsync -av ./training-data/ /mnt/nvme/ml-training/
Monitor available storage:
watch -n 5 df -h
During long-running jobs, monitor disk usage:
watch -n 10 'du -sh /mnt/nvme/ml-training'
The principle is simple.
Remote storage provides durability and collaboration.
Fast local storage provides performance.
Neither layer needs to replace the other.
Data Lifecycle Automation: Promote Instead of Copying Randomly
A mature workflow should automate the movement between storage states.
For example:
./prepare-data.sh ./validate-data.sh ./publish-data.sh ./start-training.sh
The validation stage could verify checksums:
sha256sum -c checksums.txt
The pipeline could then publish the finalized dataset only after validation succeeds.
This creates a clear distinction between experimental data and production data.
Security Analysis: Storage Architecture Is Also a Security Architecture
Data storage decisions affect more than performance.
They also affect access control.
Teams should identify:
Who can read training data
Who can modify datasets
Which credentials are used by CI/CD
How training infrastructure authenticates
Whether secrets are exposed in scripts
Whether temporary artifacts contain sensitive information
Never hardcode credentials:
export ACCESS_KEY="example" export SECRET_KEY="example"
Instead, use an approved secret-management system and short-lived credentials when possible.
Check environment variables carefully:
env | grep -i key
Audit repository history for accidentally committed secrets:
git log --all --full-history
A storage platform that improves performance but creates uncontrolled access is not a successful architecture.
What Undercode Say:
Storage Architecture Is Becoming a Core Part of AI Engineering
The biggest lesson from this discussion is that AI infrastructure cannot be designed as an afterthought.
A model may be brilliant.
The training code may be optimized.
The GPU cluster may be expensive and powerful.
But if the data pipeline spends hours cloning, downloading, duplicating, and rebuilding massive files, the infrastructure is still inefficient.
AI workloads are increasingly data movement workloads.
The industry often focuses on compute.
Everyone talks about GPUs.
Everyone talks about model parameters.
Everyone talks about inference.
But enormous amounts of engineering time and cloud spending can disappear into storage and network operations.
The next generation of AI infrastructure will need to optimize movement as aggressively as it optimizes computation.
Git remains one of the greatest developer tools ever created.
That does not mean Git should become a data lake.
The same principle applies to every technology.
A tool can be excellent and still be the wrong tool for a specific job.
Git should manage the instructions that describe how the model is built.
Storage systems should manage the large-scale artifacts used to build it.
Fast disks should support active computation.
Cloud object storage should provide scale.
Mutable storage should support experimentation.
Versioned repositories should preserve finalized and reproducible results.
The separation may initially appear more complicated.
In reality, it often reduces complexity.
The complexity already exists in the data lifecycle.
A good architecture simply acknowledges it instead of hiding it inside a massive repository.
Another important point is developer experience.
Infrastructure teams sometimes build technically powerful systems that developers dislike using.
If engineers need five environment variables, three configuration files, custom scripts, and a forty-page internal document just to download training data, adoption will suffer.
The best infrastructure often feels almost invisible.
Developers should be able to prepare data, synchronize it, train a model, and publish results without constantly fighting the storage layer.
Deduplication is particularly important for the AI era.
Machine learning produces repetition by design.
Checkpoints are related.
Datasets evolve gradually.
Embeddings may be regenerated with only partial changes.
Agent traces can contain repeated structures.
Model artifacts can share enormous amounts of similar data.
Treating every new artifact as completely independent is becoming increasingly inefficient.
Content-addressable storage introduces a different philosophy.
Instead of asking, “Which file should I upload?”
The system can ask, “Which pieces of this content already exist?”
That is a major architectural shift.
It transforms storage from simple file placement into content-aware infrastructure.
However, organizations should not blindly migrate everything.
Traditional S3 environments remain extremely powerful.
Existing data lakes may already have mature governance, compliance, lifecycle policies, encryption, monitoring, and access controls.
A new storage layer should complement those capabilities rather than automatically replacing them.
The correct architecture depends on the workload.
A startup experimenting with rapidly changing training datasets has different requirements from a financial institution maintaining a regulated enterprise data lake.
The principle remains the same.
Match the storage technology to the data lifecycle.
Another challenge will be reproducibility.
Mutable storage is excellent during experimentation.
But AI teams must know exactly which dataset produced a particular model.
That means the final production workflow should preserve dataset identifiers, hashes, manifests, and metadata.
For example:
sha256sum final_dataset/.parquet > dataset-manifest.sha256
Then preserve the training environment:
pip freeze > requirements.txt
And capture the Git revision:
git rev-parse HEAD > source-commit.txt
This creates a reproducibility record linking code, dependencies, and data.
That combination may matter more than simply placing a 500GB dataset inside a Git repository.
The future of AI storage will likely become increasingly hybrid.
Local NVMe will continue to matter for performance.
Object storage will continue to matter for scale.
Version-controlled systems will continue to matter for reproducibility.
Content-addressable systems will become more attractive as datasets and model artifacts grow.
The organizations that understand this separation early will build faster pipelines.
They will reduce unnecessary cloud transfer.
They will improve developer productivity.
They will make experimentation cheaper.
And perhaps most importantly, they will prevent infrastructure from becoming the invisible bottleneck holding back AI development.
The question is not whether your current storage works today.
The question is whether it will still work when your 500GB dataset becomes 5TB, your team doubles, your experiments multiply, and your cloud training pipeline runs continuously.
That is the moment architecture becomes strategy.
✅ Git is highly effective for source code and small, text-oriented files, while large mutable datasets can create significant workflow friction when forced into Git-centric processes.
✅ Local file storage, object storage, and version-controlled repositories each solve different technical problems, making a hybrid storage architecture a practical approach for many ML workflows.
❌ It would be inaccurate to claim that every machine learning project should abandon Git LFS or traditional S3 storage, because the best design depends on dataset size, update patterns, collaboration requirements, security, compliance, and existing infrastructure.
Prediction
(-1) As AI datasets continue growing from hundreds of gigabytes into multi-terabyte and petabyte-scale environments, organizations that continue treating Git repositories as general-purpose data warehouses will face slower developer workflows, higher storage overhead, and increasingly complex CI/CD pipelines.
Large binary artifacts will become less practical inside standard source repositories.
Data deduplication and content-aware synchronization will become more important as model checkpoints and datasets grow.
Teams that separate mutable experimental data from immutable production datasets will achieve cleaner and more reproducible ML pipelines.
AI storage platforms that combine S3 compatibility, efficient synchronization, content-addressable storage, and strong dataset reproducibility will become increasingly attractive to developers.
The future of machine learning infrastructure will likely be hybrid, combining Git for code, high-performance local storage for compute, cloud-scale object storage for collaboration, and dedicated systems for versioned datasets and model artifacts.
▶️ Related Video (76% Match):
🕵️📝Let’s dive deep and fact‑check.
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
References:
Reported By: huggingface.co
Extra Source Hub (Possible Sources for article):
https://stackoverflow.com
Wikipedia
OpenAi & Undercode AI
Image Source:
Unsplash
Undercode AI DI v2
🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]
📢 Follow UndercodeNews & Stay Tuned:
𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky | 🐘Mastodon | 📺Youtube
