DDStore

Efficient distributed data loading for distributed data-parallel (DDP) training.

Each rank holds a shard of the dataset in memory; any rank reads any sample by global index through one-sided remote memory access (MPI RMA or libfabric RDMA), without coordination.

• Batched reads: get_batch() reads a whole batch in one call.

• GPUDirect RDMA: data can live in, and be read straight into, GPU memory (details).

• PyTorch integration: pyddstore.torch: DistDataset (any map-style dataset), WindowedDataset (multi-row samples), ThreadDataLoader (threads; safe with MPI and GPU buffers). PyTorch’s DataLoader with forked workers also works (methods 1 and 2).

• Thread-safe reads, a read-time profiler, and a split mode (method=2) where one job reads data published by another.

• MPI optional: methods 1 and 2 also run without MPI (DDSTORE_MPI=0), launched by srun.

Getting started

• Installation
  • Prerequisites
  • Building and installing
  • Building without MPI
  • Notes
• Quick start

User guide

• Backends
  • MPI RMA (method=0, default)
  • libfabric RDMA (method=1)
  • File-based handshake (method=2)
  • Without MPI (methods 1 and 2)
  • Partitioned / Sub-communicator Usage
• GPUDirect RDMA
• PyTorch integration
  • Forked DataLoader workers
  • Without MPI
• HPC systems (Slurm, Slingshot)
  • Slurm job scripts
  • Slingshot VNIs and --network
  • Troubleshooting (cxi)
  • Perlmutter filesystems
• Performance
• Concurrency

Reference

• PyDDStore reference
  • PyDDStore(comm_or_none=None, method=0, handshake_dir="", n_core=0, nic_map=None, rank=None, size=0, exchange_ns="")
  • init(name, nrows, disp, itemsize=1)
  • add(name, arr)
  • update(name, arr, offset)
  • get(name, arr, start=0)
  • get_batch(name, arr, indices)
  • register_recv(name, arr) / unregister_recv(name, arr)
  • get_profile(name)
  • join(name)
  • fork_reader()
  • info(name)
  • epoch_begin() / epoch_end()
  • free()
  • pyddstore.HAVE_MPI
  • pyddstore.FileComm(dir, rank, size, ns)
• pyddstore.torch reference
  • DistDataset
  • DistDatasetReader
  • WindowedDataset
  • row_of()
  • ThreadDataLoader
• Environment variables

More

• Testing
• DDStore measurements and findings
• Citation and license