Testing

Unit tests (pytest)

Install test dependencies:

pip install pytest pytest-mpi

Single-rank — no cluster required, covers all dtypes, add/get/init/update, and error cases:

mpirun -n 1 python -m pytest test/test_single.py -v

Multi-rank — verifies remote reads across all rank pairs and sub-communicator grouping:

mpirun -n 4 python -m pytest test/test_multirank.py -v

Batched reads and the PyTorch layer — method 0 everywhere; method 1, method 2 and GPU cases run inside a Slurm step with a CXI device (provider from DDSTORE_FABRIC, default cxi):

mpirun -n 4 python -m pytest test/test_get_batch.py test/test_torch.py -v

GPUDirect RDMA — requires a live cxi fabric and a CUDA/HIP GPU per rank (skipped automatically otherwise); see GPUDirect RDMA:

DDSTORE_FABRIC=cxi mpirun -n 2 python -m pytest test/test_gpu_rdma.py -v

Test file	Min ranks	What is tested
test/test_single.py	1	All dtypes, add/get, init/update/get, error handling, double free()
test/test_multirank.py	2 (4 recommended)	Remote reads, shard boundaries, multiple variables, ddstore_width grouping
test/test_gpu_rdma.py	2	GPU-resident add()/get() in both directions, both libfabric methods, negative/error cases
test/test_get_batch.py	2 (4 recommended)	get_batch(): shuffled indices across ranks with repeats, single row, dtypes, error recovery, GPU destination, concurrent threads, registered destination buffers, wide rows (with DDSTORE_MAX_READ_BYTES=4096 they are read in pieces); method 0, plus method 1 over cxi inside a Slurm step
test/test_file_exchange.py	2 (4 recommended)	Methods 1 and 2 without an MPI communicator (rank=/size=, records exchanged through files): get/get_batch, uneven shards with init/update, several stores and variables, stale exchange files, an extra member joining, same layout as the MPI exchange, shape mismatch, bad arguments, GPU buffers; inside a Slurm step with a CXI device
test/test_torch_nompi.py	2 (4 recommended)	pyddstore.torch without MPI (never imports mpi4py; either build): DistDataset over a FileComm and over a torch.distributed gloo group, methods 1 and 2, default method 1, ddstore_width groups, a DistDatasetReader, PyTorch DataLoader with forked workers (methods 1 and 2, a reader, the kdreg2 error), errors raised on every rank, method=0 refused, FileComm.allgather; launch with srun inside a Slurm step with a CXI device
test/test_torch.py	2 (4 recommended)	pyddstore.torch: tuple/dict/single samples of every field kind, numpy records, loaders vs the plain source (direct batch reads on and off), ThreadDataLoader per-thread readers, iterators and prefetch depth, read_rows/alloc buffers, reuse_buffers (also with GPU buffers), WindowedDataset, row_of, encode/decode/fields, chunked loading, error handling, ddstore_width, GPU placement, DistDatasetReader
test/test_cpu_nic_map.py	none (plain pytest)	CPU→NIC map from fake sysfs trees (Frontier, Perlmutter GPU/CPU nodes, Aurora, no NUMA info), the hwloc-calc fallback, and on a node with hsn* NICs the same map as hwloc-calc

test_file_exchange.py and test_torch_nompi.py create their files under DDSTORE_TEST_DIR (default: the current directory), which every rank must see. Use Lustre (Frontier, Perlmutter $SCRATCH); on Perlmutter CFS the file handshake is erratic across nodes and torch’s file:// rendezvous fails (no flock() on compute nodes). Run test_torch_nompi.py on its own.

Integration scripts

# Basic functional test (libfabric, method=1)
mpirun -n 4 python examples/scripts/demo.py

# Integration test with PyTorch DDP (libfabric, method=1)
mpirun -n 4 python examples/scripts/test.py

Optional arguments for examples/scripts/demo.py and examples/scripts/test.py:

Flag	Default	Description
--num	1048576	Rows per rank
--dim	64	Elements per row
--nbatch	32	Number of random reads
--gloo / --nccl	--gloo	test.py only: torch.distributed backend

Method 2 (file-based handshake)

Two separate launches sharing a handshake directory on a shared filesystem — not a single mpirun, since core and extra are independent jobs:

# Terminal 1 — core (data-holding) side
mpirun -n 4 python test/test_method2_core.py /path/to/shared/ddstore_hs

# Terminal 2 — extra (reader) side, after or while the core side is running
python test/test_method2_extra.py /path/to/shared/ddstore_hs 4