All Projects

2026

Decentralizing EXAMM — A Peer-to-Peer Architecture for Fault-Tolerant Evolutionary Model Training

A complete architectural redesign of the EXAMM neuroevolution framework from a centralized manager-worker topology to an asynchronous peer-to-peer ring, achieving 2.5× higher genome throughput and 23% lower prediction error with snapshot-based fault recovery.

With Ashrith Mudundi·Diana Velychko

C++ MPI Neuroevolution Distributed Systems Fault Tolerance Island Model
View Report Download PDF
Decentralizing EXAMM — A Peer-to-Peer Architecture for Fault-Tolerant Evolutionary Model Training — image 1
Decentralizing EXAMM — A Peer-to-Peer Architecture for Fault-Tolerant Evolutionary Model Training — image 2
Decentralizing EXAMM — A Peer-to-Peer Architecture for Fault-Tolerant Evolutionary Model Training — image 3
Decentralizing EXAMM — A Peer-to-Peer Architecture for Fault-Tolerant Evolutionary Model Training — image 4
Decentralizing EXAMM — A Peer-to-Peer Architecture for Fault-Tolerant Evolutionary Model Training — image 5
Decentralizing EXAMM — A Peer-to-Peer Architecture for Fault-Tolerant Evolutionary Model Training — image 6

The Problem

EXAMM is a neuroevolutionary framework for training recurrent neural networks on distributed hardware. Its original design used a centralized manager-worker architecture — one node controlled everything: genome generation, parent selection, crossover, and global state. That single node was both the system’s throughput bottleneck and its greatest point of failure. A crash meant losing potentially weeks of compute time with no recovery path.

What We Built

We redesigned EXAMM from scratch as an asynchronous peer-to-peer ring where every node is equal. No manager. Each peer independently runs its own evolutionary islands, shares high-fitness genomes with neighbors, and handles its own failure detection and recovery.

Key mechanisms:

  • Deterministic structural hashing — every peer independently agrees on genome ownership without any central coordination, preventing redundant evaluations across the network
  • Dual-mode genome replication — MIGRATE routes genomes to their canonical owner by structural hash; BACKUP propagates them around the ring as a geographic failsafe
  • Snapshot-based fault recovery — each peer continuously maintains a rolling checkpoint of its neighbor’s island state; when a peer crashes and rejoins, it restores with state at most 10 seconds stale instead of rebuilding from a single genome
  • Token-ring termination consensus — a circulating token accumulates per-peer completion flags; the system shuts down only when every peer independently agrees it’s done

Results

Evaluated across 54 independent trials on the NGAFID C172 aviation time-series dataset (predicting pitch angle from 32 flight sensor parameters):

ConditionFinal MSEvs. Centralized
Centralized EXAMM (baseline)2.15 × 10⁻³—
P2P EXAMM (fault-free)1.64 × 10⁻³−23%
1-peer dropout, improved FT1.93 × 10⁻³−10%
3-peer dropout, improved FT2.06 × 10⁻³−4%

The fault-free P2P system also achieved 2.5× higher genome throughput than the centralized baseline. With snapshot recovery, the system still outperformed centralized EXAMM after losing 75% of the cluster for over a minute — recovering peers rejoined with full island populations and immediately resumed high-fitness search.