Run Models Bigger Than Any of Your GPUs: SilentSwarm Is Now in Open Beta

There is a quiet frustration in every homelab and small ML team: the model you want to fine-tune doesn't fit on the GPU you have. The 24 GB card can't hold the 14B model. The second machine with the older 12 GB card sits idle, because getting two mismatched boxes on different networks to train one model together is a networking-and-DevOps project of its own.

SilentSwarm is our answer. It is a self-hosted system that splits a large language model into pipeline stages, spreads those stages across the machines you already have (mismatched GPUs, different servers, even behind NAT and firewalls) and lets you train, fine-tune and serve as if it were one machine. Today we're opening the beta.

What it does

Pipeline parallelism without the platform team. SilentSwarm loads any HuggingFace causal LM (GPT-2, Qwen2.5, Llama-family and others), inspects the free VRAM on every GPU in your swarm, and assigns more layers to stronger cards automatically. Each node holds only its stage, so the model ceiling is the sum of your cards, not the largest one.

Train less, fit more. Full fine-tuning is one option. The more interesting one: freeze the pretrained backbone and train only a small learned compression layer at the stage boundary. That "bottleneck" approach cuts inter-node traffic and raises the trainable-model ceiling roughly 6x compared to full fine-tuning on the same hardware, which matters when your "cluster" is a gaming PC and a used server.

Our largest validated run so far shows what that looks like in practice: a Qwen2.5-3B model split across a weak node with 2x6 GB GPUs and a strong node with 16 GB and 24 GB cards. The backbone stays frozen; only the bottleneck compressor at the stage boundary (2048 to 384 dimensions, int8) trains. The full model fits because no node ever needs to hold all the gradients.

SilentSwarm architecture: a torch-free leader control plane, with Fellow A holding pipeline stage 0 and Fellow B holding stage 1, exchanging compressed activations and gradients directly over ZeroMQ

No network heroics. Worker nodes ("fellows") connect out to the leader and communicate through its built-in relay, so it works across NAT, firewalls and home connections with zero port-forwarding. If nodes can reach each other, transport selection upgrades them to direct fellow-to-fellow ZeroMQ links automatically. An optional WireGuard-based mesh (Tailscale, or fully self-hosted with Headscale) makes that direct path the default: every node gets a stable overlay address, activations and gradients flow peer to peer, and the leader relay stays as a fallback instead of a bottleneck.

Ten minutes to a swarm. The control plane is a single docker compose up, and it is torch-free, so it runs on any small box. Only the GPU fellows need PyTorch. Adding a GPU node is one curl command copied from the dashboard; it bootstraps its own environment (source download, virtualenv, dependencies) and appears in the live swarm graph within seconds. The bootstrap is HTTPS-only, checksum-verified, and can be locked to invite tokens. If you'd rather not pipe a script you haven't read, the docs include a read-before-run flow: fetch, verify the checksum, inspect, then run.

From training run to chat. Trained models export automatically into an inference catalog. Load one from the web UI and chat with it in the browser, or hit it through the OpenAI-compatible API from your existing tools. Training and inference run on the same topology: the same swarm that trained the model serves it, with the head stage streaming tokens back through the relay.

What it looks like

The Training console shows your swarm as a live graph: every fellow a node, colored by state, with GPU inventory, VRAM and throughput one click away. Jobs stream live loss curves; fellow logs stream straight into the browser. The Inference page is a plain model-picker-and-chat: load, wait for the green light, talk to your model.

If you prefer the terminal, everything is also driven by the swarm CLI: swarm nodes for cluster status, swarm sweep --name <config> to launch a run, swarm watch <job_id> for live loss, swarm infer for one-shot prompts, and swarm update to push new code to every connected fellow at once.

How it compares

Petals, Exo, GPUStack and Ray all touch parts of this space. What sets SilentSwarm apart is the combination: learned bottleneck compression that makes pipeline parallelism over normal internet connections economical, NAT-friendly transport with zero network configuration, one-curl node bootstrap, and the fact that the same self-hosted topology both trains and serves your model.

Honest beta notes

This is a soft launch, and we'd rather set expectations straight:

Distributed training currently supports two pipeline stages end-to-end (real activation and gradient tensors over ZeroMQ, with the leader relay as fallback). Generalizing to N stages is the top item on the roadmap.
WikiText-103 is the built-in dataset; the loader is pluggable.
Expect rough edges. That's what the beta is for, and it's why we recommend token-gating the fellow bootstrap while you kick the tires.

The soft launch also means access is gated for now: SilentSwarm currently runs with a group of test users, and you can register at silentswarm.nexpatch.ai to become one of them. Once the soft launch is over, the source code will be published openly for everyone to self-host. If you have questions in the meantime, write to info@nexpatch.ai.

We're also honest about where this is going. SilentSwarm today is Phase 1 of three: a trusted cluster of machines you own or control. Phase 2 is federated pools, where friends or organizations share capacity (accounts, pools with role-based quotas and node credentials are already in; metering is not). Phase 3 is an open compute marketplace between strangers, which needs sandboxing, fault tolerance and reputation before we'll claim it. The roadmap is public and meant to be shaped in the open.

Licensing: free for you, paid for companies

After the soft launch, SilentSwarm's source will be published openly under the PolyForm Noncommercial license: self-host it for personal projects, research and homelabs for free, forever. Commercial use requires a paid license; that's what funds the project. If you want to run it inside a company, talk to us.

Try it

During the soft launch, the fastest way in is to register as a test user at silentswarm.nexpatch.ai. Once the source is published after the soft launch, self-hosting will be this simple:

git clone https://github.com/Franzelfx/nxpSilentSwarm.git && cd nxpSilentSwarm
docker compose up -d --build leader frontend
# open http://localhost:3000, the first account becomes admin