A Cluster filesystem can scale beyond as every node must only offer a part of the overall performance. Above 1 GB/s you are in the region of a very special and not common setup.Īs ZFS is a local high security filesystem, performance is limited by the performance of a local pool (and not as good as filesystems without the security of ZFS). With Jumboframes and tuning this can double. 10G Ethernet without tcp tuning and Jumboframes lands at around 400 MB/s. When your pool can satisfy your needs, then it comes to network. Optane can land at around 800 MB/s sequentially. If you care about sync performance this is again very special. Then you can reduce to less Optane, NVMe or 12G SAS/ Sata SSD instead or use other cost limiting factors like dedicated Slog/ special vdev with a slower pool. This marks the upper end of what is possible with ZFS on your hardware. What I would try is to start with some Optane 900 in a raid-0 setup with as much RAM as available and a recsize like 32k or 64k when using VMs.

If there is a lot of concurrent or small io, this will be very different. But the info that a 5 node cluster of 24nvme drives each might be needed would be helpful (if you actually have seen it deliver and not read the specs only )ĥ00 MB/s pool performance sequentially is not a problem. If you think an off-the-shelf all flash array can then lets hear it too (although most are not catering for single users), but o/c that is most likely not the way to go. Now lets have a look at this regardless of the hardware I already have - if I had unlimited funds what hardware would be capable of achieving this?

I want a write speed of 500MB/s per VM when moving from off system to on system and vice versa (cold or hot vmotion).

I want the ZFS system to be HA capable with at least 2 nodes.

I want to run an ESXi cluster with a shared storage on a ZFS system, ideally via nfs (so sync writes).

Now I am totally aware that this is highly depending on what the goal/expectation is so here are mine (I am sure I have stated these quite often in my various rants/complain threads but just to have all in one place). I will leave all the old issues out for now and start with a fresh idea if possible (but of course reuse in case I can in the actual build).

So I thought I'd start fresh and ask for help from real experts since my tinkering didnt work out So I've been trying to build the ultimate ZFS storage system for a while now, and have thrown tons of time and probably even more money at it but without actually getting *there*.

Esxi nfs datastore