Multi-GPU performance/benchmarks

[rikingurditta]

Hello,

I’m quite new to this project, I’m working with @francispoulin on attempting to profile multi-GPU performance. In particular, we have been struggling to reproduce the multi-GPU speedups recorded in the benchmarks page of the documentation (I understand that this page is in the process of being updated.)

Firstly we cannot find the corresponding script to generate that multi-GPU data in the benchmark folder. (It may be useful to add script names to the benchmark results page.) We encountered this file for benchmarking with MPI, and it mentions GPUs in the comments and uses @sync_gpu, but it ultimately uses a distributed CPU architecture arch = Distributed(CPU(), …). In fact I cannot find any benchmark files that use a distributed GPU architecture. So which file do these multi-GPU results come from?

In its stead, we have instead tried to update and use the scripts in validation/distributed_simulations/distributed_scaling, and have found significant slowdown for strong scaling of the non-hydrostatic model when using the Distributed(GPU(), ...) architecture, on a single machine (Pelee at UWaterloo) using 1 vs 2 vs 4 GPUs. Is this expected of the current version of Oceananigans? I see that others are happy using many more GPUs, so I wonder if there is some property of these tests that makes them problematic on multiple GPUs.

Thank you,

Rikin

[glwagner]

@simone-silvestri

[francispoulin]

Thanks @rikingurditta for posting this. Can you share the simple code you are using?

@simone-silvestri , might you want to have a zoom meeting sometime to discuss this? We are happy to work around your schedule.

[francispoulin]

Thanks @simone-silvestri , we'll get back to you in the next couple of days with a reply and maybe we can plan to meet on Monday, or sometime next week?

[rikingurditta]

Hi @simone-silvestri, sorry for the delay in scheduling a meeting, we're having some slowdowns. In the meantime we've been wondering, can you elaborate on your note on the non-hydrostatic model not scaling? Is there an explanation of the scaling problem somewhere in the documentation?

[francispoulin]

Just to add to this question, I know that a 3D transpose in MPI is usually the slowest part of the Poisson solve. I know that other codes do this in CPUs and can be very efficient. Is there something about GPUs that makes this not efficient, or just that Oceananigans does not do a very fancy enough version to make it efficient. Thanks in advance!

[simone-silvestri]

I am not sure about it. We implemented our custom version of the 3D transpose which in general is a very simple operation, with most of the time is spent in the MPI.AlltoAll communication. If you have a reference on how other codes make it efficient we can try optimizing what we have.

[francispoulin]

Not sure if we need to optimized at this stage. I'm just hoping for us to first learn how things scale and then learn where the bottlenecks are. If this is the biggest one, maybe it's a place to start, but first we need to do some simulations and explore what results we get. Thanks!