A multi-GPUs test for Oceananigans #4791
Answered
by
simone-silvestri
XinghaoJ
asked this question in
Computational science!
-
|
I can successfully run my simulation with single GPU, but after I changed the architecture from GPU() to Distributed(GPU()), errors occour. Below is simplified julia code for simulation: using Random
using Printf
using CUDA
using Oceananigans
using Oceananigans.Units: minute, minutes, hour, hours, days,day
using MPI
MPI.Init()
#grid setup
....
child_architecture = GPU()
architecture = Distributed(child_architecture)
grid = RectilinearGrid(architecture, size = (Nx, Ny, Nz),
x = (0, Lx),
y = (0, Ly),
z = z_faces,
halo = (3,3,3))
##boundary conditions
...
u_bsc = FieldBoundaryConditions(top=FluxBoundaryCondition(τx));
v_bsc = FieldBoundaryConditions(top=FluxBoundaryCondition(τy));
b_bsc = FieldBoundaryConditions(top=FluxBoundaryCondition(B_surf),bottom = GradientBoundaryCondition(dbdz_interior));
boundary_conditions=(u=u_bsc, v=v_bsc, b=b_bsc);
##initial conditions and model setup
bᵢ(x,y,z) = ...
gaussian_mask = GaussianMask{:z}(center=-grid.Lz, width=grid.Lz/10)
u_sponge = v_sponge = w_sponge = ...
b_sponge = ...
model = NonhydrostaticModel(; grid,
advection = WENO(order=5),
buoyancy = BuoyancyTracer(),
coriolis = FPlane(f=0.729e-4),
closure = AnisotropicMinimumDissipation(),
boundary_conditions,
tracers = :b,
timestepper = :RungeKutta3,
forcing = (u=u_sponge, v=v_sponge, w=w_sponge, b=b_sponge)
set!(model, b=bᵢ)
simulation = Simulation(model, Δt=5.0, stop_time=2days)
wizard = TimeStepWizard(cfl=1.0, max_change=1.1, max_Δt=10minute)
simulation.callbacks[:wizard] = Callback(wizard, IterationInterval(10))
##Print a progress message
progress_message(sim) = @printf("Iteration: %04d, time: %s, Δt: %s, max(|w|) = %.1e ms⁻¹, wall time: %s\n",
iteration(sim), prettytime(sim), prettytime(sim.Δt),
maximum(abs, sim.model.velocities.w), prettytime(sim.run_wall_time))
add_callback!(simulation, progress_message, IterationInterval(20))
run!(simulation)I run this julia script in an interactive mode with 2 GPUs: The run commands are: However, I got error.txt as below: 
The code in long_test.jl:96 is: Apart from errors, I also have doubts about if it's right to wrrite command as to run our simulation in two GPUs? |
Beta Was this translation helpful? Give feedback.
Answered by
simone-silvestri
Sep 19, 2025
Replies: 2 comments 6 replies
-
|
I could be wrong but I don't think you need to manually set the visible devices. With two processes you will use device 0 and 1 by default anyways? Since the segmentation fault occurs at bᵢ(x,y,z) = ...so I am not sure what the problem is. But one way to test this is to use |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
If which fills halos and communicates between devices. But |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I thought the format is recognized automatically. I will format it for later comments. |
Beta Was this translation helpful? Give feedback.
-
|
I don't know why model = NonhydrostaticModel(;grid=distributed_grid,
advection = WENO(order=5),
buoyancy = BuoyancyTracer(),
coriolis = FPlane(f=0.729e-4),
closure = AnisotropicMinimumDissipation(),
boundary_conditions,
tracers = :b,
timestepper = :RungeKutta3)And than errors show up at next line: set!(model, b=bᵢ)If I comment |
Beta Was this translation helpful? Give feedback.
-
|
it might be that your cuda-aware MPI is not enabled. I see that you are using openmpi. You can try doing using MPI
MPI.has_cuda()if the output is false, then it means that cuda-aware MPI is not active. You can find more information here https://juliaparallel.org/MPI.jl/stable/knownissues/#CUDA-aware-MPI |
Beta Was this translation helpful? Give feedback.
Answer selected by
XinghaoJ
-
|
Hi, I have got a similar problem. How did you fix it? Thanks！ using CUDA
using Oceananigans
using Oceananigans.Units: minute, minutes, hours
using Oceananigans.Units: GiB, MiB, KiB
using Statistics
using Oceananigans.TurbulenceClosures: viscous_flux_uz
using CairoMakie
using MPI
if !MPI.Initialized()
MPI.Init()
end
child_architecture = GPU()
architecture = Distributed(child_architecture)
const H=15 #/m
grid = RectilinearGrid(architecture, size=(128,128,128), extent=(π*H, π*H/2, H),halo = (4,4,4))
const u★=0.01 #friction velocity
const g_Earth = 9.81 # m/s
Fx(x,y,z,t)=u★^2/H #forcing
z₀ = 0.01 # m (roughness length)
κ = 0.4 # von Karman constant
z₁ = grid.Lz+znodes(grid, Center())[1] # Closest grid center to the bottom
cᴰᵇ = (κ / log(z₁ / z₀))^2 # Drag coefficient
@inline drag_u(x, y, t, u, v, p) = - p.cᴰᵇ * √(u^2 + v^2) * (u)
@inline drag_v(x, y, t, u, v, p) = - p.cᴰᵇ * √(u^2 + v^2) * (v)
drag_bc_u = FluxBoundaryCondition(drag_u, field_dependencies=(:u, :v), parameters=(; cᴰᵇ))
drag_bc_v = FluxBoundaryCondition(drag_v, field_dependencies=(:u, :v), parameters=(; cᴰᵇ))
u_bcs = FieldBoundaryConditions(bottom = drag_bc_u)
v_bcs = FieldBoundaryConditions(bottom = drag_bc_v)
T_bcs = FieldBoundaryConditions(top=FluxBoundaryCondition(0.0),
bottom=FluxBoundaryCondition(0.0))
S_bcs = FieldBoundaryConditions(top=FluxBoundaryCondition(0.0),bottom=FluxBoundaryCondition(0.0))
model = NonhydrostaticModel(; grid,
advection = WENO(order=9),
timestepper = :RungeKutta3,
tracers =(:T,:S),
buoyancy = SeawaterBuoyancy(),
closure = AnisotropicMinimumDissipation(),
boundary_conditions = (u=u_bcs,v=v_bcs,T=T_bcs,S=S_bcs),
forcing=(u=Fx,))
Ξ(z) = randn()*exp(z/4)
uᵢ(x, y, z) = u★ * 1e-2 * Ξ(z) + 0.21
wᵢ(x, y, z) = u★ * 1e-2 * Ξ(z)
vᵢ(x, y, z) = u★ * 1e-2 * Ξ(z)
Tᵢ=290 #K
Sᵢ=35 #PSU
set!(model)
simulation = Simulation(model, Δt=0.4, stop_time=24hours)
wizard = TimeStepWizard(cfl=0.8, max_change=1.1, max_Δt=0.1minute)
simulation.callbacks[:wizard] = Callback(wizard, IterationInterval(10))
using Printf
function progress(simulation)
u, v, w = simulation.model.velocities
## Print a progress message
msg = @sprintf("i: %04d, t: %s, Δt: %s, umax = (%.5e, %.5e, %.5e) ms⁻¹, wall time: %s\n",
iteration(simulation),
prettytime(time(simulation)),
prettytime(simulation.Δt),
maximum(abs, u), maximum(abs, v), maximum(abs, w),
prettytime(simulation.run_wall_time))
@info msg
return nothing
end
simulation.callbacks[:progress] = Callback(progress, IterationInterval(50))
run!(simulation) |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
it might be that your cuda-aware MPI is not enabled. I see that you are using openmpi. You can try doing
if the output is false, then it means that cuda-aware MPI is not active. You can find more information here https://juliaparallel.org/MPI.jl/stable/knownissues/#CUDA-aware-MPI