Update cmake script #53
Conversation
src/CMakeLists.txt
Outdated
| hip::device) | ||
| list(APPEND CMAKE_BRANSON_CXXFLAGS -DUSE_GPU) | ||
| list(APPEND CMAKE_BRANSON_CXXFLAGS -DHAS_GPU) | ||
| list(APPEND CMAKE_BRANSON_CXXFLAGS -x hip) |
There was a problem hiding this comment.
Does appending these as CXX flags work for all HIP compilation workflows?
There was a problem hiding this comment.
@alexrlongne branson source uses -DUSE_GPU (particle_pass_transport.h) and -DHAS_GPU (replicated_transport.h, gpu_setup.h and history_based_transport.h); hence we need those two flags. I've changed the HIP cmake workflow to remove explicit inclusion of the -x hip flag. We now use find_package(HIP) and enable_langauge(HIP) to handle the required compiler flags and link-time dependencies
| if(caliper_FOUND) | ||
| set( branson_deps "caliper;${branson_deps}") | ||
| message(STATUS "Looking for adiak...") | ||
| find_package(adiak REQUIRED) |
There was a problem hiding this comment.
Sorry, what is this package and why is it required? There's no mention of adding it in the PR description.
There was a problem hiding this comment.
@alexrlongne Adiak is used to collect metadata (program parameters, environment, versions etc.) about the experiment. Caliper when built with Adiak records this metadata in the .cali files.
src/CMakeLists.txt
Outdated
| if(ENABLE_CUDA) | ||
| set_target_properties(BRANSON | ||
| PROPERTIES | ||
| CUDA_SEPARABLE_COMPILATION ON |
There was a problem hiding this comment.
Separable compilation is not required in Branson (all GPU kernels have a single compilation unit). Was this required to get Branson to build for a certain compiler or set of tools?
There was a problem hiding this comment.
@alexrlongne The separable compilation flag is not required and has been removed.
src/CMakeLists.txt
Outdated
| if(ENABLE_HIP) | ||
| set_target_properties(BRANSON | ||
| PROPERTIES | ||
| HIP_SEPARABLE_COMPILATION ON |
There was a problem hiding this comment.
Same comment as above.
There was a problem hiding this comment.
@alexrlongne The separable compilation flag is not required and has been removed.
There was a problem hiding this comment.
Was there a reason to remove the function definitions for finding third party libraries and moving them into the main CMake?
There was a problem hiding this comment.
@alexrlongne The changes to the main CMakeLists.txt - especially with the new cmake options added - were substantial enough that it was easier to put the TPL definitions in the main cmake itself. Given the number of cmake changes, I'd like to know if this breaks any existing builds.
|
|
||
| auto sync_error = cudaDeviceSynchronize(); | ||
| if(!sync_error) std::cout<<"ERROR: <GPU_Type>DeviceSynchronize"<<std::endl; | ||
| if(sync_error != cudaSuccess) std::cout << "ERROR: <GPU_Type>DeviceSynchronize (pre-kernel): " << sync_error << std::endl; |
There was a problem hiding this comment.
@alexrlongne Fixed the error check for cudaDeviceSynchronize. Currently the code (incorrectly) prints the error message on cudaSuccess
| Insist(!(cudaGetLastError()), "CUDA error in transport kernel launch"); | ||
| sync_error = cudaDeviceSynchronize(); | ||
| if(!sync_error) std::cout<<"ERROR: <GPU_Type>DeviceSynchronize"<<std::endl; | ||
| if(sync_error != cudaSuccess) std::cout << "ERROR: <GPU_Type>DeviceSynchronize (post-kernel): " << sync_error << std::endl; |
There was a problem hiding this comment.
@alexrlongne Fixed the error check for cudaDeviceSynchronize. Currently the code (incorrectly) prints the error message on cudaSuccess
| wrapped_cali_mark_begin("particle pass soa"); | ||
| imc_particle_pass_driver<PhotonArray>(mesh, imc_state, imc_p, mpi_types, mpi_info); | ||
| wrapped_cali_mark_end("particle pass soa"); | ||
| } | ||
| else if(input.get_particle_storage() == SOA) { | ||
| wrapped_cali_mark_begin("particle pass aos"); | ||
| imc_particle_pass_driver<std::vector<Photon>>(mesh, imc_state, imc_p, mpi_types, mpi_info); | ||
| wrapped_cali_mark_end("particle pass aos"); | ||
| } | ||
| else { | ||
| cout << "Driver for array currently not supported" << endl; | ||
| exit(EXIT_FAILURE); | ||
| } | ||
| } | ||
| else if (input.get_dd_mode() == REPLICATED) { | ||
| if( input.get_particle_storage() == SOA) { | ||
| wrapped_cali_mark_begin("replicated soa"); | ||
| imc_replicated_driver<PhotonArray>(mesh, imc_state, imc_p, mpi_types, mpi_info); | ||
| wrapped_cali_mark_end("replicated soa"); | ||
| } | ||
| else if(input.get_particle_storage() == AOS) { | ||
| wrapped_cali_mark_begin("replicated aos"); | ||
| imc_replicated_driver<std::vector<Photon>>(mesh, imc_state, imc_p, mpi_types, mpi_info); | ||
| wrapped_cali_mark_end("replicated aos"); |
There was a problem hiding this comment.
@alexrlongne @pearce8 Added caliper instrumentation around the transport step
| wrapped_cali_mark_begin("gpu transport photons"); | ||
| gpu_transport_photons(rank_cell_offset, all_photons, gpu_setup.get_device_cells_ptr(), cell_tallies); | ||
| wrapped_cali_mark_end("gpu transport photons"); | ||
| wrapped_cali_mark_begin("post process photons"); | ||
| auto batch_complete = post_process_photons(next_dt, all_photons, census_list, census_E, exit_E); | ||
| wrapped_cali_mark_end("post process photons"); | ||
| t_transport.stop_timer("gpu transport"); | ||
| std::cout<<"gpu transport time: "<<t_transport.get_time("gpu transport")<<std::endl; |
There was a problem hiding this comment.
@alexrlongne @pearce8 Added caliper instrumentation around the replicated GPU transport step
2 participants
The PR updates the cmake script to build the code independent of any pre-loaded modules or environment variables. The following options have been added to the CMakeLists.txt
If ENABLE_CALIPER=ON, the caliper_DIR variable can be used to specify the location of an already existing caliper installation
Similarly, with ENABLE_METIS=ON, METIS_ROOT_DIR can be set to an existing METIS installation