   #[1]Monte Carlo eXtreme: GPU-based Monte Carlo Simulations

Frequently Asked Questions about MCX

          1. [2]I am getting a "kernel launch timed-out" or "unspecified
          launch failure" error, what is that?

          2. [3]Do I need to install CUDA to run MCX? Can I run MCX without
          an NVIDIA GPU?

          3. [4]Should I use the atomic version of MCX?

          4. [5]How do I interpret MCX's output data?

          5. [6]How do I read the .jnii, .bnii and .jdt output files?

          6. [7]Why do two runs with the same seed give slightly different
          results?

          7. [8]Does MCX support multiple GPUs in a single computer?

          8. [9]Will you consider porting MCX to MPI to run on my cluster?

          9. [10]What is the maximum number of media types that MCX can
          handle?

          10. [11]What is the maximum number of detectors in MCX?

          11. [12]My simulation created an empty detected photon file, why
          is that?

          12. [13]MCX stopped working on my Linux laptop after a suspend,
          what should I do?

1. I am getting a "kernel launch timed-out" or "unspecified launch
failure" error, what is that?

   Answer: This error happens when you are using a non-dedicated GPU, i.e.
   a graphics card that is used both for display and GPU computation. The
   graphics driver imposes a time-limit to the response time of a kernel
   (a function running on a GPU). This time-limit is referred to as the
   "driver watch-dog time limit". On Windows, this limit is about 2
   seconds; on Linux, it applies only when an X server runs on that GPU.
   When a kernel runs on a GPU for longer than this limit, the driver will
   kill this kernel.

   For Windows users, you may increase the TdrDelay value in the registry
   to extend this time-out limit. The MCX Studio installer sets this value
   for you. If you use the zip packages, please open the
   [14]mcx/setup/win64 folder in a file browser, right click on the file
   "apply_timeout_registry_fix.bat" and select "Run as administrator". You
   should see a command line window reporting success; then you must
   reboot your computer before this setting becomes effective. This is
   important!

   If you have only one graphics card on your system and you have to use
   it in a non-dedicated way, MCX also allows you to split the simulation
   into chunks, so that the run-time of each chunk is shorter than the
   watch-dog time limit. This is done by setting a negative "-r"
   (repetition) parameter, for example, "-r -10" runs the photons in 10
   batches.

   If your computer has more than one GPU, you can connect your display to
   one of the cards (the weaker one, or the integrated GPU), and use the
   other card for computation (selected with -G). On Linux, you may also
   use nvidia-prime to let the Intel integrated GPU drive the display, or
   run mcx without a graphical desktop.

2. Do I need to install CUDA to run MCX? Can I run MCX without an NVIDIA
GPU?

   Answer: The released mcx binaries, MCXLAB mex files and pmcx modules
   are statically linked with the CUDA runtime library. You only need an
   NVIDIA GPU and an up-to-date NVIDIA driver; the CUDA toolkit is only
   needed if you compile MCX from the source code.

   If you have an AMD GPU, you can compile MCX from the source code using
   ROCm/HIP (cmake -DUSE_HIP=ON, see Setup_MCX.txt). For Intel/AMD/Apple
   GPUs and CPUs, please use [15]MCX-CL (mcxcl, mcxlabcl or "pip install
   pmcxcl"), which accepts the same input files and command line options
   as MCX. On macOS, pmcx installs only the utility functions, and
   pmcx.mcxlab() runs the simulation using pmcxcl.

3. Should I use the atomic version of MCX?

   Answer: In an MCX simulation, many GPU threads may write photon weights
   to the same voxel at the same time. To avoid such race conditions, CUDA
   provides "atomic" operations, whose read-compute-write process can not
   be interrupted by other threads.

   On the first generation CUDA devices made around 2008, atomic
   operations came with a great speed penalty, so early MCX releases also
   provided a non-atomic mode. This overhead was removed in all NVIDIA
   GPUs made after 2010 (Fermi). Starting from MCX v2026, atomic
   accumulation is always used; the --atomic flag is deprecated and
   ignored, because the non-atomic mode lost a significant fraction of the
   deposited energy on modern GPUs at no speed benefit.

4. How do I interpret MCX's output data?

   Answer: By default, MCX outputs the time-resolved fluence rate
   (1/(mm^2 s)) normalized to a unitary source, i.e. the Green's
   function, at each voxel and time gate. Multiplying it by the time gate
   width gives the fluence of each gate, and summing over all time gates
   gives the continuous-wave fluence. Other quantities can be selected
   with -O, such as fluence (-O F), energy deposition (-O E), replay-based
   Jacobians (-O J/P/R/S), or adjoint Jacobians (-O A/D/U/V/W/Q). Please
   read the "Interpreting the Output" section of README.md, and the
   [16]output interpretation of MMC (Mesh-based Monte Carlo); the meanings
   of the outputs from both software are almost identical, except that
   MCX saves the output on a voxelated grid, and MMC saves on a mesh.

5. How do I read the .jnii, .bnii and .jdt output files?

   Answer: Starting from MCX v2023, MCX saves the volumetric output in the
   JNIfTI format (.jnii) by default; -F bnii saves the binary JNIfTI
   (.bnii) format, which is smaller and faster to parse. Detected photons
   and trajectories are saved in session_detp.jdt and session_traj.jdt
   (JSON), or .jdb (binary JSON) when -F bnii is used; files saved by
   earlier versions use the .jdat suffix. All of these are JSON or binary
   JSON (BJData) files following the [17]NeuroJSON specifications, and can
   be read by

     * MATLAB/Octave: [18]JSONLab (loadjson/loadbj) or the [19]JNIfTI
       toolbox (loadjnifti)
     * Python: [20]PyJData ("pip install jdata", then jdata.load())
     * JavaScript: [21]JSData

   The legacy .mc2 (raw binary float), .mch/.mct (detected photon and
   trajectory) and .nii outputs can still be produced using -F mc2 or -F
   nii, and loaded with loadmc2.m, loadmch.m, mcxloadnii.m or
   mcxloadfile.m in mcx/utils.

6. Why do two runs with the same seed give slightly different results?

   Answer: In the default autopilot mode (-A 1), MCX dispatches photons
   to the GPU threads dynamically for better speed, so the photons
   simulated by each thread depend on the thread scheduling. Runs with
   the same seed agree statistically but not bit-wise. To reproduce a
   simulation exactly, use -A 0 with fixed thread (-t) and block (-T)
   sizes on the same GPU, and the same seed (-E). The replay mode (-E
   file_detp.jdt) always reproduces the detected photons.

7. Does MCX support multiple GPUs in a single computer?

   Answer: Yes, this feature has been supported since [22]MCX v2016.4. Use
   -G with a mask string, such as -G 1101, to select the devices, and -W
   to set the relative workload of each device.

8. Will you consider porting MCX to MPI to run on my cluster?

   Answer: There are simple alternatives, and you can find my arguments on
   this at [23]this link. The support for distributed systems is similar
   to the support for multiple GPUs in the same box. You are recommended
   to use [24]GNU Parallel to manage parallel jobs. Examples can be found
   [25]here. Simulations can also be submitted to [26]MCX Cloud, which
   dispatches the jobs to a cluster of GPU servers.

9. What is the maximum number of media types that MCX can handle?

   Answer: For a label-based volume, the total number of tissue types
   plus the total number of detectors must be less than 4000, limited by
   the size of the constant memory on an NVIDIA GPU. In the autopilot
   mode, label volumes with up to 127 tissue types are stored using 1 byte
   per voxel to save memory and speed up the simulation.

   MCX also supports continuously varying media, where the optical
   properties are specified for every voxel in the volume. Please see the
   -K/--mediabyte flag (for example, asgn_float, muamus_float or
   muamus_half), or the cfg.vol input of mcxlab/pmcx.

10. What is the maximum number of detectors in MCX?

   Answer: The total number of tissue types plus the total number of
   detectors must be less than 4000, limited by the size of the constant
   memory on an NVIDIA GPU. For example, if one uses a volume containing 5
   tissue types, the maximum detector number is 4000-5=3995.

   Alternatively, one can use --bc with 12 letters to detect all photons
   escaping from selected faces of the bounding box, for example, --bc
   ______001000 saves all photons exiting at z=0, without defining any
   detector.

11. My simulation created an empty detected photon file, why is that?

   Answer: This is typically caused by the detector position offset due to
   the incorrectly assumed coordinate system origin.

   In MCX, the default coordinate system is the MATLAB volume index (in
   {x,y,z} float triplet, all starting from 1.0). As a result, the origin
   of the volume (the corner of the diagonal direction of the first voxel)
   is (1,1,1) instead of (0,0,0). If you want to use (0,0,0) as the
   origin, you can do so by adding "--srcfrom0 1" (-z 1) in the command
   line, or setting "Domain":{"OriginType":1} in the JSON input file. The
   following two figures (the bottom face of an 8x8x8 volume) show the
   differences between these two options:

   default or --srcfrom0 0            --srcfrom0 1
   [27]upload:detmask_coordinates.png
   [28]upload:detmask_coordinates_srcfrom0.png

   A photon detection event only happens when a photon escapes from the
   target to the exterior space. This includes two situations:

    1. a photon moving from a non-zero voxel to a 0-voxel
    2. a photon moving beyond the bounding box of the volume

   Thus, in order for a detector to capture an escaped photon, it MUST be
   located on the interface between the zero/non-zero voxels, or on the
   bounding box (within the detector radius). This makes it very sensitive
   to the coordinate origin issue above when the detector radius is 1mm or
   less, because if you mistakenly offset your detector by 1mm, the
   detector will capture nothing, thus giving you an empty detected
   photon file. You can use -M 1 (--dumpmask) to save the detector mask
   and check whether the detectors are located correctly.

   This setting is effective for both source and detector positions.

12. MCX stopped working on my Linux laptop after a suspend, what should I
do?

   Answer: On some Linux laptops with NVIDIA drivers, the nvidia-uvm
   kernel module fails to reload after waking up from suspension, and any
   CUDA program, including MCX, fails with an "unknown error". Please
   close MATLAB or any other program using the GPU, and run

 sudo rmmod nvidia-uvm
 sudo modprobe nvidia-uvm

   after which MCX should be able to run again. On laptops with a hybrid
   Intel/NVIDIA GPU, you may also need to select the NVIDIA GPU as the
   primary GPU ("sudo prime-select nvidia"). Please see the "Requirement
   and Installation" section of README.md for details.

References

   1. https://mcx.space/wiki/index.cgi?action=rss
   2. https://mcx.space/wiki/index.cgi?Doc/FAQ#I_am_getting_a_kernel_launch_timed_out_error_what_is_that
   3. https://mcx.space/wiki/index.cgi?Doc/FAQ#Do_I_need_to_install_CUDA_to_run_MCX
   4. https://mcx.space/wiki/index.cgi?Doc/FAQ#Should_I_use_the_atomic_version_of_MCX
   5. https://mcx.space/wiki/index.cgi?Doc/FAQ#How_do_I_interpret_MCX_s_output_data
   6. https://mcx.space/wiki/index.cgi?Doc/FAQ#How_do_I_read_the_jnii_bnii_and_jdt_output_files
   7. https://mcx.space/wiki/index.cgi?Doc/FAQ#Why_do_two_runs_with_the_same_seed_give_slightly_different_results
   8. https://mcx.space/wiki/index.cgi?Doc/FAQ#Does_MCX_support_multiple_GPUs_in_a_single_computer
   9. https://mcx.space/wiki/index.cgi?Doc/FAQ#Will_you_consider_porting_MCX_to_MPI_to_run_on_my_cluster
  10. https://mcx.space/wiki/index.cgi?Doc/FAQ#What_is_the_maximum_number_of_media_types_that_MCX_can_handle
  11. https://mcx.space/wiki/index.cgi?Doc/FAQ#What_is_the_maximum_number_of_detectors_in_MCX
  12. https://mcx.space/wiki/index.cgi?Doc/FAQ#My_simulation_created_an_empty_history_file_why_is_that
  13. https://mcx.space/wiki/index.cgi?Doc/FAQ#MCX_stopped_working_on_my_Linux_laptop_after_a_suspend
  14. https://github.com/fangq/mcx/tree/master/setup/win64
  15. https://mcx.space/wiki/index.cgi?MCXCL
  16. https://mcx.space/wiki/index.cgi?MMC/Doc/FAQ#How_do_I_interpret_MMC_s_output_data
  17. https://neurojson.org
  18. https://neurojson.org/download/jsonlab
  19. https://neurojson.org/download/jnifty
  20. https://neurojson.org/download/pyjdata
  21. https://neurojson.org/download/jsdata
  22. https://mcx.space/wiki/index.cgi?Doc/ReleaseNotes/v2016.4
  23. https://mcx.space/wiki/index.cgi?MMC/Doc/FAQ#Will_you_consider_porting_MMC_to_MPI_to_run_on_my_cluster
  24. https://www.gnu.org/software/parallel/
  25. https://mcx.space/wiki/index.cgi?MMC/Doc/MMCCluster
  26. https://mcx.space/cloud/
  27. https://mcx.space/wiki/upload/detmask_coordinates.png
  28. https://mcx.space/wiki/upload/detmask_coordinates_srcfrom0.png
