Customizing MCX-CL simulations at run-time

MCX-CL runs on OpenCL framework. A key advantage of OpenCL is the JIT (just-in-time) compilation - that means the simulation source code is not pre-compiled in the binary (because at the compile time, the program does not know what device it will be running on), but dynamically compiled on the device at the run-time (i.e. when it is executed).

As a result of the JIT mechanism, MCX-CL photon simulation kernels (C functions that implement the MC simulation) are simply embedded inside mcxcl binary in the form of source code, stored as a string constant. When someone runs mcxcl on a device, the OpenCL driver of the device, if properly installed, calls the JIT compiler to dynamically compile the kernels to create device assembly and execute the simulation on the device. Usually this JIT compilation takes a fraction of a second to complete. You can typically see the below two lines in the mcxcl command line log, reporting both the JIT compilation flags as well as the build time

 building kernel with option: -cl-mad-enable -DMCX_USE_NATIVE -DMCX_SRC_PENCIL -DMED_TYPE=1  -DUSE_ATOMIC -DMCX_SAVE_DETECTORS  -DINTERNAL_SOURCE -DUSE_NVIDIA_GPU
 build program complete : 530 ms

Because OpenCL can perform run-time JIT compilation, it is possible to obtain optimized binary by considering user-inputs. For example, when a user disables detected photon feature by using -b 0, we can simply remove the -DMCX_SAVE_DETECTORS macro flag among the default build flags (see above log) to allow compilers to completely remove the code blocks related to the boundary handling, substantially reducing registers and other resources, resulting in higher simulation speed.

Another major benefit of OpenCL JIT compilation is that you can modify/re-implement MC simulation kernels without needing to recompile mcxcl.

To do this, you can first use the --showkernel flag to let mcxcl print the full source code of the embedded MC simulation kernels. You can save this source code to a text file using the > redirect syntax

 mcxcl --showkernel > mcx_kernels.cl

With the full source code, you can then open it with a text editor and modify/add/remove the simulations you want to customize. For example, adding a new source type, or changing the scattering phase function etc.

Once you are done with the kernel change, you can then tell mcxcl to use the modified kernel source file instead using the --kernel or -k flag, such as

 mcxcl --kernel mcx_kernels_modified.cl

when the --kernel flag is used, mcxcl will ignore the built-in kernels and JIT-compiles your supplied kernel file instead. If your kernel file contains errors and fails to build by the JIT compiler, warnings or error messages will be printed in the command line so that you can use the message to further edit your kernel code. When it is fully debugged, the mcxcl simulation using your kernel files will return the result based on your customized kernel files.

Please keep in mind that you can not use #include preprocessor in OpenCL kernel file as the source code handler inside mcxcl does not scan such flag and expand the source files (same for most JIT OpenCL compilers in the runtimes).