Customizing MCX-CL simulations at run-time

MCX-CL runs on the OpenCL framework. A key advantage of OpenCL is JIT (just-in-time) compilation - this means the simulation source code is not pre-compiled in the binary (because at compile time, the program does not know what device it will be running on), but is dynamically compiled on the device at run-time (i.e., when it is executed).

As a result of the JIT mechanism, MCX-CL photon simulation kernels (C functions that implement the MC simulation) are simply embedded inside the mcxcl binary in the form of source code, stored as a string constant. When someone runs mcxcl on a device, the OpenCL driver of the device, if properly installed, calls the JIT compiler to dynamically compile the kernels to create device assembly and execute the simulation on the device. Usually this JIT compilation takes a fraction of a second to complete. You can typically see the below two lines in the mcxcl command line log, reporting both the JIT compilation flags as well as the build time:

 building kernel with option: -cl-mad-enable -DMCX_USE_NATIVE -DMCX_SRC_PENCIL -DMED_TYPE=1  -DUSE_ATOMIC -DMCX_SAVE_DETECTORS  -DINTERNAL_SOURCE -DUSE_NVIDIA_GPU
 build program complete : 530 ms

Because OpenCL can perform run-time JIT compilation, it is possible to obtain optimized binaries by considering user inputs. For example, when a user disables the detected photon feature by using -b 0, we can simply remove the -DMCX_SAVE_DETECTORS macro flag from the default build flags (see above log) to allow compilers to completely remove the code blocks related to boundary handling, substantially reducing registers and other resources, resulting in higher simulation speed.

Another major benefit of OpenCL JIT compilation is that you can modify/reimplement MC simulation kernels without needing to recompile mcxcl.

To do this, you can first use the --showkernel flag to let mcxcl print the full source code of the embedded MC simulation kernels. You can save this source code to a text file using the > redirect syntax:

 mcxcl --showkernel > mcx_kernels.cl

With the full source code, you can then open it with a text editor and modify/add/remove the simulations you want to customize. For example, you can add a new source type or change the scattering phase function, etc.

Once you are done with the kernel changes, you can then tell mcxcl to use the modified kernel source file instead by using the --kernel or -k flag, such as:

 mcxcl --kernel mcx_kernels_modified.cl

When the --kernel flag is used, mcxcl will ignore the built-in kernels and JIT-compile your supplied kernel file instead. If your kernel file contains errors and fails to build by the JIT compiler, warnings or error messages will be printed on the command line so that you can use the messages to further edit your kernel code. When it is fully debugged, the mcxcl simulation using your kernel file will return the result based on your customized kernel.

Please keep in mind that you cannot use #include preprocessor directives in OpenCL kernel files as the source code handler inside mcxcl does not scan for such flags and expand the source files (same for most JIT OpenCL compilers in the runtimes).