## Design Decisions - Workloads are written completely in Python rather than kernels being written in C++ or a DSL with Python used just for orchestration and configuration. Supporting workloads with kernels written in C++ is a possible extension. - The C++ is generated through translation of the AST which means that no Python code within work units is evaluated. - Data is passed back and forth between the Python code and the Kokkos environment using "views" which are Kokkos' abstraction of multidimensional arrays. - For now at least, scalar data types are neither explicitly annotated nor inferred, everything is just cast to doubles with the exception of array indices. ## Components ### KokkosWorkload class This is a base class that users must inherit from in their "workload" class. The reason to have them define all Kokkos related code in a class rather than wherever they want with top level declarations is that it makes it far simpler to translate. For example when searching for functions or data that need to be transfered to C++, the class's member variables can be searched rather than having to look through the entire python environment. It also avoids issues with defining multiple Kokkos workloads in the same python module because their translation and execution are handled separately. The one drawback is that if a user has many workloads that all use the same work units, they have to redefine them for each class. ### View1D/2D/3D classes All data that's used by the Kokkos computations is stored in views, which are an abstraction of multidimensional arrays. These are allocated and initialized in python as members of the workload class and then serialized. Additional C++ code is generated to load them from disk and store them back when the computation is finished. ### work_unit annotation Any methods of the workload with this annotation are treated as kernels to be translated to C++ lambdas. They either take an index (for `parallel_for`) or an index and accumulator (for `parallel_reduce`) as arguments. The set of python language constructs that can be translated is still growing, but the essential ones like arithmetic, view access, and flow control are already working. In addition it's possible to use data members of the python workload within work units (read only), where they'll get translated to constants. The main features currently missing are local variable declaration (which is tricky due to scoping and typing) and function invocation, which will be handled by maintaining a whitelist of functions that are legal to call from within work units. ### run_work_units This is a special method the user must define in their workload class. It dictates how the work units are actually executed. Like the work units themselves the method is translated to C++, but currently only a small subset of language constructs are allowed. The only statements that the user can currently write a are function invocations of `parallel_for` and `parallel_reduce`, and assignments to store the result of said reductions. This limitation is primarily used to make translation simpler, but if useful it would be possible to allow the use of flow control or other constructs. ### use_results This is another special method the user can define. It is run after `run_work_units` has completed and the updated data from C++ has been copied back into python. Even if `run_work_units` executes many work units, `use_results` is only invoked once at the very end to avoid continuously having to copy large views back and forth between python and C++. This limits the ability to write workloads that alter their execution based on the result of previous steps such as fixed point numerical algorithms, so we're still considering alternate approaches. ### execute This is a method from the base `KokkosWorkload` class, calling it triggers the translation and execution of a workload. It will accept an argument to determine which "Execution Space" (CUDA/OpenMP/Python/PThreads/Serial C++) to use. This functionality is currently not implemented, but the Execution Space will change the generated C++ code to handle specific issues with each one such as copying data over to the GPU. See the next section for an in depth example of what steps this method performs. ## Execution example Here is a simple example of a Kokkos workload which adds 1 to every element of an array: ```python class AddOne(KokkosWorkload): def __init__(self, n): self.N = n self.a = View1D(n) for i in range(self.N): self.a[i] = 2 print("Initialized view:", ) def run_work_units(self): parallel_for(self.N, self.add1) def use_results(self): print("Modified view:", self.a) @work_unit def add1(self, i): self.a[i] += 1 my_workload = AddOne(1000) my_workload.execute() ``` When `my_workload = AddOne(1000)` is run, the workload's constructor executes which initializes its view. Then when `my_workload.execute()` is run the following steps happen in order: 1. A build directory is created and essential files like protobufs and the Makefile for the C++ code are copied in. 2. The members of `AddOne` are inspected and the view `self.a` is serialized into the build directory. 3. Boilerplate C++ is generated to initialize the Kokkos runtime and load in the view that has been serialized. 4. The workload `add1` is detected because of its `@work_unit` decorator and translated to a C++ lambda. 5. `run_work_units` is translated to C++ (this is a simple case because no reductions occur). 6. More C++ is generated to write the modified views back out to disk and shut down the Kokkos runtime. 7. The finished C++ file is compiled and run. Any errors are passed back to the user. 8. If the C++ code compiled and executed successfully, the views are read back into python. 9. `use_results` is run and the updated view gets printed.