In order to better understand how jobs are executed on a supercomputer, and if such jobs can be grouped in some way to extract valuable information, we developed a pipeline architecture that combines Big Data and Machine Learning. More specifically, this pipeline processes resource time series (e.g., User CPU or Cached memory) to extract features, which are then used to group the jobs in an unsupervised way through various dimension reduction and clustering techniques. The heavy bulk of this processing is carried out using a Big Data cluster and Spark.

Additionally, jobs are not only grouped, but also those which do not fit in any group are detected and isolated, thus being considered potential anomalies. These anomalies can be useful to users or system administrators.

In this project, a special usage case is studied where the energy of containers is monitored and controlled with the objective of being able to set an energy limit and have it enforced. In this experimental environment, energy is considered just as another accountable resource, alongside CPU or memory. This also implies that it can be shared between different containers, applications or users.

This environment is created with the combination of several tools. For the monitoring of the resources and the processing and handling of the time series, BDWatchdog is used. The CPU scaling to implement the energy limitation and enforcement is carried out by using the Serverless Containers framework. Finally, lying at the core of this study is the energy monitoring, which is provided by the PowerAPI framework developed at the Spirals group of INRIA Lille – Nord Europe. You can check out their tool here.

This framework has been developed to implement serverless computing, that is, to only allocate the resources an application needs at any moment, but for containers. By varying such resources it is possible to optimize them, potentially lowering billing costs, or improving their usage efficiency.

Also interestingly, the user is given a virtual environment that is really close to a virtual machine or a Cloud instance. More on the technical side, this tools continuously changes the resource limits applied to a container, or a group of containers, in real time and in an automatic way without the need for any user intervention. To do so, a microservice architecture has been used, along with monitoring with BDWatchdog.

BDWatchdog is a framework to provide both resource monitoring (CPU, memory, disk and network), as well as profiling information. All of this information is generated, processed and stored in real time by using a time series database (OpenTSDB) for monitoring and a document-based database (mongodb) for profiling. The architecture used is also scalable to support large clusters. The end result of using this framework is two-fold:

• Per-process resource time series detailing how such resources were used during execution, with the possibility of filtering by using tags like host, command, and performing aggregation operations like summatory o average.

• Interactive profiling flame graphs showing the percentage of time spent in a specific Java class, being subsequently subdivided across its calls.

This platform was designed from the start with novel concepts to specifically enhance Big Data workloads in two ways:

• Docker containers instead of virtual machines backed by hypervisors.
• A disk-aware capability implemented on Mesos as the resource scheduler, giving users the option of choosing disks as dedicated resources in the same way as CPUs and memory.