Through the collection and association of discrete time-series resource metrics and workloads, we can both provide benchmark and intra-job resource collations, along with system-wide job profiling. Traditional RDBMSes are not designed to store and process long-term discrete time-series metrics and the commonly used resolution-reducing round robin databases (RRDB), make poor long-term sources of data for workload analytics. We implemented a system that employs “Big-data” (Hadoop/HBase) and other analytics (R) techniques and tools to store, process, and characterize HPC workloads. Using this system we have collected and processed over a 30 billion time-series metrics from existing short-term high-resolution (15-sec RRDB) sources, profiling over 200 thousand jobs across a wide spectrum of workloads. The system is currently in use at the University of Kentucky for better understanding of individual jobs and system-wide profiling as well as a strategic source of data for resource allocation and future acquisitions.
Published in: Network Operations and Management Symposium (NOMS), 2016 IEEE/IFIP
Date of Conference: 25-29 April 2016
Date Added to IEEE Xplore: 04 July 2016
Electronic ISSN: 2374-9709
INSPEC Accession Number: 16124063
Bumgardner, VK Cody, Victor W. Marek, and Ray L. Hyatt. “Collating time-series resource data for system-wide job profiling.” Network Operations and Management Symposium (NOMS), 2016 IEEE/IFIP. IEEE, 2016.