| Craig Ulmer |
SPARC Burst Buffer WorkIn the Fall we did an initial investigation into how Sandia's new SPARC application could leverage the Cray XC40's DataWarp burst buffers to improve I/O performance. As part of this effort we looked at four options: simply mapping existing IO to the burst buffer, writing checkpoints out using LANL's Hierarchical I/O (HIO) library, using DataElevator to route our HDF5 calls to the burst buffers, and writing checkpoints out via Kelpie. We presented what we'd learned to the SPARC lead and wrote up the lessons learned in this ECP report. The main take away for us was that the easiest thing to do for our workflows was just use the DataWarp directives in slurm and write files out directly. HIO was more complicated than we needed it to be. DataElevator was not compatible with the per-rank file I/O that Sandia uses. We did write a checkpoint stub for SPARC that wrote results out to Kelpie and demo'd it to the lead. However, we warned that since the cp/rs data wasn't useful for downstream analytics, Kelpie would be overkill in most cases. SPARC was updated to have multiple I/O backends, but I don't think anyone used anything but the plain posix io version. AbstractRecent high-performance computing (HPC) platforms such as the Trinity Advanced Technology System (ATS-1) feature burst buffer resources that can have a dramatic impact on an application's I/O performance. While these non-volatile memory (NVM) resources provide a new tier in the storage hierarchy, developers must find the right way to incorporate the technology into their applications in order to reap the benefits. Similar to other laboratories, Sandia is actively investigating ways in which these resources can be incorporated into our existing libraries and workflows without burdening our application developers with excessive, platform-specific details. This FY18Q1 milestone summaries our progress in adapting the Sandia Parallel Aerodynamics and Reentry Code (SPARC) in Sandia's ATDM program to leverage Trinity's burst buffers for checkpoint/restart operations. We investigated four different approaches with varying tradeoffs in this work: (1) simply updating job script to use stage-in/stage out burst buffer directives, (2) modifying SPARC to use LANL's hierarchical I/O (HIO) library to store/retrieve checkpoints, (3) updating Sandia's IOSS library to incorporate the burst buffer in all meshing I/O operations, and (4) modifying SPARC to use our Kelpie distributed memory library to store/retrieve checkpoints. Team members were successful in generating initial implementation for all four approaches, but were unable to obtain performance numbers in time for this report (reasons: initial problem sizes were not large enough to stress I/O, and SPARC refactor will require changes to our code). When we presented our work to the SPARC team, they expressed the most interest in the second and third approaches. The HIO work was favored because it is lightweight, unobtrusive, and should be portable to ATS-2. The IOSS work is seen as a long-term solution, and is favored because all I/O work (including checkpoints) can be deferred to a single library. Publications
|