Using File System Allocator (FILE)

Umpire supports the use of file based memory allocation. When ENABLE_FILE_RESOURCE is enabled, the environment variables UMPIRE_MEMORY_FILE_DIR can be used to determine where memory can be allocated from:

Variable

Default

Description

UMPIRE_MEMORY_FILE_DIR

./

Directory to create and allocate file based allocations

Requesting the allocation takes two steps: 1) getting a “FILE” allocator, 2) requesting the amount of memory to allocate.

  auto& rm = umpire::ResourceManager::getInstance();
  umpire::Allocator alloc = rm.getAllocator("FILE");

  std::size_t* A = (std::size_t*)alloc.allocate(sizeof(size_t));

To deallocate:

  alloc.deallocate(A);

The complete example is included below:

//////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and Umpire
// project contributors. See the COPYRIGHT file for details.
//
// SPDX-License-Identifier: (MIT)
//////////////////////////////////////////////////////////////////////////////

#include "umpire/Allocator.hpp"
#include "umpire/ResourceManager.hpp"

int main(int, char** argv)
{
  // _sphinx_tag_tut_file_allocate_start
  auto& rm = umpire::ResourceManager::getInstance();
  umpire::Allocator alloc = rm.getAllocator("FILE");

  std::size_t* A = (std::size_t*)alloc.allocate(sizeof(size_t));
  // _sphinx_tag_tut_file_allocate_end

  // _sphinx_tag_tut_file_deallocate_start
  alloc.deallocate(A);
  // _sphinx_tag_tut_file_deallocate_end

  return 0;
}

Using Burst Buffers On Lassen

On Lassen, 1) Download the latest version of Umpire 2) request a private node to build:

$ git clone --recursive https://github.com/LLNL/Umpire.git
$ lalloc 1 -stage storage=64

Note that -stage storage=64 is needed in order to work with the Burst Buffers. 3) Additionally, the environment variable needs to set to $BBPATH :

$ export UMPIRE_MEMORY_FILE_DIR=$BBPATH/

Running File Resource Benchmarks

Continue building Umpire on 1 node, and set the -DENABLE_FILE_RESOURCE=On :

$ mkdir build && cd build
$ lrun -n 1 cmake -DENABLE_FILE_RESOURCE=On -DENABLE_OPENMP=On ../ && make

To run the built-in benchmarks in Umpire from the build run:

$ lrun -n 1 --threads=** ./bin/file_resource_benchmarks ##

** is a placeholder for the amount of threads wanted to run the benchmark on. ## stands for the number of array elements wanted to be passed through the benchmark. This number can range from 1-100,000,000,000.

Results should appear like:

Array Size:   1        Memory Size: 8e-06 MB
Total Arrays: 3               Total Memory Size: 2.4e-05 MB

HOST
  Initialization:      0.0247461 MB/sec
  Initialization Time: 0.000969849 sec
  ---------------------------------------
  Copy:                0.890918 MB/sec
  Copy Time:           1.7959e-05 sec
  ---------------------------------------
  Scale:               0.928074 MB/sec
  Scale Time:          1.724e-05 sec
  ---------------------------------------
  Add:                 1.321 MB/sec
  Add Time:            1.8168e-05 sec
  ---------------------------------------
  Triad:               1.24102 MB/sec
  Triad Time:          1.9339e-05 sec
  ---------------------------------------
  Total Time:          0.00104323 sec

FILE
  Initialization:      0.210659 MB/sec
  Initialization Time: 0.000113928 sec
  ---------------------------------------
  Copy:                0.84091 MB/sec
  Copy Time:           1.9027e-05 sec
  ---------------------------------------
  Scale:               0.938086 MB/sec
  Scale Time:          1.7056e-05 sec
  ---------------------------------------
  Add:                 1.28542 MB/sec
  Add Time:            1.8671e-05 sec
  ---------------------------------------
  Triad:               1.54689 MB/sec
  Triad Time:          1.5515e-05 sec
  ---------------------------------------
  Total Time:          0.000184726 sec

This benchmark run similar to the STREAM Benchmark test and can also run a benchmark for the additional allocators like UM for CUDA and DEVICE for HIP.