Data Management Practices on Large-Scale Lustre Scratch File Systems

Managing large-scale Lustre scratch file systems is a necessity on shared storage resources. The lack of proper data management can halt all computation that is contingent upon any type of output to disk. Lustre scratch areas are typically provided for users to utilize as high-performance temporary space to stage job input data and store job output data. Common techniques for managing the amount of stored temporary data involve the use of file system quotas or the enforcement of a purge policy that limits how long files can reside in the scratch space. This paper reviews the challenges of balancing usability from the end user's perspective and the administration of these scratch areas, along with the use of tools to decrease the overall amount of time required for file purges on large-scale Lustre systems.