Lightweight File System

Efficient I/O is sometimes referred to as the “Achilles' heel” of massively-parallel processing (MPP) computing. While part of the blame can be placed on the inability of the hardware advances for I/O systems to keep pace with advances in CPU, memory, and networks, we believe the real problem is in the I/O system software. Today's parallel file systems are unable to meet the specific needs of many data-intensive MPP applications. Current parallel file systems and I/O libraries limit applications to a standard, predefined set of access interfaces and policies. However, data-intensive applications have a wide variety of needs and often do not perform well using general-purpose solutions. In addition, data-intensive applications show significant performance benefits when using application-specific interfaces that enable advanced parallel-I/O techniques. Examples include collective I/O, prefetching, and data sieving; tailoring prefetching and caching policies to match an application's access patterns, reducing latency and avoiding unnecessary data requests; intelligent application-control of data consistency and synchronization virtually eliminating the need for file locking; and matching data-distribution policies to the application's access patterns in order to optimize parallel access to distributed disks.

The Lightweight File System (LWFS) project is a collaboration between Sandia National Laboratories and the University of New Mexico investigating the applicability of “lightweight” approaches for I/O on MPP systems. Lightweight designs identify the essential functionality needed to meet basic operation requirements. The design of Catamount (the lightweight OS for Sandia's Red Storm machine) focused on the need to support MPI style programs on a space-shared system, i.e., a system in which nodes in the compute partition are allocated to different applications. Because compute nodes are the unit of allocation, the lightweight kernel needs to insure that applications running on different nodes cannot interfere with one another, but does not need to address issues related to competition for resources within a single compute node. Once this essential functionality has been defined and implemented, additional functionality is relegated to the libraries and the application itself. The Compute Node Kernel developed for BlueGene/L follows a similar strategy. The advantages of the lightweight approach are that underlying services do not implement functionality that might degrade the scalability of an application and applications are free to implement the functionality they need in a way that is optimal for the application. The clear disadvantage is that many needed services must be implemented either in libraries or in some cases within an application itself.

While the benefits of the lightweight approach have been demonstrated in the context of operating systems for MPP architectures, this approach has not been applied to the design of other system services. LWFS represents a lightweight approach to I/O in which the core system consists of a small set of critical functionality that the I/O library or file system developer extends to provide custom services, features, and optimizations required by the target applications.

• A Lightweight Approach to File System Development
  • We should change the title - the paper doesn't really talk about how to extend LWFS to build a file system.
  • Address comments from IPDPS reviews.
  • Conference submission Jan/Feb 06

• The Lightweight File System Programming Interface
  • SAND report only
  • Design/API specifications (similar to the Portals document)

• Efficient data-movement protocols for scalable I/O
  • SAND report –> conference paper
  • Describe the motivation, design and implementation of the server-directed approach.
  • Compare to other approaches
  • Lots of performance analysis (1/2 of the paper)
  • We need a Portals implementation that has NALs for Infiniband and/or Myrinet… more than just TCP/IP

• Scalability of LWFS access control protocols
  • SAND report, perhaps a good match for a journal

• Integrating capability-based security in PVFS
  • SAND report –> conference paper
  • Expand Todd's implementation report

• Contrast User-level PVFS (in LWFS) with capability-enhanced PVFS
  • Implement the PVFS API on top of LWFS

• I/O efficient libraries for LWFS
  • Candidates are MPI-I/O, Exodus, SAF (Sets and Fields), HDF5
  • Exodus and SAF are data formats (extensions of netCDF) with I/O libraries
  • Exodus is simple and widely used at Sandia

• Characterizing I/O in Sandia Applications

• POSIX API and semantics in LWFS

• Consistency and coherency for LWFS
  • LWFS currently has no way to enforce consistency
  • Possibly other significant/desirable global properties as well

• Ron's interview notes with application developers on characterizing their I/O requirements (pdf)