VxFS System Administrator's Guide

Performance and Tuning

Chapter 5

Introduction

The following topics are covered in this chapter:

• Choosing a Block Size

• Choosing an Intent Log Size

• Choosing Mount Options
  • log

  • delaylog

  • tmplog

  • nolog

  • nodatainlog

  • blkclear

  • mincache

  • convosync

  • Combining mount Options

• Kernel Tuneables
  • Internal Inode Table Size

  • VxVM Maximum I/O Size

• Monitoring Free Space
  • Monitoring Fragmentation

• I/O Tuning
  • Tuning VxFS I/O Parameters

  • Tuneable VxFS I/O Parameters

Choosing a Block Size

---

Note: The block size is chosen when a file system is created and cannot be changed later.

---

The unit of allocation in VxFS is a block. There are no fragments, since storage is allocated in extents that consist of one or more blocks. For the most efficient space utilization, the smallest block size available on the system should be used. Typically, this provides the best performance as well. The smallest block size available is 1K, which is also the default block size for VxFS file systems created on the system. Unless there are special concerns, there should never be a need to specify a block size when creating file systems.

For large file systems, with relatively few files, the system administrator may wish to experiment with larger block sizes. The trade off of specifying larger block sizes is a decrease in the amount of space used to hold the free extent bitmaps for each allocation unit, an increase in the maximum extent size, and a decrease in the number of extents used per file versus an increase in the amount of space wasted at the end of files that are not a multiple of the block size. Larger block sizes use less disk space in file system overhead, but consume more space for files that are not a multiple of the block size.

Overall file system performance may be improved or degraded by changing the block size. For most applications, it is recommended that the default values for the system be used. However, certain applications may benefit from a larger block size. The easiest way to judge which block sizes will provide the greatest system efficiency is to try representative system loads against various sizes and pick the fastest.

Choosing an Intent Log Size

---

Note: The intent log size is chosen when a file system is created and cannot be subsequently changed.

---

There are several system performance benchmark suites for which VxFS performs better with larger log sizes. The best way to pick the log size is to try representative system loads against various sizes and pick the fastest.

---

Note: When a larger intent log size is chosen, recovery time will be proportionately longer and the file system may consume more system resources (such as memory) during normal operation.

---

Choosing Mount Options

The delaylog, tmplog, and nolog modes are capable of significantly improving performance. The improvement over log mode is typically about 15 to 20 percent with delaylog; with tmplog and nolog, the improvement is even higher. Performance improvement varies, depending on the operations being performed and the workload. Read/write intensive loads should show less improvement, while file system structure intensive loads (such as mkdir, create, rename, etc.) may show over 100 percent improvement. The best way to select a mode is to test representative system loads against the logging modes and compare the performance results.

Most of the modes can be used in combination. For example, a desktop machine might use both the blkclear and mincache=closesync modes.

Additional information on mount options can be found in the mount_vxfs(1M) manual page.

log

delaylog

tmplog

nolog

---

CAUTION! Care should be taken when using the nolog option, as it can cause files to be lost, even if those files were written synchronously. With nolog, the potential loss after a crash is greater than with ufs.

---

nodatainlog

If a disk supports bad block revectoring, then a failure on the data update is unlikely, so logging synchronous writes should be allowed. If the disk does not support bad block revectoring, then a failure is more likely, so the nodatainlog mode should be used.

A nodatainlog mode file system should be approximately 50 percent slower than a standard mode VxFS file system for synchronous writes. Other operations are not affected.

blkclear

mincache

• mincache=closesync

• mincache=direct

• mincache=dsync

• mincache=unbuffered

• mincache=tmpcache

To improve performance, most file systems do not synchronously update data and inode changes to disk. If the system crashes, files that have been updated within the past minute are in danger of losing data. With the mincache=closesync mode, if the system crashes or is switched off, only files that are currently open can lose data. A mincache=closesync mode file system should be approximately 15 percent slower than a standard mode VxFS file system, depending on the workload.

The mincache=direct, mincache=unbuffered, and mincache=dsync modes are used in environments where applications are experiencing reliability problems caused by the kernel buffering of I/O and delayed flushing of non-synchronous I/O. The mincache=direct and mincache=unbuffered modes guarantee that all non-synchronous I/O requests to files will be handled as if the VX_DIRECT or VX_UNBUFFERED caching advisories had been specified. The mincache=dsync mode guarantees that all non-synchronous I/O requests to files will be handled as if the VX_DSYNC caching advisory had been specified. Refer to the vxfsio(7) manual page for explanations of VX_DIRECT, VX_UNBUFFERED, and VX_DSYNC. The mincache=direct, mincache=unbuffered, and mincache=dsync modes also flush file data on close as mincache=closesync does.

Since the mincache=direct, mincache=unbuffered, and mincache=dsync modes change non-synchronous I/O to synchronous I/O, there can be a substantial degradation in throughput for small to medium size files for most applications. Since the VX_DIRECT and VX_UNBUFFERED advisories do not allow any caching of data, applications that would normally benefit from caching for reads will usually experience less degradation with the mincache=dsync mode. mincache=direct and mincache=unbuffered require significantly less CPU time than buffered I/O.

If performance is more important than data integrity, the mincache=tmpcache mode may be used. The mincache=tmpcache mode disables special delayed extending write handling, trading off less integrity for better performance. Unlike the other mincache modes, tmpcache does not flush the file to disk when it is closed. When this option is used, garbage may appear in a file that was being extended when a crash occurred.

convosync

---

Note: Use of the convosync=dsync option violates POSIX guarantees for synchronous I/O.

---

The convosync=closesync mode converts synchronous and data synchronous writes to non-synchronous writes and flushes the changes to the file to disk when the file is closed.

The convosync=delay mode causes synchronous and data synchronous writes to be delayed rather than to take effect immediately. No special action is performed when closing a file. This option effectively cancels any data integrity guarantees normally provided by opening a file with O_SYNC. See the open(2), fcntl(2), and vxfsio(7) manual pages for more information on O_SYNC.

---

Note: Extreme care should be taken when using the convosync=closesync or convosync=delay mode, as they actually change synchronous I/O into non-synchronous I/O. This may cause applications that use synchronous I/O for data reliability to fail if the system crashes and synchronously written data is lost.

---

The convosync=dsync mode converts synchronous writes to data synchronous writes.

As with closesync, the direct, unbuffered, and dsync modes flush changes to the file to disk when it is closed. These modes can be used to speed up applications that use synchronous I/O. Many applications that are concerned with data integrity specify the O_SYNC fcntl in order to write the file data synchronously. However, this has the undesirable side effect of updating inode times and therefore slowing down performance. The convosync=dsync, convosync=unbuffered, and convosync=direct modes alleviate this problem by allowing applications to take advantage of synchronous writes without modifying inode times as well.

---

Note: Before using convosync=dsync, convosync=unbuffered, or convosync=direct, make sure that all applications that use the file system do not require synchronous inode time updates for O_SYNC writes.

---

Combining mount Options

	mount -F vxfs -o nolog,blkclear

The following examples provide some common and reasonable mount option combinations.

Example 1 - Desktop File System

	mount -F vxfs -o log,mincache=closesync /dev/dsk/c1b0t3d0s1 /mnt

Example 2 - Temporary File System or Restoring from Backup

	mount -F vxfs -o tmplog,convosync=delay,mincache=tmpcache \
	/dev/dsk/c1b0t3d0s1 /mnt

Example 3 - Data Synchronous Writes

	mount -F vxfs -o log,convosync=dsync /dev/dsk/c1b0t3d0s1 /mnt

Kernel Tuneables

The kernel tuneables are found in files located in /etc/conf/mtune.d. The mtune.d files contain the system default, minimum, and maximum values for each tuneable. The /etc/conf/cf.d/stune file contains entries for tuneables that cannot use the default value in the mtune.d file. The stune file is used to change tuneable values to something other than the default, when necessary.

Internal Inode Table Size

The vxfs file system type (FSType) uses the tuneable VXFSNINODE as the maximum number of entries in the vxfs inode table. The actual size of the inode table is dynamically adjusted as the system activity changes. The VXFSNINODE parameter is the upper bound on the size of the inode table.

The value of VXFSNINODE is determined automatically at boot time, based on the amount of memory in the system. The value for VXFSNINODE should generally be left alone and not altered. The current size of the table and number of inodes in use can be monitored using sar with the -t option.

If the VXFSNINODE value is too small, the system may run out of inode-table entries, and system calls may fail with ENFILE. If the value is too large, excessive memory may be consumed by inode-table entries, which would adversely affect system performance. Before overriding the automatically determined value by specifying a positive value for this tuneable, the system administrator should verify that the problem prompting this change is not due to an application that forgets to close files.

VxVM Maximum I/O Size

Monitoring Free Space

Monitoring Fragmentation

The easiest way to ensure that fragmentation does not become a problem is to schedule regular defragmentation runs from cron.

Defragmentation scheduling should range from weekly (for frequently used file systems) to monthly (for infrequently used file systems). Extent fragmentation should be monitored with fsadm or the -o s option of df. There are three factors which can be used to determine the degree of fragmentation:

• percentage of free space in extents of less than eight blocks in length

• percentage of free space in extents of less than 64 blocks in length

• percentage of free space in extents of length 64 blocks or greater

• less than 1 percent of free space in extents of less than eight blocks in length

• less than 5 percent of free space in extents of less than 64 blocks in length

• more than 5 percent of the total file system size available as free extents in lengths of 64 or more blocks

• greater than 5 percent of free space in extents of less than 8 blocks in length

• more than 50 percent of free space in extents of less than 64 blocks in length

• less than 5 percent of the total file system size available as free extents in lengths of 64 or more blocks

The "before" result is the degree of fragmentation prior to the reorganization. If the degree of fragmentation is approaching the figures for bad fragmentation, then the interval between fsadm runs should be reduced. If the degree of fragmentation is low, the interval between fsadm runs can be increased.

The "after" result is an indication of how well the reorganizer is performing. If the degree of fragmentation is not close to the characteristics of an unfragmented file system, then the extent reorganizer is not functioning properly. The file system may be a candidate for expansion. (Full file systems tend to fragment and are difficult to defragment.) It is also possible that the reorganization is not being performed at a time during which the file system in question is relatively idle.

Directory reorganization is not nearly as critical as extent reorganization, but regular directory reorganization will improve performance. It is advisable to schedule directory reorganization for file systems when the extent reorganization is scheduled. The following is a sample script that is run periodically at 3:00 A.M. from cron for a number of file systems:

outfile=/usr/spool/fsadm/out.'/bin/date +'%m%d''
for i in /home /home2 /project /db
do
	/bin/echo "Reorganizing $i"
	/bin/timex /etc/fs/vxfs/fsadm -e -E -s $i
	/bin/timex /etc/fs/vxfs/fsadm -s -d -D $i
done > $outfile 2>&1 

I/O Tuning

---

Note: The tuneables and the techniques described in this section are for tuning on a per file system basis and should be used judiciously based on the underlying device properties and characteristics of the applications that use the file system.

---

Tuning VxFS I/O Parameters

If the VxFS file system is being used with the VERITAS Volume Manager, the file system queries the Volume Manager to find out the geometry of the underlying volume and automatically sets the I/O parameters. The Volume Manager is queried by mkfs when the file system is created to automatically align the file system to the volume geometry. Then the mount command queries the Volume Manager when the file system is mounted and downloads the I/O parameters.

If the default parameters are not acceptable or the file system is being used without the Volume Manager, then the /etc/vx/tunefstab file can be used to set values for I/O parameters. The mount command reads the
/etc/vx/tunefstab file and downloads any parameters specified for a file system. The tunefstab file overrides any values obtained from the Volume Manager. While the file system is mounted, any I/O parameters can be changed using the vxtunefs command which can have tuneables specified on the command line or can read them from the /etc/vx/tunefstab file. For more details, see the vxtunefs(1M) and tunefstab(4) manual pages. The vxtunefs command can be used to print the current values of the I/O parameters.

If the default alignment from mkfs is not acceptable, the -o align=n option can be used to override alignment information obtained from the Volume Manager.

Tuneable VxFS I/O Parameters

read_pref_io
The preferred read request size. The file system uses this in conjunction with the read_nstream value to determine how much data to read ahead. The default value is 64K.

read_nstream
The desired number of parallel read requests of size read_pref_io to have outstanding at one time. The file system uses the product of read_nstream multiplied by read_pref_io to determine its read ahead size. The default value for read_nstream is 1.

read_unit_io
This is a less preferred request size. Currently, the file system does not use this tuneable.

write_pref_io
The preferred write request size. The file system uses this in conjunction with the write_nstream value to determine how to do flush behind on writes. The default value is 64K.

write_nstream
The desired number of parallel write requests of size write_pref_io to have outstanding at one time. The file system uses the product of write_nstream multiplied by write_pref_io to determine when to do flush behind on writes. The default value for write_nstream is 1.

write_unit_io
This is a less preferred request size. Currently, the file system does not use this tuneable.

pref_strength
Indicates to the file system how large a performance gain might be made by adhering to the preferred I/O sizes. The file system does not use this tuneable.

buf_breakup_size
Tells the file system how large an I/O it can issue without a driver breaking up the request. The file system does not use this tuneable.

max_direct_iosz
The maximum size of a direct I/O request that will be issued by the file system. If a larger I/O request comes in, then it is broken up into max_direct_iosz chunks. This parameter defines how much memory an I/O request can lock at once, so it should not be set to more than 20 percent of memory.

discovered_direct_iosz
Any file I/O requests larger than the discovered_direct_iosz are handled as discovered direct I/O. A discovered direct I/O is unbuffered similar to direct I/O, but it does not require a synchronous commit of the inode when the file is extended or blocks are allocated. For larger I/O requests, the CPU time for copying the data into the page cache and the cost of using memory to buffer the I/O data becomes more expensive than the cost of doing the disk I/O. For these I/O requests, using discovered direct I/O is more efficient than regular I/O. The default value of this parameter is 256K.

default_indir_size
On VxFS, files can have up to 10 direct extents of variable size stored in the inode. Once these extents are used up, the file must use indirect extents which are a fixed size that is set when the file first uses indirect extents. These indirect extents are 8K by default. The file system does not use larger indirect extents because it must fail a write and return ENOSPC if there are no extents available that are the indirect extent size. For file systems with a lot of large files, the 8K indirect extent size is too small. The files that get into indirect extents use a lot of smaller extents instead of a few larger ones. By using this parameter, the default indirect extent size can be increased so large that files in indirects use fewer larger extents.

The tuneable default_indir_size should be used carefully. If it is set too large, then writes will fail when they are unable to allocate extents of the indirect extent size to a file. In

max_diskq
Limits the maximum disk queue generated by a single file. When the file system is flushing data for a file and the number of pages being flushed exceeds max_diskq, processes will block until the amount of data being flushed decreases. Although this doesn't limit the actual disk queue, it prevents flushing processes from making the system unresponsive. The default value is 1 MB.

max_extent_size
Increases or decreases the maximum size of an extent. When the file system is following its default allocation policy for sequential writes to a file, it allocates an initial extent which is large enough for the first write to the file. When additional extents are allocated, they are progressively larger (the algorithm tries to double the size of the file with each new extent) so each extent can hold several writes worth of data. This is done to reduce the total number of extents in anticipation of continued sequential writes. When the file stops being written, any unused space is freed for other files to use.

Normally this allocation stops increasing the size of extents at 2048 blocks which prevents one file from holding too much unused space. max_extent_size is measured in file system blocks.

def_init_extent
Changes the default initial extent size. VxFS determines, based on the first write to a new file, the size of the first extent to be allocated to the file. Normally the first extent is the smallest power of 2 that is larger than the size of the first write. If that power of 2 is less than 8K, the first extent allocated is 8K. After the initial extent, the file system increases the size of subsequent extents (see max_extent_size) with each allocation.

Since most applications write to files using a buffer size of 8K or less, the increasing extents start doubling from a small initial extent. def_init_extent can change the default initial extent size to be larger, so the doubling policy will start from a much larger initial size and the file system will not allocate a set of small extents at the start of file.

This parameter should only be used on file systems that will have a very large average file size. On these file systems it will r

If the file system is being used with a hardware disk array or volume manager other than VxVM, try to align the parameters to match the geometry of the logical disk. With striping or RAID-5, it is common to set read_pref_io to the stripe unit size and read_nstream to the number of columns in the stripe. For striping arrays, use the same values for write_pref_io and write_nstream, but for RAID-5 arrays, set write_pref_io to the full stripe size and write_nstream to 1.

For an application to do efficient disk I/O, it should issue read requests that are equal to the product of read_nstream multiplied by read_pref_io. Generally, any multiple or factor of read_nstream multiplied by read_pref_io should be a good size for performance. For writing, the same rule of thumb applies to the write_pref_io and write_nstream parameters. When tuning a file system, the best thing to do is try out the tuning parameters under a real life workload.

If an application is doing sequential I/O to large files, it should try to issue requests larger than the discovered_direct_iosz. This causes the I/O requests to be performed as discovered direct I/O requests, which are unbuffered like direct I/O but do not require synchronous inode updates when extending the file. If the file is larger than can fit in the cache, then using unbuffered I/O avoids throwing useful data out of the cache and it avoids a lot of CPU overhead.