Thursday, February 25, 2010

ZVOLs - Write Cache

When you create a ZFS volume its write cache is disabled by default meaning that all writes to the volume will be synchronous. Sometimes it might be handy though to be able to enable a write cache for a particular zvol. I wrote a small C program which allows you to check if WC is enabled or not. It also allows you to enable or disable write cache for a specified zvol.

First lets check if write cache is disabled for a zvol rpool/iscsi/vol1

milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1
Write Cache: disabled

Now lets issue 1000 writes

milek@r600:~/progs# ptime ./sync_file_create_loop /dev/zvol/rdsk/rpool/iscsi/vol1 1000

real 12.013566363
user 0.003144874
sys 0.104826470

So it took 12s and I also confirmed that writes were actually being issued to a disk drive. Lets enable write cache now and repeat 1000 writes

milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1 1
milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1
Write Cache: enabled

milek@r600:~/progs# ptime ./sync_file_create_loop /dev/zvol/rdsk/rpool/iscsi/vol1 1000

real 0.239360231
user 0.000949655
sys 0.019019552

Worked fine.

The zvol_wce program is not idiot-proof and it doesn't check if operation succeeded or not. You should be able to compile it by issuing: gcc -o zvol_wce zwol_wce.c

milek@r600:~/progs# cat zvol_wce.c

/* Robert Milkowski
http://milek.blogspot.com
*/

#include <unistd.h>
#include <stropts.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stropts.h>
#include <sys/dkio.h>


int main(int argc, char **argv)
{
char *path;
int wce = 0;
int rc;
int fd;

path = argv[1];

if ((fd = open(path, O_RDONLY|O_LARGEFILE)) == -1)
exit(2);

if (argc>2) {
wce = atoi(argv[2]) ? 1 : 0;
rc = ioctl(fd, DKIOCSETWCE, &wce);
}
else {
rc = ioctl(fd, DKIOCGETWCE, &wce);
printf("Write Cache: %s\n", wce ? "enabled" : "disabled");
}

close(fd);
exit(0);
}

6 comments:

jamesd_wi said...

Looks neat, have you tested it with iSCSI being exported to Solaris and/or Linux?

Joshua M. Clulow said...

Would enabling this for swap zvols produce increased swap write performance?

Richard Elling said...

DKIOCSETWCE was added to Zvols in b113 via
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6794730

prior to this time, the DKIOCSETWCE ioctl was not implemented.

For the iSCSI case, the client is responsible for setting WCE (or not setting WCD, as the case may be)

James Mansion said...

If a zvol is write-cache-enabled, can the client see that and control it with cache flush commands (as it should with a physical disk that offers write cache)?

milek said...

James - there is a DKIOGETWCE, so yes an application can detect if Write Cache is enabled or not as my C program does :)

Then if an application sends DKIOCFLUSHWRITECACHE it will force all outstanding transactions to be synced.

However please notce that COMSTAR iSCSI provides a nice way to enable/disable Write Cache so there is no need to use my program with it.

Unknown said...

Since I still see this blog post coming up from time to time, I just wanted to stop in quickly and say:

If you enable the writeback cache on a LU (and, depending on what Solaris derivative you're running it may very well be enabled by DEFAULT), and you do not have a very, very detailed and complete understanding of your data path from start to finish, you are setting yourself up for data loss.

You should NEVER, EVER enable the writeback cache on a zvol unless you are 110% sure you know what you're doing, or are 110% sure you're OK with losing all your data.