Get NFS and iSCSI latency

I am using dd and ioping to determine read/write latency on NFS and iSCSI.
This is a test lab so the latencies will be very low compared to production environments.

For the purpose of this test I will use 2 Arch Linux VMs. One will act as an NFS and iSCSI server, the other one will be the client. server1 and client1.
dd will help me find out the read / write speed on the disk / nfs export from the client. This is useful because the read / write actually takes place on the remote server, over TCP.
ioping will show me latency. Latency is different from read/write speed, as it is the read or write response time over the network.

1. iSCSI 

I mounted a LUN from server1 to client1 in /media/lun

[root@client1 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdl         50G   33M   50G   1% /media/lun
[root@client1 ~]# 

1.1 Use dd to perform a test write and read.

[root@client1 ~]# time dd if=/dev/zero of=/media/lun/testfile bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 2.01766 s, 133 MB/s

real    0m2.054s
user    0m0.007s
sys     0m1.677s
[root@client1 ~]#

Write performance in this case is 133 MB/s.

[root@client1 ~]# time dd if=/media/lun/testfile of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.0614949 s, 4.4 GB/s

real    0m0.064s
user    0m0.007s
sys     0m0.057s
[root@client1 ~]# 

Read performance 4.4 GB/s.

1.2 Use ioping to test disk latency.

Measure disk seek rate (iops, avg)

[root@client1 ~]# ioping -R /dev/sdl

--- /dev/sdl (device 50.0 GiB) ioping statistics ---
4.9 k requests completed in 3.0 s, 1.6 k iops, 6.4 MiB/s
min/avg/max/mdev = 347 us / 609 us / 10.6 ms / 733 us
[root@client1 ~]# 

Measure disk sequential speed (MiB/s)

[root@client1 ~]# ioping -R /dev/sdl

--- /dev/sdl (device 50.0 GiB) ioping statistics ---
4.9 k requests completed in 3.0 s, 1.6 k iops, 6.4 MiB/s
min/avg/max/mdev = 347 us / 609 us / 10.6 ms / 733 us
[root@client1 ~]# ioping -RL /dev/sda

--- /dev/sda (device 8.0 GiB) ioping statistics ---
918 requests completed in 3.0 s, 306 iops, 76.6 MiB/s
min/avg/max/mdev = 1.0 ms / 3.3 ms / 74.6 ms / 4.6 ms
[root@client1 ~]# 

Measure read latency:

[root@client1 ~]# ioping /dev/sdl
4.0 KiB from /dev/sdl (device 50.0 GiB): request=1 time=3.3 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=2 time=2.3 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=3 time=2.0 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=4 time=2.5 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=5 time=2.1 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=6 time=2.9 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=7 time=2.6 ms
^C
--- /dev/sdl (device 50.0 GiB) ioping statistics ---
7 requests completed in 6.7 s, 395 iops, 1.5 MiB/s
min/avg/max/mdev = 2.0 ms / 2.5 ms / 3.3 ms / 414 us

In troubleshooting cases use one of the options
      -s <size>       request size (4k)
      -S <wsize>      working set size (1m)
to get values closer to the real life workload.

2. NFS

Mounted an NFS export from server1 into /mnt

[root@client1 ~]# df -h
Filesystem                           Size  Used Avail Use% Mounted on
172.16.41.132:/san/vol1  2.0G  256K  2.0G   1% /mnt
[root@client1 ~]#

2.1 Test using dd. (same as for iSCSI)

[root@client1 ~]# time dd if=/dev/zero of=/mnt/testfile bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 4.73394 s, 56.7 MB/s

real    0m4.751s
user    0m0.000s
sys     0m0.993s
[root@client1 ~]#


As expected, I am getting about half the write speed as compared to iSCSI performance.

The same is valid for read performance:

[root@client1 ~]# time dd if=/mnt/testfile of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.109984 s, 2.4 GB/s

real    0m0.131s
user    0m0.007s
sys     0m0.087s
[root@client1 ~]#

2.2 Using ioping to test NFS performance:

Use the same commands, except replace /dev/sdl with /mnt (mount point). Ioping will recognize network filesystems.

[root@client1 ~]# ioping -RL /mnt

--- /mnt (nfs4 172.16.41.132:/san/vol1) ioping statistics ---
512 requests completed in 3.0 s, 171 iops, 42.8 MiB/s
min/avg/max/mdev = 3.2 ms / 5.8 ms / 15.5 ms / 2.0 ms
[root@client1 ~]#

[root@client1 ~]# ioping -RL -s 512 /mnt

--- /mnt (nfs4 172.16.41.132:/san/vol1) ioping statistics ---
22.2 k requests completed in 3.0 s, 7.6 k iops, 3.7 MiB/s
min/avg/max/mdev = 0 us / 132 us / 111.0 ms / 1.9 ms
[root@client1 ~]#

Ioping options:


[root@client1 ~]# ioping
 Usage: ioping [-LABCDWRq] [-c count] [-w deadline] [-pP period] [-i interval]
               [-s size] [-S wsize] [-o offset] directory|file|device
        ioping -h | -v

      -c <count>      stop after <count> requests
      -w <deadline>   stop after <deadline>
      -p <period>     print raw statistics for every <period> requests
      -P <period>     print raw statistics for every <period> in time
      -i <interval>   interval between requests (1s)
      -s <size>       request size (4k)
      -S <wsize>      working set size (1m)
      -o <offset>     working set offset (0)
      -L              use sequential operations (includes -s 256k)
      -A              use asynchronous I/O
      -C              use cached I/O
      -D              use direct I/O
      -W              use write I/O *DANGEROUS*
      -R              seek rate test (same as -q -i 0 -w 3 -S 64m)
      -B              print final statistics in raw format
      -q              suppress human-readable output
      -h              display this message and exit
      -v              display version and exit

[root@client1 ~]#