fio i/o errors and exit on windows.

[github-dub]

Hi, I am using log_avg_msec and write_iops_log to measure the duration of network volume outages. This works well when testing nfs v4 volumes on linux, but when testing cifs volumes on windows it fails. On linux, if the volume is inaccessible for 2 minutes or more, fio waits for the volume to come back online, and I can see the duration of the outage in the write_iops_log. On windows, if the volume is unavailable for 10 seconds or more, fio spews i/o error messages and exits. I tried setting max_latency=240s, but that does not help. Is there anything else I can do in the windows fio config, such that it will behave the same way as it does on linux? Thanks.

job file:

[global]
allow_file_create=0
ioengine=windowsaio
thread=1
direct=1
group_reporting
verify=crc32c
do_verify=0
refill_buffers
runtime=600
time_based
buffer_pattern=0xdeadface
max_latency=240s

[vol.0]
directory=c:\t\vol
bs=32k
buffer_compress_percentage=50
iodepth=4
numjobs=16
rw=randrw
rwmixread=50
size=1G
rate=5750k,5750k
log_avg_msec=1000
write_iops_log=c:\t\outage_iops_logs\vol-outage

PS C:\t> fio .\fio-cifs-file-load-vol-test.fio
vol.0: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=windowsaio, iodepth=4
...
fio-3.38
Starting 16 threads
fio: windows error 67 not handledm(16)][64.6%][r=88.0MiB/s,w=88.6MiB/s][r=2814,w=2835 IOPS][eta 03m:33s]
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: windows error 67 not handled
fio: io_u error on file c:\t\vol\vol.0.5.0: Input/output error: write offset=329121792, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.6.0: Input/output error: write offset=625639424, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.6.0: Input/output error: write offset=907182080, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.1.0: Input/output error: write offset=77496320, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.7.0: Input/output error: write offset=24739840, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.3.0: Input/output error: write offset=677773312, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.10.0: Input/output error: write offset=891158528, buflen=32768
fio: windows error 67 not handled
fio: io_u error on file c:\t\vol\vol.0.2.0: Input/output error: write offset=430047232, buflen=32768
fio: windows error 67 not handled
fio: io_u error on file c:\t\vol\vol.0.4.0: Input/output error: write offset=896270336, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.8.0: Input/output error: write offset=827621376, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.8.0: Input/output error: write offset=1001029632, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.0.0: Input/output error: write offset=285605888, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.0.0: Input/output error: write offset=1031667712, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.0.0: Input/output error: write offset=336134144, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.13.0: Input/output error: write offset=533200896, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.9.0: Input/output error: write offset=905019392, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.9.0: Input/output error: write offset=99090432, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.9.0: Input/output error: write offset=390397952, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.11.0: Input/output error: write offset=67436544, buflen=32768
fio: io_u error on file c:\t\vol\vol.0.11.0: Input/output error: write offset=371490816, buflen=32768
fio: windows error 67 not handled
fio: pid=1220, err=5/file:io_u.c:1976, func=io_u error, error=Input/output error
fio: pid=7048, err=5/file:io_u.c:1976, func=io_u error, error=Input/output error
fio: pid=4316, err=5/file:io_u.c:1976, func=io_u error, error=Input/output error
fio: pid=7120, err=5/file:io_u.c:1976, func=io_u error, error=Input/output error

vol.0: (groupid=0, jobs=16): err= 5 (file:io_u.c:1976, func=io_u error, error=Input/output error): pid=7044: Mon Feb 3 22:05:43 2025
read: IOPS=2874, BW=89.8MiB/s (94.2MB/s)(34.0GiB/387912msec)
slat (usec): min=2, max=11532, avg=15.44, stdev=28.86
clat (nsec): min=1171, max=515684k, avg=3438308.19, stdev=4156987.28
lat (usec): min=338, max=515693, avg=3453.75, stdev=4157.42
clat percentiles (usec):
| 1.00th=[ 474], 5.00th=[ 660], 10.00th=[ 840], 20.00th=[ 1237],
| 30.00th=[ 1745], 40.00th=[ 2507], 50.00th=[ 2966], 60.00th=[ 3392],
| 70.00th=[ 3949], 80.00th=[ 4686], 90.00th=[ 5866], 95.00th=[ 7177],
| 99.00th=[17433], 99.50th=[27395], 99.90th=[43779], 99.95th=[49021],
| 99.99th=[65274]
bw ( KiB/s): min=19589, max=211341, per=100.00%, avg=92072.69, stdev=830.98, samples=12404
iops : min= 1421, max= 4715, avg=2872.10, stdev=19.65, samples=6196
write: IOPS=2874, BW=89.8MiB/s (94.2MB/s)(34.0GiB/387912msec); 0 zone resets
slat (usec): min=13, max=10150, avg=31.53, stdev=43.00
clat (usec): min=1745, max=122018, avg=9532.94, stdev=7382.26
lat (msec): min=3, max=122, avg= 9.56, stdev= 7.39
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6],
| 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 8],
| 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 20], 95.00th=[ 26],
| 99.00th=[ 39], 99.50th=[ 45], 99.90th=[ 65], 99.95th=[ 75],
| 99.99th=[ 101]
bw ( KiB/s): min=20822, max=190580, per=100.00%, avg=92071.30, stdev=757.13, samples=12404
iops : min= 1485, max= 4525, avg=2872.09, stdev=18.34, samples=6196
lat (usec) : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.73%, 750=3.00%
lat (usec) : 1000=3.32%
lat (msec) : 2=9.81%, 4=19.52%, 10=50.89%, 20=7.48%, 50=5.07%
lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=13.73%, sys=0.52%, ctx=0, majf=0, minf=0
IO depths : 1=19.8%, 2=36.2%, 4=44.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1114919,1114920,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: bw=89.8MiB/s (94.2MB/s), 89.8MiB/s-89.8MiB/s (94.2MB/s-94.2MB/s), io=34.0GiB (36.5GB), run=387912-387912msec
WRITE: bw=89.8MiB/s (94.2MB/s), 89.8MiB/s-89.8MiB/s (94.2MB/s-94.2MB/s), io=34.0GiB (36.5GB), run=387912-387912msec

[sitsofe]

Hi @github-dub:

On linux, if the volume is inaccessible for 2 minutes or more, fio waits for the volume to come back online, and I can see the duration of the outage in the write_iops_log. On windows, if the volume is unavailable for 10 seconds or more, fio spews i/o error messages and exits. [...] Is there anything else I can do in the windows fio config, such that it will behave the same way as it does on linux?

TLDR: There's no fio configuration change that will help because the behaviour is down to how your filesystem is configured.

With regular ioengines fio is at the mercy of filesystem/block device when it comes to how long a sent I/O can be waited for. If the filesystem returns an error after a given time for a particular I/O, the most that fio can do is ignore the error and carry on (see the ignore_error option).

As for why you see a difference between Windows and Linux: my guess is that on your Linux setup NFS has been configured to wait indefinitely if the NFS server goes away (see https://linux-nfs.org/wiki/index.php/NewNfsManPage#Valid_options_for_either_the_nfs_or_nfs4_file_system_type and note how hard is the default if nothing is set). I'm not a Windows expert but a Google for "windows smb timeout" turned up https://learn.microsoft.com/en-us/powershell/module/smbshare/set-smbclientconfiguration?view=windowsserver2025-ps which mentions that a SessionTimeout parameter can be set. However the MS Q&A SessionTimeout set by Set-SmbClientConfiguration does not work and alawys [sic] less than 20 seconds suggests that the default SMB timeout is 20 seconds (rather than the 10 seconds you reported) and that it can only be lowered below 20 and not raised (alas that Q&A never received a definitive answer from MS).