Can fio with the verify parameter optimize performance in the same way like pi calls ISAL when calculating CRC?

[MorningWeicheng]

Run verify fio command below:
sudo fio --filename=/dev/nvme0n1
--name=read-128k-1job-128qd-verifycrc32-do_verify1-1buffer_compress_percentage
--direct=1
--iodepth=128
--size=100%
--numjobs=1
--ioengine=libaio
--thread=1
--group_reporting
--log_avg_msec=1000
--randrepeat=0
--buffer_compress_percentage=1
--buffer_compress_chunk=4096
--norandommap=1
--percentile_list=10:20:30:40:50:60:70:80:90:99:99.9:99.99:99.999:99.9999:99.99999:99.999999:99.9999999
--verify=crc32
--do_verify=1
--verify_dump=1
--verify_interval=512
--rw=read
--bs=128k > "$read_log" 2>&1
I believe this is mainly because the CRC calculation in fio takes a lot of time, so are there any optimization and acceleration methods available now?

[sitsofe]

Hello @MorningWeicheng,

If you change your checksum to verify=crc32c it will likely be hardware accelerated (you didn't list your platform so it's hard to say for sure but given you mention ISAL that likely means you're on Linux but which architecture?). So I'll bounce this question back: does changing the checksum reduce the overhead to something acceptable? Please can you use a tool like perf to actually quantify the time spent calculating the checksum and let us know before and after comparison?

[vincentkfu]

Fio also has a --crctest option which will show you the speed of the different checksum functions.

$ fio --crctest
md5:		 1012.13 MiB/sec
crc64:		  662.94 MiB/sec
crc32:		  588.96 MiB/sec
crc32c:		13973.80 MiB/sec
crc16:		  664.58 MiB/sec
crc7:		  759.01 MiB/sec
sha1:		 1077.85 MiB/sec
sha256:		  496.02 MiB/sec
sha512:		  755.90 MiB/sec
xxhash:		 4596.30 MiB/sec
murmur3:	 4853.91 MiB/sec
jhash:		 4467.25 MiB/sec
fnv:		10438.75 MiB/sec
sha3-224:	  121.29 MiB/sec
sha3-256:	  114.54 MiB/sec
sha3-384:	   87.10 MiB/sec
sha3-512:	   60.64 MiB/sec

[MorningWeicheng]

Thanks! It seems that using crc32c will result in much better performance on my x86 platform.

[sitsofe]

@MorningWeicheng based on your comment I'm closing this discussion as I don't think there's urgency to wire up ISA-L versions of crc16/crc32*/crc64 .

[MorningWeicheng]

@MorningWeicheng based on your comment I'm closing this discussion as I don't think there's urgency to wire up ISA-L versions of crc16/crc32*/crc64 .

OK, but I want to make sure that: fio currently has no methods to improve performance for the 'crc16/crc32/crc64...' checksum functions.

[sitsofe]

@MorningWeicheng:

OK, but I want to make sure that: fio currently has no methods to improve performance for the 'crc16/crc32/crc64...' checksum functions.

I can't tell if that's a question or a statement :-) As shown by Vincent's table above, those checksums are "slow" and this won't change until someone submits a patch/pull request speeding them up. For crc32 using the crc32c variant seems acceptable if you care about speed, I don't think many people are using crc16 when a 32 bit CRC would be faster and stronger, so I'd say the main performance need would be for crc64 but if a nice, clean well tested patch comes along speeding everything up who am I to say no?

[MorningWeicheng]

Thank you for your reply! It really makes sense that using crc32c actually aligns with my needs when I just need to verify data consistency, so this helps a lot!