Consistency check of Helicity Decoder Board data failed!

[aaust]

Processing the data from the currently ongoing run period 2025-01, I see many error messages of this kind:

14:07:56.128 [error] Consistency check of Helicity Decoder Board data failed!

It seems like it happens for a certain period of time, then it stops, and resumes a little bit later. Since it is declared as an error, I wonder if there is something wrong with the data.
Here is an exaple to see it:

hd_root -PPLUGINS=occupancy_online,highlevel_online -PNTHREADS=16 /cache/halld/RunPeriod-2025-01/rawdata/Run133255/hd_rawdata_133255_000.evio

[sdobbs]

I added PR #1089 that adds some more useful debugging. It looks sometimes we are off by a bit

10:29:22.647 [error] Consistency check of Helicity Decoder Board data failed!
10:29:22.647 [error]   Bad word: 0x8abd5179
10:29:22.647 [error]      Compare 0x180179321 to 180179320

Someone closer to the analysis should double check that the consistency check is being done correctly, and maybe that points into a firmware issue that should be tracked down.

We can look at how often this shows up more broadly, but at least for this file, it seems like just an occasional problem.

[sdobbs]

suggested next steps via Mark Dalton:

I would suggest also checking the value of "helicity_seed" for these events (particularly see if "helicity_seed&0x3fffffff" matches "last_helicity_state_pattern_sync&0x3fffffff").
I would also suggest looking at the two "*tstable_count" variables, the "pattern_sync_count" and the "trigger_pattern_phase_count".

Maybe also check the "last_pattern_sync" to make sure that isn't showing that the pattern sync in two sequential events, in case you need to use the HD board debounce setting.

[sdobbs]

I found a bit of time to look into this more. Here is an extraction of the requested values.
Note that we are comparing last_helicity_state_pattern_sync&0x3fffffff and (last_helicity_state_pattern_sync>>2)&0x3fffffff as the sanity check.

13:10:21.371 [error] Consistency check of Helicity Decoder Board data failed!
13:10:21.371 [error]   Bad word: 0x8abd5179
13:10:21.371 [error]      Compare 0x180179321 to 0x180179320
13:10:21.371 [error] 
13:10:21.371 [error] Compare helicity_seed&0x3fffffff = 0xabd51f9 and last_helicity_state_pattern_sync&0x3fffffff = 0xabd5179
13:10:21.371 [error]  falling_edge_tstable_count  = 170262
13:10:21.371 [error]  rising_edge_tstable_count   = 170389
13:10:21.372 [error]  pattern_sync_count          = 42618
13:10:21.372 [error]  trigger_pattern_phase_count = 3
13:10:21.555 [error] Consistency check of Helicity Decoder Board data failed!
13:10:21.555 [error]   Bad word: 0x8abd5179
13:10:21.555 [error]      Compare 0x180179321 to 0x180179320
13:10:21.555 [error] 
13:10:21.555 [error] Compare helicity_seed&0x3fffffff = 0xabd51f9 and last_helicity_state_pattern_sync&0x3fffffff = 0xabd5179
13:10:21.555 [error]  falling_edge_tstable_count  = 170390
13:10:21.555 [error]  rising_edge_tstable_count   = 170389
13:10:21.555 [error]  pattern_sync_count          = 42618
13:10:21.555 [error]  trigger_pattern_phase_count = 3

[sdobbs]

The last_pattern_sync is pretty much the same for these neighboring events. Not sure what else to look at regarding it.

[sdobbs]

circling back to this - @aaust, could you please check the logs of recent monitoring launches over the 2026 data to see if this error shows up again?

My impression about what happened here is that these errors from the firmware might be spurious, but if they aren't still showing up in the data, then there's not much to do in debugging them further. IMO, I think these errors in the 2025 are a very small fraction of the data, so it would be fine to keep reporting that the helicity information is bad, to be on the safe side.

I could change the default printouts to be less verbose, if that sounds like a good idea.

[markdalton]

I think a switch to turn on the verbose output is a good idea.
Could you count the number of events that fail and print it at the end? That would help in assessing the extent of any problems. I agree that since the asymmetries seem to make sense in magnitude and precision, this is likely benign.

[sdobbs]

yeah that should be possible

[markdalton]

Thanks for the new Sean!

Oh boy, run 133255_000 had 10% of events with errors. That's a level that's concerning. Is this a known bad case or a random file? Perhaps before we close this we should wait until the next monitoring launch to get an idea of the extent of the problem over the full data set.

[sdobbs]

It was not consistently this bad in a few spot checks I've seen, but yes we need to do a broader survey of the data with at least a monitoring launch.

[aaust]

circling back to this - @aaust, could you please check the logs of recent monitoring launches over the 2026 data to see if this error shows up again?

I don't see it in the incoming monitoring for 2026-06