`Device_name` fails to load

[bruno-f-cruz]

The following error is returned when attempting to read a DeviceName register with the interface. Likely an issue with how the data is interpreted since each position of the array is a character.

ValueError                                Traceback (most recent call last)
Cell In[27], line 7
      5 if register_name in device_reader.registers:
      6     _this_reader = device_reader.registers[register_name]
----> 7     _data = _this_reader.read(fname)
      8     #except ValueError as e:
      9     #    if register_name != "DeviceName":
     10     #        raise(e)

File \.venv\Lib\site-packages\harp\reader.py:162, in _create_register_reader.<locals>.reader(file, columns, epoch, keep_type)
    159 if file is None:
    160     file = f"{params.path}_{register.address}.bin"
--> 162 data = read(
    163     file,
    164     address=register.address,
    165     dtype=dtype(register.type),
    166     length=register.length,
    167     columns=columns,
    168     epoch=epoch,
    169     keep_type=keep_type,
    170 )
    171 return data

File \.venv\Lib\site-packages\harp\io.py:122, in read(file, address, dtype, length, columns, epoch, keep_type)
    112     raise ValueError(f"expected payload length {length} but got {payloadshape[1]}")
    114 payload = np.ndarray(
    115     payloadshape,
    116     dtype=payloadtype,
   (...)
    119     strides=(stride, elementsize),
    120 )
--> 122 result = pd.DataFrame(payload, index=index, columns=columns)
    123 if keep_type:
    124     msgtype = np.ndarray(
    125         nrows, dtype=np.uint8, buffer=data, offset=0, strides=stride
    126     )

File \.venv\Lib\site-packages\pandas\core\frame.py:816, in DataFrame.__init__(self, data, index, columns, dtype, copy)
    805         mgr = dict_to_mgr(
    806             # error: Item "ndarray" of "Union[ndarray, Series, Index]" has no
    807             # attribute "name"
   (...)
    813             copy=_copy,
    814         )
    815     else:
--> 816         mgr = ndarray_to_mgr(
    817             data,
    818             index,
    819             columns,
    820             dtype=dtype,
    821             copy=copy,
    822             typ=manager,
    823         )
    825 # For data is list-like, or Iterable (will consume into list)
    826 elif is_list_like(data):

File \.venv\Lib\site-packages\pandas\core\internals\construction.py:336, in ndarray_to_mgr(values, index, columns, dtype, copy, typ)
    331 # _prep_ndarraylike ensures that values.ndim == 2 at this point
    332 index, columns = _get_axes(
    333     values.shape[0], values.shape[1], index=index, columns=columns
    334 )
--> 336 _check_values_indices_shape_match(values, index, columns)
    338 if typ == "array":
    339     if issubclass(values.dtype.type, str):

File \.venv\Lib\site-packages\pandas\core\internals\construction.py:420, in _check_values_indices_shape_match(values, index, columns)
    418 passed = values.shape
    419 implied = (len(index), len(columns))
--> 420 raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")

ValueError: Shape of passed values is (1, 25), indices imply (1, 1)

[RoboDoig]

Ran across this issue today when reading vestibularH1 data. I think the culprit is this line:

harp-python/harp/reader.py

Line 223 in ebd12ce

reader = partial(reader, columns=[name])

For registers with length > 1 the _create_register_parser function will make a reader with only a single column name which ends up raising the pandas ValueError as the data payload is a different shape to the implied DataFrame shape.

Could be fixed with a single line like:

columns = [name] if register.length == 1 else [f"{name}_{i}" for i in range(register.length)]

but depends how you'd prefer each column of the register to be named. Is RegisterName_0, RegisterName_1, ... informative enough?

This doesn't work for your original case where the issue is interpretation of each character of a string, so alternate solution would be to put the whole array of data under a single column.

[glopesdev]

@RoboDoig well spotted, I have raised this as a separate issue which should definitely be solved, and we will keep this one for the string parsing problem.