Roi axis stack

[sergey-yaroslavtsev]

Support flatten data to be open in ROI tool.
Closing request ticket

[sergey-yaroslavtsev]

Do you think shortenScales is enough or more complicated logic should be setted?
Another "problem" is that this approach utilize old logic of setting Scales which are starting point + step. Thus, if scan was collected in a "snake", then image will be incorrect.

[woutdenolf]

Thus, if scan was collected in a "snake", then image will be incorrect.

The concat files are more complex than that. It is neither snake nor zig-zag. It is the concatenation of several 2D maps, potentially overlapping or with gaps.

The problem is not simple to solve in general.

• Silx uses matplotlib's Triangulation and LinearTriInterpolator (not sure how it determines the grid).
• Ewoksfluo has ScatterDataInterpolator and optimal_grid_axes which uses scipy's griddata to interpolate and minimize to find the grid.

[sergey-yaroslavtsev]

The concat files are more complex than that.

In cases which i saw it is saved as zig-zag like in hdf5 file. Same as dmesh do...

If i understood Silx approach correctly - it try to handle regular grid as Z-like or as a line.
Otherwise it handle irregular grid, but i doubt that ROI tool is adjusted to irregular grid at all... I am not aware of requests for irregular grid support in ROI tool.

[sergey-yaroslavtsev]

My feeling is that it is OK to add support for Z-like flatten data with regular grid as a most common case.
I can add disclaimer after opening such a data.

And on user request to start work on irregular grid support.

[woutdenolf]

To be clear this assumes the slow motor readout is not noisy.

[sergey-yaroslavtsev]

Indeed i thought it is most regular case, but it appears to be 50-50.
I need to fix it.

[woutdenolf]

Yes but the route of analyzing noise motors of regular scans to find the regular grid has been tried many times and is inherently flaky.

Your next implementation might involve some kind of noise threshold which of course is impossible to know.

Then you might realize there is only one property you can rely on: the values of the fast axis are "piecewise monotonic". Est is using that to split black-and-forth energy scans, but that's only a 1D problem. You need to first find the fast axis. Silx is using this for silx view.

And after all this work you realize you need to handle CTRL-C etc.

In my opinion all this is a dead end and we should not rely on regular vs. irregular. ewoksfluo uses an optimal grid finding approach for any list of nD coordinates.

[woutdenolf]

We sample the coordinates and take this as the final regular grid coordinates. To make it more robust we could take the median of all possible samplings. Not sure we care. All this is very approximate anyway.

[sergey-yaroslavtsev]

Not exactly. We find the grid from Z-like type. Then to reuse existing logic we take the first element and find mean step of the grid - this is xScale and yScale. So full grid is not going further.

[woutdenolf]

What I mean is you are using fast[:n] and slow[::n] which is a single sample. You are not using all the values in fast and slow.

You could do something like

ny = len(fast) // n

fast_scale = np.array([np.median(fast[j::n]) for j in range(n)])

slow_scale = np.array([np.median(slow[i * n : (i + 1) * n]) for i in range(ny)])

But as I said, trying to recognize zig-zag and snake coordinates is very flaky anyway. So this is not the biggest problem of the approach.

[woutdenolf]

mcaIndex, self.info['McaIndex'] and stackMcaIndex.

I'm going to assume you know what you're doing because I have no idea. Same for all the other places where you changed it.

[sergey-yaroslavtsev]

mcaIndex is initial index of channels. In most cases it goes to self.info["McaIndex"] to be stored but in some specific cases self.info["McaIndex"] stores not the mcaIndex but exact value.
line 459 in HDF5Stack1D.py for example:

        if (not DONE) and (not considerAsImages):
            _logger.info("Data in memory as spectra")
            self.info["McaIndex"] = 2
            n = 0

So i believe it is what should be used. But in most cases they are the same.

[woutdenolf]

How is 2 the exact value? Looks like an index to me, although it seems very random to suddenly choose 2.

[vasole]

Thus, if scan was collected in a "snake", then image will be incorrect.

The snake case is handled as a ROI Stack Plugin (ReverseStackPlugin).

However, I repeat the simplest and clearer way to handle all cases is to use the MaskScatterViewPlugin.

You have a flattened stack and the associated motor positions. That's all.

[vasole]

My opinion is that generic scans is what is to be supported. No regular mesh, no patch but arbitrary data collection points according to any custom strategy. That case is already supported.

You have a set of spectra measured at certain positions. Just make sure the positions are properly written in the input data file (and they will be automatically read) or make sure they can be read from an external file.

[vasole]

With the code as it is, the regular mesh case and the generic case are handled.

In my opinion you are looking at the wrong place to intervene.

[sergey-yaroslavtsev]

However, I repeat the simplest and clearer way to handle all cases is to use the MaskScatterViewPlugin.
...
In my opinion you are looking at the wrong place to intervene.

Probably you are right.
Just to clarify - the idea of a different approach would be to add possibility to transfer axis from Mask Scatter View to the ROI window.
Then the user will need to:

1. open ROI tool and select only data (no axis, because it will cause errors without this PR)
2. load positioners from file
3. open Mask Scatter View and select proper axis

It can work but sounds complicated to find out by user. Or do i miss a possibility to "cut the corner"?

[sergey-yaroslavtsev]

With the code as it is, the regular mesh case and the generic case are handled.

Would it make sense to you to have both possibilities? Because now "standard" ESRF scans could not be opened directly - i mean if data is collected with shape N*M (N is number of points, M is number of channels) and 2 motor arrays have shape of just N.

I would like to have a direct possibility to open "standard" scan without extra workaround. But if i am missing the way please point me out.

And for general case the approach with Mask Scatter View described above can be done.

[vasole]

It can work but sounds complicated to find out by user. Or do i miss a possibility to "cut the corner"?

Yes you do.

If the motor positions are properly stored they will be loaded automatically.

We are generating the data. So, let's generate them properly and every code and not just PyMca will be able to deal with them. PyMca is not there to workaround data generation issues.

If you want to improve the handling of non regular stacks, it would be more convenient to have the aternative to replace the regular ROI Imaging Window view by the Scatter one in the main ROI Imaging Window.

Even the person issuing the request was wondering if PyMca was the proper place to attack the problem.

The way to go is to properly deal with the generic case and to not try to combine patched acquisitions into a regular stack. Sooner or later you are going to face arbitrary acquisitions and all this work will be useless for it besides being prone to introduction of bugs.