Data consistency and homogeneity

[jddingemanse]

Multiple participants have mentioned the wish to work on code for testing data consistency and homogeneity. Can you explain better what testing consistency and homogeneity implies?

[BeyeneSe]

For a meaningful study, it was important to fill in the missing data and assess the quality of the hydro-meteorological data after gathering the necessary data. In various studies, several data quality approaches have been utilized, with some of them being used to assess the quality of the obtained data. After filling the missed values we should check the quality of our data by using consistency and homogeneity because during measurement and filling missing data different errors are produced. So we should check the quality of our data using this two methods since the data should be homogeneous and independent.

Homogeneity and consistency assures that the observations are from the same population. Several methods are used to check the quality of the data, but the double mass curve method (DMC) is the most effective. It is a plot of the accumulated rainfall data at a station in question against the mean of the cumulative rainfall for a group of surrounding stations. This method is used after the data at all stations were filled.

[Meselegithubmarkos]

In addition to this DMC is limited to monthly and yearly totals, as it normally does not work with daily ones.

[Yitugoshu0915695361]

In addition, to the above discussion
The first step of any hydrological and meteorological study is accessing reliable data. After collecting the necessary data, filling the missing data and checking the quality of the hydro-meteorological data is mandatory. Therefore, after filling the missing data, the consistency and homogeneity of rainfall data from individual stations was evaluated. The non-dimensional of rainfall record is one of the methods to check the homogeneity of the rainfall data. Homogeneity for all stations was tested by computing non-dimensional of rainfall data by dividing the monthly time series data by the average rainfall amount of the respective year.
The homogeneity of flow data was checked by using RAINBOW software. RAINBOW software is easy to install and use for checking and adjusting inconsistency in hydro meteorological data caused by changes in observation method or data processing. Frequency analysis of data requires that the data must be homogeneous and independent. The restriction of homogeneity assures that the observations are from the same population. RAINBOW offers a test of homogeneity which is based on the cumulative deviations from the mean.

[jddingemanse]

As for the RAINBOW software, that sounds interesting, but hopefully you can explain (or find out) what the RAINBOW software exactly does, so that we can try to do the same with Python.

[jddingemanse]

So, if I understand correctly, for precipitation at least the statement is that DMC is relevant. Bahafta has worked on code for DMC (see here), please check it and comment / recommend any changes or clarifications. Also, can someone please post the equations (a structured description) of DMC? Then later on we can add that in the documentation, and based on that improve the code as started by Bahafta or propose additions.

For homogeneity, if I understand correctly, that is simply a matter of plotting multiple lines in the same graph in order to compare them, correct?

For the rest, I read 'several methods are used to check the quality' - @BeyeneSe (and others): could you list (besides DMC) a couple of the ones commonly used within AWTI studies?
And, @Meselegithubmarkos , you state a limitation of DMC (if I understand you correctly). Can you propose other methods that work on a finer timescale?

[BeyeneSe]

The consistency, independence, and stationery of the data are checked using the following techniques:
I. The W-W (Wald-Wolfowitz) test
II. The correlation tests
III. The M-W (Mann-Whitney) test

In AWTI research, the double mass curve method is the most popular one.

[jddingemanse]

Correlation tests sounds like something that we should not try to write code for, but maybe some classroom material (there can be several correlation tests, I think). However, W-W and M-W: can you share (links to) equations for that, is it relevant if we turn those equations into code and potentially a function, such that a user can supply columns of data for stations to compare, and automatically W-W or M-W are applied?
Bahafta is already working on DMC.

[BeyeneSe]

Here is the link: https://link.springer.com/book/10.1007/978-94-007-1861-6