Implement near-range and far-range telesope glueing based on signals

[HolgerPollyNet]

According to the NOA approach as another option.

This could lead to profiles of bsc, ex, and depol combining both telescopes...

[mtsichla]

I have added HolgerPollyNet to Gluing Repository to access the code. I attach the read me file for the code.
GLUING CODE INPUT.docx
How do you want to proceed?

[HolgerPollyNet]

Dear @mtsichla ,

according to

you glue on the averaged signals, correct?
Then the correct position for including it in the chain would be here:

Pollynet_Processing_Chain/lib/qc/pollyOLCor.m

Line 154 in e85bef0

% signal gleuing according to maria T script

Could you define which VARIABLES you need for your code to run (preferably the range-corrected signal would be great),

then @ulysses78 can help implementing the python code executed out of matlab!

[mtsichla]

Hello @HolgerPollyNet & @ulysses78

I calculate the gluing region and the normalization factor after averaging the range corrected signal but then I implement these values in each 30s range corrected profile seperately and not in the averaging signal (so the gluing can be shown in a quicklook also).

The input to my code is the range-corrected signal yes and the following variables and constants:

From a raw file:

1. the length of the variable "measurement_time"
2. the length of the dimension "channel"
3. the length of the dimenson "height"
4. the variable "depol_cal_angle"
5. the variable "if_center"
6. the variable "polstate"
7. the variable "telescope"

(The 5, 6 and 7 will define which channels to glue since not all PollyXTs has the same near field channels.)

Constants:

8. the average interval
9. overlap (in bins, not meters) for each gluing channel (for example 80 which corresponds to 600m)
10. first bin (this can be omitted)
11. correlation coefficient threshold for each gluing channel (for example 0.75)
12. signal to noise ratio threshold for each gluing channel (for example 5)
13. the depolarization calibration angle when the polarizer is in the normal position (for example for aky is 0)

If in your code the sigFR is the range corrected signal and we can have the variable information from another part of the code and insert them there I think it will work.

[HolgerPollyNet]

Hi @mtsichla ,

can you provide a little bit more detail of the variables and constants. E.g. type (array, which dimensions, etc.) and what values are possible. E.g. for telescope it is not 100% clear what is meant. The I can prepare the implementation. And what is the output? The merged signal only?

[mtsichla]

Hello @HolgerPollyNet ,

The variables I read from the netcdf are: "depol_cal_angle" (float), "if_center" (float), "polstate"(integer) and "telescope"(integer). All these are 1d arrays.

The telescope variable has 0 and 1 values which correspond to far and near field and I am using it in combination with the if_center and polstate (values: 0, 1, 2) to find the pairs for gluing.
If in pollynet you don't read the netcdfs and you take these information in another way let me know so we can modify the code.

The other values I take from netcdf is
time: integer value (the number of time profiles)
channels : integer value (the number of channels )

The constants I use are:
the average interval: integer value (<=number of time profiles)
overlap: integer value (in bins) for each gluing channel (for example 80 corresponds to 600m)
correlation coefficient threshold: float value (0-1) for each gluing channel (for example 0.75)
signal to noise ratio threshold: float value. I used a simplified expression of the signal to noise ratio (mean of array/standard deviation of array) so the range is not standard. I used the value 5 in my runs which suggests that the signal is 5 times stronger than the noise.
the depolarization calibration angle value: float value. It is the angle when the polarizer is in the normal position (for example for aky is 0).

The output is a 3d array with the merged signals and dimensions (time, height, channels).

We can arrange also a meeting to explain these better if you want.

[HolgerPollyNet]

Hey, yes I think a meeting would be useful, as we may have all this variables you need already there. I think, we should start on averaged signals, the same like used for the optical values?

But I see that you are keen to do it high resolution....?

What you need I guess: The elastic far and near range channel, the respective raman ones.

But lets have a brief meeting,maybe with @ulysses78 and @HenniHG .

[mtsichla]

Hello @HolgerPollyNet,

Yes I believe that the averaged interval should be the same as for the optical products.

For the high resolution you mention I had in mind that we could provide quicklooks similar to the "Calibrated attenuated backscatter coefficient OC" you used to show in the previous version of tropos but for the glued signals now.
If we are not interested in showing that or if there is another way then I will adjust the code to glue only the averaged signals.

Lets arrange a meeting when we are all available to conclude.

Can you provide me with @ulysses78 and @HenniHG emails to send a doodle?

Thank you!

[ulysses78]

Hi Holger,
I have implemented Marias python-script into picasso by using pyrun. The commit of branch "signal_glueing_mira" is here:
d59991a

I would suggest to test this branch on the rsd2-server before merging into dev-branch.

[ulysses78]

I just tested it on rsd2 and there are still some things to take care of ;-)

[HolgerPollyNet]

okidoki, full trust in you ;-)

[ulysses78]

Now it runs the whole way through with some new variables: sigGL[wavelength] of dimension[len(profiles),len(heights)]. But still there is no netcdf-file written out with the glued signals.
At the moment it is not optional to merge NR+FR signals, but will be done automatically within the overlap-correction calculations.