How to start v3.0

[Ropzn]

@ZPYin : Hi Zhenping,
I'm planing to integrate the dual-field-of-view depolarization method (cloud-microphysics), which was developed by Cristofer.
I was wondering how I would start the pollynet_processing_chain (v.3?) for a single file, which I've got locally on my pc.

It doesn't have to work perfectly, but i would like to run it a little to get an idea of the workflow, variable names and so on.

Is it something like:
[report] = picassoProcV3('C:\Users\robzn\Programmierwiesen\Daten\Arielle\mosaic\2019_11_14_Thu_ARI_00_00_02.nc', <PollyType>, 'C:\Users\ropzn\Programmierwiesen\Pollynet\Pollynet_Processing_Chain\config\pollyConfigs\arielle_config_20191201.json');

But what is the variable "PollyType"?

Or do i start with a shell-script?

The problem is, that I cannot wait for the final release of v3.0, since I only have spare-time on this thing for the next two/three weeks, afterwards I wont have much time :-S

Greetings R.

[HolgerPollyNet]

@ZPYin , could you give fast advise? Shall we have a short vidcon about this?

[ZPYin]

@Ropzn Hi,

So far, Picasso v3.0 should be ready for further development, at least well tested with previous polly data.
But unfortunately, there are no full documentation yet. I'm still busy with resolving the github issues associated with Picasso v3.0. So you would need spend some time reading the main program. The main program as you mentioned, is picassoProcV3.m.

You can run it with any polly data by the command below (let's take arielle for example):

picassoProcV3('D:\Data\PollyXT\arielle\data_zip\201903\2019_03_22_Fri_ARI_12_00_01.nc', 'arielle', 'D:\Coding\Matlab\Pollynet_Processing_Chain\config\pollynet_processing_chain_config.json')

Regarding to polly type, it's the short name for different polly systems. Below is the full list of polly type so far, built for PollyNET:

• arielle
• Polly_1st
• Polly_1V2
• PollyXT_CGE
• PollyXT_DWD
• PollyXT_FMI
• PollyXT_IFT
• PollyXT_LACROS
• PollyXT_NIER
• PollyXT_NOA
• PollyXT_TROPOS
• PollyXT_UW
• PollyXT_TJK
• PollyXT_TAU
• PollyXT_CYP
• PollyXT_CPV

You can start your implementation with Picasso v3.0. But maybe it's good to create new branch from v3.0 for hosting your implementation, see the git command below:

git fetch origin v3.0:v3.0   # pull Picasso v3.0
git checkout v3.0
git checkout -b [branch_name]

# start your implementation under your new branch. I will help to merge your implementation after you finish your work.

If there is anything unclear for you (or bugs), just let me know.

Best greetings!
Zhenping

[Ropzn]

@ZPYin Thank you.
Very impressive Program !

To use it I had to restore the two folders pollyConfig and pollyDefaults which where deleted by this commit
99b1cca

Then it seems to run nicely until stops with an error:

Index in position 1 is invalid. Array indices must be positive integers or logical values.

Error in picassoProcV3 (line 1192)
    if isnan(refHInd355(iGrp, 1)) || (sum(flag355FR) ~= 1)

Full Matlab-output:

> picassoProcV3('C:\Users\*****\Programmierwiesen\Daten\Arielle\mosaic\2019_11_14_Thu_ARI_00_00_02.nc', 'arielle', 'C:\Users\*****\Programmierwiesen\Pollynet\Pollynet_Processing_Chain\config\pollynet_processing_chain_config.json')
Warning: Directory already exists. 
> In picassoProcV3 (line 123) 

%------------------------------------------------------%
    ____  _                               _____  ____
   / __ \(_)________ _______________     |__  / / __ \
  / /_/ / / ___/ __ `/ ___/ ___/ __ \     /_ < / / / /
 / ____/ / /__/ /_/ (__  |__  ) /_/ /   ___/ // /_/ /
/_/   /_/\___/\__,_/____/____/\____/   /____(_)____/

Start pollynet processing chain
Picasso config file: C:\Users\*****\Programmierwiesen\Pollynet\Pollynet_Processing_Chain\config\pollynet_processing_chain_config.json
Polly Type: arielle
Polly Data: C:\Users\*****\Programmierwiesen\Daten\Arielle\mosaic\2019_11_14_Thu_ARI_00_00_02.nc
%------------------------------------------------------%
## PC Info
USER: *****
HOME: C:\Users\*****
OS: win
MATLAB: 9.10.0.1669831 (R2021a) Update 2
[2021-07-07 10:43:46] Start searching polly campaign information and polly configurations.
arielle campaign info:
location: Arctic
Lat: 13.000000
Lon: 52.000000
asl(m): 83.000000
startTime: 2019-09-15 00:00
caption: , RV Polarstern, Mosaic, Arctic
arielle process info:
config file: arielle_config_20191201.json
process func: pollynet_processing_chain_arielle
Instrument info: , RV Polarstern, Mosaic, Arctic
polly defaults file: arielle_defaults_mosaic_dec2019.json
[2021-07-07 10:43:48] Finish.
[2021-07-07 10:43:48] Start loading polly config.
Warning: 'channelTag' was deprecated 
> In loadPollyConfig (line 135)
In picassoProcV3 (line 256) 
Warning: 'minSNR_4_sigNorm' was deprecated 
> In loadPollyConfig (line 123)
In picassoProcV3 (line 256) 
[2021-07-07 10:43:48] Finish.
[2021-07-07 10:43:48] Start loading polly defaults.
[2021-07-07 10:43:48] Finish.
[2021-07-07 10:43:48] Start reading arielle data.
[2021-07-07 10:44:05] Finish.
Warning: Unknown channel tags (0) at channel 13 
> In pollyChannelTags (line 142)
In picassoProcV3 (line 326) 
[2021-07-07 10:44:05] Start reading laserlogbook file.

Warning: laserlogbook file does not exist.

 
> In readPollyXTLaserlogbook (line 68)
In readPollyLaserlogbook (line 58)
In picassoProcV3 (line 356) 
[2021-07-07 10:44:05] Finish.
[2021-07-07 10:44:05] Start lidar data pre-processing.
[2021-07-07 10:45:28] Finish.
[2021-07-07 10:45:31] Start detecting signal saturation.
[2021-07-07 10:45:34] Finish.
[2021-07-07 10:45:34] Start polarization calibration.
[2021-07-07 10:45:58] Finish.
[2021-07-07 10:45:58] Start cloud screening.
[2021-07-07 10:46:00] Finish.
[2021-07-07 10:46:00] Start overlap estimation.
[2021-07-07 10:46:02] Finish.
[2021-07-07 10:46:02] Start overlap correction.
[2021-07-07 10:46:06] Finish.
[2021-07-07 10:46:06] Start cloud-free profiles segmentation.
No cloud-free groups were found.
[2021-07-07 10:46:06] Finish.
[2021-07-07 10:46:06] Start loading meteorological data.
gdas File (C:\Users\*****\Programmierwiesen\Pollynet\Pollynet_Processing_Chain\gdas1\polarstern\ NaN\NaN\*_ NaNNaNNaN_NaN*.gdas1) does not exist.
The meteorological data of gdas1 is not ready.
Use standard_atmosphere data as a replacement.
Warning: Number of valid parameter data points is less than 3. 
> In interpMeteor (line 28)
In loadMeteor (line 103)
In picassoProcV3 (line 711) 
Warning: Number of valid parameter data points is less than 3. 
> In interpMeteor (line 28)
In loadMeteor (line 104)
In picassoProcV3 (line 711) 
Warning: Number of valid parameter data points is less than 3. 
> In interpMeteor (line 28)
In loadMeteor (line 105)
In picassoProcV3 (line 711) 
[2021-07-07 10:46:08] Finish.
[2021-07-07 10:46:08] Start loading AERONET data.
Could not extract photometer data (Level: 15)
https://aeronet.gsfc.nasa.gov/cgi-bin/print_web_data_v2?site=&year=119&month=11&day=13&year2=119&month2=11&day2=15&LEV15=1&AVG=10
[2021-07-07 10:46:12] Finish
[2021-07-07 10:46:12] Start Rayleigh fitting.
[2021-07-07 10:46:12] Finish.
[2021-07-07 10:46:12] Start to retrieve aerosol optical properties.
Meteorological file :  standard_atmosphere.
Index in position 1 is invalid. Array indices must be positive integers or logical values.

Error in picassoProcV3 (line 1192)
    if isnan(refHInd355(iGrp, 1)) || (sum(flag355FR) ~= 1)

Concerning Git: I forked the project and will work there on my branches.

Greetings, Ropzn

[ZPYin]

I've pushed a fix for this bug to v3.0. Please give it a try.