Multi class settings

[shikunyu8]

Hi, neven. Could you update this repo to multi class settings as decribed in your paper? I'm trying to reproduce the results on Cityscapes. Thanks.

[charlotte12l]

@davyneven I'm reproducing your multi class results recently, but the current results are not good. I think there may be something different between my implementation and yours. May I ask these several questions:

• Are your seed maps' channels 8 or 9 (8+bkg)?
• How did you calculate loss when it is multi class? I just saw your multi-class lovasz_softmax loss, the input channel is C, but I think the input embedding should always be 2 channels no matter it is one class or multi-class? In my implementation, I still use the lovasz_hinge loss rather than lovasz_softmax loss.
• How did you cluster when it is multi class? I did sequential clustering on each seed map one by one...
  Thank you very much!!!

[davyneven]

Hi,

• Are your seed maps' channels 8 or 9 (8+bkg)?
  I use a seed map per class, so for Cityscapes this would mean 8 seed maps. Each seed map is trained as a one-vs-all map, so no softmax.
• How did you calculate loss when it is multi class?
  I use the same loss for the multi-class setting, but add an extra for loop to loop over the 8 classes. This time, the foreground weight for the seedmap is different for each class, and is calculated as described in the ERFNet paper
• How did you cluster when it is multi class?
  I indeed do a sequential clustering on each seed map.

[charlotte12l]

@davyneven Thank you so much for your quick reply!! My seed map is also 8 channels, and I also used an extra for loop over the 8 classes, but I use the same foreground weight, this may be a problem. Also, I have a question about clustering:

• How did you choose the seed map sequence to cluster? eg. First car, then person, train, motorcycle, bicycle...
  I think the order matters because there may be some cases like, the object is a bicycle, but in the motorcycle seed map, it happens to have some prediction values, and the max value is larger than the threshold 0.9 (but maybe smaller than that on the bicycle). Due to the order, it will first cluster to a motorcycle instance rather than bicycle.

[davyneven]

The foreground weights are very import to reach good results, so you should definitely see some improvements. The order of clustering the seed maps of the different classes doesn't matter. I cluster each seed map separately, and save each clustered object. So indeed, this way you can have a same object detected in two seed maps, eg a bicycle can be detected as a bicycle and also as a motorcycle.

[charlotte12l]

@davyneven Thanks for your reply! I use your clustering method and the performance imporved! However, I double-checked the ERFNet paper and found nothing about class weights, their code is also not available now. Can you share me your code or paste part of them on this issue?:) Thanks!

[Croooooow]

Hi, I have a question about the for loop over seed map in calculating the seed loss. Since an instance can only belong to a single class, so why do we need this for loop? In my implementation, I simply do it this way: seed_loss = seed_map[c][mask] - dist[mask], where c is the instance class. But when it comes to a for loop, the mask region of other channels in seed map are useless and should be zero, right? If so, seed_loss = seed_map[c][mask] - 0 ? Any idea? @davyneven @charlotte12l

[Linwei-Chen]

I would like to do some research about instance segmentation based on your work, so I'm reproducing your multi class results, and I just can not reach the same performance as yours, my configuration:

1. 8 seed maps
2. sequence cluster for each class
3. same weights as yours

Training:

1. pretrain 200 epochs, adam, 371 iter/epoch, batch_size = 8, crop_size = 512, lr = 0.0005, object center transform
2. freeze bn for another 50 epochs, lr = 0.00005. crop_size = 936, bath_size = 2, random crop and object center transform reach the same performance.

My questions:

1. I can only reach 26 AP on val. Especially, my model reach really low AP with Train, motocycle and cycle. (0.12, 0.12, 0.2), could you give me some advice? What's more, I do not generate the crop, I just use the CropRandomObject transfrom. Is this wrong?
2. what's your total iteration? Maybe mine is not enough, which is about 80k + 100k.
3. And I wonder what the performance in validation ? 33 AP?
4. Do you use sync_bathnorm?
5. Should I set foreground weight for each class? How should I set the loss weight properly?

Your reply will be very important and helpful for me. Thanks a lot.
@davyneven

[Croooooow]

Hi， can you share your code? @GitHberChen

[Linwei-Chen]

Hi， can you share your code? @GitHberChen

Sorry, I can not, maybe later.

[Croooooow]

Hi， can you share your code? @GitHberChen

Sorry, I can not, maybe later.

How is your performance in the first training stage?

[Linwei-Chen]

Hi， can you share your code? @GitHberChen

Sorry, I can not, maybe later.

How is your performance in the first training stage?

about 21 AP on val, what about yours?

[Croooooow]

Hi， can you share your code? @GitHberChen

Sorry, I can not, maybe later.

How is your performance in the first training stage?

about 21 AP on val, what about yours?

11:( There must be something wrong with my code. Can you tell me how you calculate the seed loss? I think I may have problem here. And by the way, do you test on 2048*1024?

[Linwei-Chen]

Hi， can you share your code? @GitHberChen

Sorry, I can not, maybe later.

How is your performance in the first training stage?

about 21 AP on val, what about yours?

11:( There must be something wrong with my code. Can you tell me how you calculate the seed loss? I think I may have problem here. And by the way, do you test on 2048*1024?

yes, of course

[Croooooow]

Hi， can you share your code? @GitHberChen

Sorry, I can not, maybe later.

How is your performance in the first training stage?

about 21 AP on val, what about yours?

11:( There must be something wrong with my code. Can you tell me how you calculate the seed loss? I think I may have problem here. And by the way, do you test on 2048*1024?

yes, of course

Can you show me your seed loss？Your 26 AP is very close to the 27.6 claimed in paper and I think your implementation is right.

[charlotte12l]

@GitHberChen Hi, the author said the foreground weight should be set as ERFNet paper said.. But the paper doesn't explain how they calculate the weight, I think their ERFNet code do but the code is now invisible... If you know how to calculate the weight, please contact me, thanks! :)