Training for better embedding using self-supervised fashion with masked language modeling (MLM) for my own dataset

[Frank-LIU-520]

I have a dataset containing millions sequences without any labels generated with specific method in lab. I would like to train with those sequences to get better embedding for those unlabeled sequence starting from the ProtTrans pretrained model. What should I do?

[mheinzinger]

We are currently working on a script that allows for continuing ProtT5's pre-training.
I'll add a note to the README where I link to it.
If its urgent, feel free to use the information provided in this issue (And the referenced issues): #158

[mheinzinger]

Update: we now have an example for continuing pre-training.