method related files

[tshu-w]

Hi, I'm interested your implementations, can you point me to method related files, I found it's hard to find it since you change trasformers and fairseq directly. BTW, I think inheritance or override class and function without change them directly is a better way.

[ivanmontero]

Hey! Thanks for your interest, I'll update the repo soon to include the configuration files we used in the paper, and update this issue. Regarding using both fairseq and transformers: we use transformers for their pretrained model checkpoints (RoBERTa-{base, large}) and classification finetuning (GLUE), and use fairseq for any generation-related tasks (autoencoding, sentiment transfer)

Regarding inheritance and overriding: I completely agree. The main reason I did what I did was to follow closely to the way the transformers (and fairseq) do things, since they, even though two models may have very similar implementations, they copy and paste a lot of the code (i.e., the RoBERTa and BERT implementations)

[tshu-w]

Hi, I read you paper carefully and understand how you do it. So code is not necessary for me to understand your work now. 😄

[NakagawaAkira]

Hi,
I'm very interested in your paper.
Fixed length embedding is very easy to treat!
I would like to run your code.
I'm looking forward to updating your document including example usage and configurations!

[ivanmontero]

Thanks for the reminder on this! I just pushed a commit that starts some documentation on the data processing, pretraining, and finetuning stages, as well as some configurations to perform those. I'll work on fleshing them out more in the near future!

[NakagawaAkira]

Thank you very much! Now it is clear for me to pretrain / fine-tune the model.
I'm also looking forward to your further update ;-)