Unsupervised fine-tuning on custom documents after the supervised fine tuning on general question-answers dataset. Will it be useful for GPT-2 model?

Question

I know the formal way of training a GPT2 model on custom documents is to first do semi-supervised fine tuning on the text of the documents followed by supervised fine-tuning on question answers from the same documents. But the sole purpose of supervised fine-tuning being to acquire style of answering question, is it possible to do supervised fine-tuning on a general dataset, and after that perform unsupervised fine-tuning on our custom text dataset from documents. This way question answering style can also be acquired by the model along with the advantage of having no need of making a question-answer dataset for the custom documents.

Will it give the desired results?

Answer 1

It is very difficult to say this methodology would 'work' reliably for use cases. One approach I have tried is taking a base model and causally fine-tuning it on the documents at hand. Following this, you can take a publicly created Q&A dataset like SQuAD and further fine-tune in a prompt + expected response way. During this supervised stage, much research has shown that using Parameter Efficient methods for this task adaptation stage is more beneficial than training all weights (see LoRa).

Finally, I will say this: for Question and Answering systems, I personally have found that using In-Context learning has been far more beneficial than fine-tuning and closed-book Q&A - even in the case that using Vector DBs and Embeddings is required to search for relevant chunks of context.