2

Hi I am trying to train the dolly-v2-12b or any of the dolly model using a custom dataset using A10 gpu. I am coding in pycharm, windows os. The task is similar to a Q&A. I am trying to use this as communication assistant that can answer the queries. I have the dataset with more than 10,000 entries each entry might have around 3000 characters.

I wanted to know whether this is possible with the GPU I have and how long will it take to train on the dataset:-training time

UPDATE: I found code and used it to train the model. With my NVIDIA GeForce RTX 3060 Graphics card, I was able to train model upto 1billion parameters anything more gets CUDA out of memory error. I didn't try 8 bit mode yet. I will update when I make it work.

UPDATE: So I was trying to make the dolly work on 8 bit mode on my pc which is windows system. But the code I find uses bitsnadbytesand deepspeed. Somehow I find both of these impossible to install and use on windows systems. If anyone got it please help. I tried the instructions on both of the Github(bitsandbytes and deepspeed). But I am not being able to install it. Is there a workaround or any solution for this.

Sneha T S
  • 21
  • 6
  • I believe you could use [WSL](https://learn.microsoft.com/en-us/windows/wsl/install). For reference; I machine translated the Open Assistant dataset which resulted in ~30k entries with much lower number of characters per entry. I can fine tune a 7B, 4bit LLaMa model with my data and 1 epoch takes rougly 1 day. I also have a 3060 with 12 GB vRAM. :^) – doneforaiur Aug 30 '23 at 04:19
  • Thank you so much for the reply. I will try WSL and see how it goes. – Sneha T S Aug 30 '23 at 05:01
  • I installed WSL today and tried to configure it on pycharm, but I am using pycharm community edition. It seems only pycharm professional support WSL. Correct me if I am wrong. – Sneha T S Aug 31 '23 at 11:49
  • Just checked [this](https://www.jetbrains.com/help/pycharm/using-wsl-as-a-remote-interpreter.html) link which says "Available only in PyCharm Professional", sadly you are correct. – doneforaiur Aug 31 '23 at 13:24
  • yes, I think It's time to explore online options. I have never tried it so need to do my research. Thinking on starting with colab which I have already worked with. Then may be rty amazon aws. If anyone has tried dolly on online instance please reply. – Sneha T S Sep 01 '23 at 05:39

0 Answers0