6

I am wondering if I can be able to use OpenAI GPT-3 for transfer learning in a text classification problem? If so, how can I get start on it using Tensorflow, Keras.

Guy Coder
  • 24,501
  • 8
  • 71
  • 136
anveshtummala
  • 432
  • 4
  • 11
  • if you told me which text classification problem, i could test it right away! @anveshtummala – Gewure Aug 10 '20 at 00:13
  • I am looking for text comment as hateful or not, I want to use my labeled data to train the model. – anveshtummala Aug 10 '20 at 00:18
  • yes, it can do that most likely. Please supply me with 3-4 samples of that labeled data and i check if it can do that and will answer your question! @anveshtummala – Gewure Aug 10 '20 at 01:51
  • without samples @anveshtummala i can't verify if it can do that. A good question on SO gives all the information necessary to reproduce. – Gewure Aug 11 '20 at 18:41
  • I am looking for samples that are less abusive or hateful. Not sure if stackoverflow might treat them as abusive comments. Say I have data like -> you look like a pig, hateful; you look like a bunny, not hateful; – anveshtummala Aug 11 '20 at 18:45
  • could you accept my answer and mark the question answered? – Gewure Sep 18 '20 at 18:16
  • 1
    I did accepted @gewure – anveshtummala Sep 18 '20 at 18:23

1 Answers1

6

(i substituted hateful language with ******** in the following samples)

Given samples like:

("You look like ****** *** to me *******", true)
("**** you *********", true)
("**** my ****", true)
("hey my name is John can you help me?", false)
("hey my name is John, i think you ****** ***!", true)
("i have a problem with my network driver hpz-3332d", false)

GPT-3 can indeed then decide if a given input is hateful or not. GPT-3 actually is implementing filters that will very effectively tell if an arbitrary comment is hatefull or not. You would just enter the msg and let GPT3 autcomplete the , true|false) part at the end, setting tokens to about ~6 and temperature setting 90%.

Boolean-ish classification that also relies on more complex context (you can insult someone without using foul-language) id doeable with GPT3 and can also be done with GPT2.

Gewure
  • 1,208
  • 18
  • 31
  • do you got any more questions, @anveshtummala? im happy to answer. – Gewure Aug 18 '20 at 01:25
  • 1
    Gewure, I have a document describing 10k categories, each with a short text (eg: "Car toy": "Toy with tires, small kick scooter, small cars with pedals. It can be electric with a maximum speed of 6km/h. Kids play with them"). Do you think it's possible to fine tune GPT-3 so that it can predict a category from some alternative description as input? – Nicoowr Sep 04 '20 at 13:40
  • 1
    @NicoLi interesting. I think you can utilize gpt3 for this, yes. But you most likely would need to supervise the outcome. I think you could use it to generate descriptions and then adapt them by hand if necessary. would most likely drastically speed up the process. – Gewure Nov 09 '20 at 18:50
  • 1
    I see! Would love to try it out but there is no sign of any release soon of GPT-3 on Azure :/ – Nicoowr Nov 10 '20 at 14:50
  • @NicoLi send me some samples to singazpacho @ gmail.com and i'll try it for you! :) – Gewure Nov 10 '20 at 18:11