1

I have a dataset of responses where people were requested to answer a set of questions. There's only one column of text data to process.

My challenge is; only very few respondents have actually written long texts that I found easy to process and gain insights from it. Most of the other responses are often found to be very short such as "Somewhat", "Yes", "No", "Larger extent". That too, it has been not possible to scale it ordinally because there's no logical order to it.

I have been able to use the longer text responses to gain insights on Sentiments, extract keywords and phrases and apply Machine learning such as RAKE and PMI. I used UDPIPE library with R.

However, for shorter "few words" responses, I am finding it really difficult to gain insights from these.

Is there any other machine learning technique possible with the current issue I'm having ? Or do I need to try any NLP technique ?

Dinesh
  • 654
  • 2
  • 9
  • 31
  • Biterm topic models are particularly good at finding clusters in short text. E.g. with the BTM R package (availble on cran https://cran.r-project.org/web/packages/BTM or at https://github.com/bnosac/BTM). Have you tried that? It allows you to find topics and next you can apply visualisation or summarisation of text of each topic. –  Jan 04 '19 at 14:17
  • I've not used BTM before, but it's an interesting concept. I shall try this now. Do you have any examples of how the model is built and if any visuals have been generated from the output of the model ? I have read the manual as reference for now but it's quite brief. – Dinesh Jan 04 '19 at 14:40
  • I'm planning to write a blogpost about this next week. –  Jan 04 '19 at 14:42
  • That's great, is this the blog site: http://www.bnosac.be/index.php/blog ? – Dinesh Jan 04 '19 at 14:49
  • Please kindly provide an update if there's anything updated on the BTM blog, I'm definitely looking forward to reading it, Thanks so much! – Dinesh Jan 07 '19 at 23:08
  • 1
    example put here for the time being: https://github.com/bnosac/BTM/issues/4 - good luck –  Jan 10 '19 at 21:46
  • Many Thanks for this! – Dinesh Jan 12 '19 at 18:26

0 Answers0