Extracting keywords in each row of a data frame using udpipe in R

Question

I am using the R package udpipe to extract keywords in my data frame. Let's start with some data contained in the package:

library(udpipe)
data(brussels_reviews)

If we look at the structure, we see it contains 1500 comments (rows) and 4 columns.

str(brussels_reviews)
'data.frame':   1500 obs. of  4 variables:
 $ id        : int  32198807 12919832 23786310 20048068 17571798 28394425 46322841 27719650 14512388 37675819 ...
 $ listing_id: int  1291276 1274584 1991750 2576349 1866754 5247223 7925019 4442255 2863621 3117760 ...
 $ feedback  : chr  "Gwen fue una magnifica anfitriona. El motivo de mi viaje a Bruselas era la busqueda de un apartamento y Gwen me"| __truncated__ "Aurelie fue muy atenta y comunicativa. Nos dio mapas, concejos turisticos y de transporte para disfrutar Brusel"| __truncated__ "La estancia fue muy agradable. Gabriel es muy atento y esta dispuesto a ayudar en todo lo que necesites. La cas"| __truncated__ "Excelente espacio, excelente anfitriona, un lugar accessible economicamente y cerca de los lugares turisticos s"| __truncated__ ...
 $ language  : chr  "es" "es" "es" "es" ...

When following this tutorial, I can extract keywords of all the data frame together. Excellent.

However, my requirement is to extract keywords in every row, and not all the data frame as a whole.

I acknowledge that with this example, it does not make much sense, as there is only one single column with text (feedback). However, in my real example, I have plenty of columns with text.

So, I would like to extract keywords in every row of the data frame. So if we extract keywords in this example, I would like to get 1500 groups of keywords, each group for each row.

How can I do it?

UPDATE with and EXAMPLE

Following these two steps, we get the keywords of all the dataframe. However, I would like to get the keywords in every row of the data frame.

First step

library(udpipe)
library(textrank)
## First step: Take the Spanish udpipe model and annotate the text. Note: this takes about 3 minutes
data(brussels_reviews)
comments <- subset(brussels_reviews, language %in% "es")
ud_model <- udpipe_download_model(language = "spanish")
ud_model <- udpipe_load_model(ud_model$file_model)
x <- udpipe_annotate(ud_model, x = comments$feedback)
x <- as.data.frame(x)

Second step

## Collocation (words following one another)
stats <- keywords_collocation(x = x, 
                             term = "token", group = c("doc_id", "paragraph_id", "sentence_id"),
                             ngram_max = 4)
## Co-occurrences: How frequent do words occur in the same sentence, in this case only nouns or adjectives
stats <- cooccurrence(x = subset(x, upos %in% c("NOUN", "ADJ")), 
                     term = "lemma", group = c("doc_id", "paragraph_id", "sentence_id"))
## Co-occurrences: How frequent do words follow one another
stats <- cooccurrence(x = x$lemma, 
                     relevant = x$upos %in% c("NOUN", "ADJ"))
## Co-occurrences: How frequent do words follow one another even if we would skip 2 words in between
stats <- cooccurrence(x = x$lemma, 
                     relevant = x$upos %in% c("NOUN", "ADJ"), skipgram = 2)

Answer 1

Simple for-loop:

result <- NULL
for(i in 1:nrow(brussels_reviews)){
    result[i] <- somefunction(brussels_reviews[i, 3])
}

Above code is a general approach to loop through all rows of brussels_reviews, apply a function to the 3rd column, and save the result to a vector. This can also include a nested loop for the the columns as well. (see below)

If you elaborate a bit on what function you use exactly I'll be glad to help.

Mock code

k <- 1
result <- NULL

for(i in 1:nrow(df)){
    for(j in 1:ncol(df)){
        result[k] <- str_extract_all(df[i, j], "[A-Z]")
        k <- k + 1
    }
}

> head(result)
[[1]]
 [1] "P" "W" "Y" "V" "L" "X" "Y" "E" "E" "V" "T" "X" "O" "O" "Y" "A" "W" "P"
[[2]]
[1] "Q" "J" "O" "J" "P" "S"
[[3]]
 [1] "M" "E" "S" "I" "A" "Y" "J" "U" "M" "V" "W" "A" "P" "U" "I" "A" "X" "K"
[[4]]
[1] "T" "R" "H" "I" "S" "I"
[[5]]
 [1] "N" "T" "L" "H" "U" "G" "B" "Z" "H" "U" "Y" "O" "W" "L" "F" "P" "O" "O"
[[6]]
[1] "S" "S" "L" "M" "T" "R"

Mock data

# Function by A5C1D2H2I1M1N2O1R2T1
# https://stackoverflow.com/a/42734863/9406040
rstrings <- function(n = 5000) {
    a <- do.call(paste0, replicate(5, sample(LETTERS, n, TRUE), FALSE))
    paste0(a, sprintf("%04d", sample(9999, n, TRUE)), sample(LETTERS, n, TRUE))
}


df <- data.frame(a = paste(rstrings(100), rstrings(100), 
                     rstrings(100)),
                 b = rstrings(100))

> head(df)
                                 a          b
1 PWYVL8045X YEEVT9271X OOYAW3194P QJOJP3673S
2 MESIA1348Y JUMVW0263A PUIAX6901K TRHIS9952I
3 NTLHU1254G BZHUY6075O WLFPO4360O SSLMT4848R
4 XIWRV0967X ERMLU3214U TNRSO3996F IJPTV3142Z
5 ESEKQ7976U RDDDK5322V ZZEJC7637W IBAJI6831N
6 PVDBQ3212K ZXDYV5256Z RVTPH3724W HTYYK5351R

Answer 2

I had the similar problem like you mentioned. The following code may be useful.

However, if you use the keywords_phrases function in the same package, you could use txt_recode_ngram function to do the similar thing.

library(data.table)
library(dplyr)
library(magrittr)
library(udpipe)
data("brussels_reviews_anno")
x <- brussels_reviews_anno
x <- as.data.table(x)
x <- subset(x, xpos %in% c("NN", "VB", "IN", "JJ"))
x <- x[, cooccurrence(lemma, order = FALSE), by = list(doc_id)]
x <- x %>%
  group_by(doc_id) %>%
  mutate(keywords = paste(term1, term2)) %>%
  summarize(keywords = paste(keywords, collapse = ", "))