I am working with the R programming language. Suppose I have the following data ("my_data"):
set.seed(123)
num_var_1 <- rnorm(1000, 10, 1)
num_var_2 <- rnorm(1000, 10, 5)
num_var_3 <- rnorm(1000, 10, 10)
num_var_4 <- rnorm(1000, 10, 10)
num_var_5 <- rnorm(1000, 10, 10)
factor_1 <- c("A","B", "C")
factor_2 <- c("AA","BB", "CC")
factor_3 <- c("AAA","BBB", "CCC", "DDD")
factor_4 <- c("AAAA","BBBB", "CCCC", "DDDD", "EEEE")
factor_5 <- c("AAAAA","BBBBB", "CCCCC", "DDDDD", "EEEEE", "FFFFFF")
factor_var_1 <- as.factor(sample(factor_1, 1000, replace=TRUE, prob=c(0.3, 0.5, 0.2)))
factor_var_2 <- as.factor(sample(factor_2, 1000, replace=TRUE, prob=c(0.5, 0.3, 0.2)))
factor_var_3 <- as.factor(sample(factor_3, 1000, replace=TRUE, prob=c(0.5, 0.2, 0.2, 0.1)))
factor_var_4 <- as.factor(sample(factor_4, 1000, replace=TRUE, prob=c(0.5, 0.2, 0.1, 0.1, 0.1)))
factor_var_5 <- as.factor(sample(factor_4, 1000, replace=TRUE, prob=c(0.3, 0.2, 0.1, 0.1, 0.1)))
id = 1:1000
my_data = data.frame(id,num_var_1, num_var_2, num_var_3, num_var_4, num_var_5, factor_var_1, factor_var_2, factor_var_3, factor_var_4, factor_var_5)
> head(my_data)
id num_var_1 num_var_2 num_var_3 num_var_4 num_var_5 factor_var_1 factor_var_2 factor_var_3 factor_var_4 factor_var_5
1 1 9.439524 5.021006 4.883963 8.496925 11.965498 B AA AAA CCCC AAAA
2 2 9.769823 4.800225 12.369379 6.722429 16.501132 B AA AAA AAAA AAAA
3 3 11.558708 9.910099 4.584108 -4.481653 16.710042 C AA BBB AAAA CCCC
4 4 10.070508 9.339124 22.192276 3.027154 -2.841578 B CC DDD BBBB AAAA
5 5 10.129288 -2.746714 11.741359 35.984902 -10.261096 B AA AAA DDDD DDDD
6 6 11.715065 15.202867 3.847317 9.625850 32.053261 B AA CCC BBBB EEEE
My Question: Given the above dataset, I am trying to create a function that (repeatedly) removes random rows from the above dataset in the following way:
Step 1: The dataset has 10 variables - in Step 1, randomly select "n" of these variables ("n" has to be less than 10).
Step 2: For the above "n" variables, if they are "factor", randomly select a subset (of size "m") of the levels for each of these factor variables. For each of the non-factor variables, split them randomly at a point between their minimum and their maximum (call this point "p").
Step 3: Generate a random number between 0 and 1 (call this "r").
Step 4: Select all rows identified in Step 2. For these rows, consider the columns that were not used in the logic condition. For these columns, there is a "r" percent probability that any element in these rows can be replaced with 0.
Step 5: Repeat Step 1 - Step 4 for 10 times.
As an example, this would look like this:
Step 1: Suppose n is randomly chosen as 4. 4 random variables are chosen : num_var_2, num_var_5, factor_var_3, factor_var_4
Step 2: For num_var_2, a point at 7 is chosen. For num_var_5, a point at 19 is chosen. For factor_var_3, 2 are chosen levels : "BBB" and "CCC". For factor_var_4, 3 levels "AAAA", "DDDD", "EEEE".
Step 3: A random number of 0.25 is chosen
Step 4:
SELECT * FROM my_table WHERE num_var_2 >7 & num_var_5 > 19 & factor_var_3 = "BBB, CCC" & factor_var_4 = "AAAA, DDDD, EEEE". For each row in the unselected columns (num_var_1, num_var_3, num_var_4, factor_var_1, factor_var_2, factor_var_5), each element in that row now has a 25% chance of being replaced with 0.Step 5: Repeat Step 1 - Step 4, 10 times. At some point, it is possible that a row will be selected that has already been replaced with a 0 in the past. This will make no difference as 0 replaced with 0 is still 0.
Can someone please show me how to write a function that does this?
Currently, I am trying to do this manually:
# 4 variables are selected
n = sample.int(10, 1)
[1] 4
# which 4 variables are selected (each number corresponds to their position):
sample.int(10, length(n))
[1] 6 2 1 4
num_var_1
num_var_2
num_var_4
factor_var_1
#select random points for the continuous variables
p1 <- runif(1, min(num_var_1), max(num_var_1))
p2 <- runif(1, min(num_var_2), max(num_var_2))
p4 <- runif(1, min(num_var_4), max(num_var_4))
#select random factor levels for the factor variable
nlevel = nlevels(factor_var_1)
nlevels = sample.int(nlevel, 1)
[1] 2
sample(factor_1, nlevels, replace=TRUE, prob=c(0.3, 0.5, 0.2))
[1] "A" "B"
#generate random probability number
r = runif(1,0,1)
[1] 0.4514667
#identify rows matching the above condition
identified_rows = my_data[which(my_data$num_var_1 > p1 & my_data$num_var_2 > p2 & my_data$num_var_4 > p4 & my_data$factor_var_1 %in% c("A", "B")), ]
> identified_rows
id num_var_1 num_var_2 num_var_3 num_var_4 num_var_5 factor_var_1 factor_var_2 factor_var_3 factor_var_4 factor_var_5
208 208 9.405383 15.53998 4.348425 29.87149 23.46945 B CC BBB DDDD DDDD
589 589 10.582991 18.84683 5.437036 31.53734 11.16494 B BB AAA BBBB CCCC
Now, for row 208, there is a 0.4514667 probability that the value in any of the remaining 6 columns (num_var_3, num_var_5, factor_var_2, factor_var_3, factor_var_4, factor_var_5) will be replaced 0. For row 589, there is a 0.4514667 probability that the value in any of the 6 remaining columns (num_var_3, num_var_5, factor_var_2, factor_var_3, factor_var_4, factor_var_5) will be replaced with 0.
After this, I would again repeat this entire process another 9 times.
This is a very long way to do this - can someone please help me write a function that will make this faster (e.g. repeat this 100 times)?
Thanks!
