How can divide a dataset based on percentage?

Question

I have a dataset like this

ID      var value
9442000 a   2.01
9442000 v   2.2
9442000 h   5.3
9442000 f   0.2
9442000 s   0.55
9442000 t   0.6
952001  d   0.22
952001  g   0.44
952001  g   0.44
952001  h   0.77
652115  a   4.66
652115  d   1.55
652115  s   2.55
652115  s   2.55

I want to separate this into two dataframes for calibration (75%) and validation (25%). Doing it for overall is easy, but I want to do it ID-wise. So basically, I want to ensure that 75% of EACH ID goes to calibration. For example, for ID 9442000, I want to put any four events (random) into calibration and 2 into validation dataframe.

Expected output:

*Calibration*
 ID var value
9442000 a   2.01
9442000 v   2.2
9442000 h   5.3
9442000 f   0.2
952001  d   0.22
952001  g   0.44
952001  g   0.44
652115  a   4.66
652115  d   1.55
652115  s   2.55

And

*validation*
ID  var value
9442000 s   0.55
9442000 t   0.6
952001  h   0.77
652115  s   2.55

score 3 · Answer 1 · answered Feb 23 '15 at 23:54

I thought you wanted a random sample and my original approach with ave went down in flames because the segments were too short, so I turned to tapply and unlist-ed the result

calib <- dat[ unlist( tapply(rownames(dat), dat$ID, 
                                        FUN=function(x) sample(x, .75*length(x)))), ]
valid <- dat[ !rownames(dat) %in% rownames(calib) , ]
#---------------
> str(calib)
'data.frame':   10 obs. of  3 variables:
 $ ID   : int  652115 652115 652115 952001 952001 952001 9442000 9442000 9442000 9442000
 $ var  : Factor w/ 8 levels "a","d","f","g",..: 1 6 2 4 2 4 7 8 1 6
 $ value: num  4.66 2.55 1.55 0.44 0.22 0.44 0.6 2.2 2.01 0.55
> str(valid)
'data.frame':   4 obs. of  3 variables:
 $ ID   : int  9442000 9442000 952001 652115
 $ var  : Factor w/ 8 levels "a","d","f","g",..: 5 3 5 6
 $ value: num  5.3 0.2 0.77 2.55
>

Oh yes, I need a random sample. Sorry, I forgot to mention it. This works as well though. Could you post your random sample code as well? — maximusdooku, Feb 23 '15 at 23:57
That was my random sample code. (Neal's code was not random.) — IRTFM, Feb 24 '15 at 00:15

score 2 · Accepted Answer · answered Feb 23 '15 at 23:43

First, define a variable for which group it goes in, then use split:

> df$test <- ave(df$ID,df$ID,FUN=function(X) seq_along(X) %% 4 == 1  )
> 
> split(df, df$test)
$`0`
        ID var value test
2  9442000   v  2.20    0
3  9442000   h  5.30    0
4  9442000   f  0.20    0
6  9442000   t  0.60    0
8   952001   g  0.44    0
9   952001   g  0.44    0
10  952001   h  0.77    0
12  652115   d  1.55    0
13  652115   s  2.55    0
14  652115   s  2.55    0

$`1`
        ID var value test
1  9442000   a  2.01    1
5  9442000   s  0.55    1
7   952001   d  0.22    1
11  652115   a  4.66    1

To randomize the sample, you can use `sample(length(X))` instead of `seq_along(X)` in the function. — Neal Fultz, Feb 25 '15 at 00:56

How can divide a dataset based on percentage?

2 Answers2

Linked