You don't actually even need to use apply. You can simply do an assignment:
df$col4 <- names(df)[max.col(df,ties.method="first")]
This also takes care of your which column to choose in a tie issue. To show how this works we can modify the original data slightly:
df <-read.table(header=T,text=" col1 col2 col3
10 20 20
20 10 10
30 10 10")
Note how for the first row col2 and col3 have the same value. Per your comment on using the alphabetically first column in a tie, we would want col2.
Running the above code we see it works as expected:
col1 col2 col3 col4
1 10 20 20 col2
2 20 10 10 col1
3 30 10 10 col1
Edit: In case you don't only have cols1 - cols3
in your dataset (there might be other variables you dont want to take the max of) a more complete solution would be to specify the columns you want to take the max of. Something like this:
Data:
#Note this has an extra id variable on it
df <-read.table(header=T,text=" col1 col2 col3 id
10 20 20 100
20 10 10 100
30 10 10 100")
Then you can run the code to see how it works:
#The columns we want to take the max of
cols <- c("col1","col2","col3")
df$col4 <- names(df)[names(df) %in% cols][max.col(df[,cols],ties.method="first")]
col1 col2 col3 id col4
1 10 20 20 100 col2
2 20 10 10 100 col1
3 30 10 10 100 col1