1

I am doing a personal project with a city data, I am stuck at where I have to normalize the data in order to apply the k-means clustering. I did some digging around and found this formula here:

enter image description here Here is the reproducible example

options(scipen = 999)
tract_per_complaints <- data.frame(
                      tract= c(7354, 5036, 17406, 5675, 2354),
                      fire= c(32, 0, 3, 78, 9),
                      flood= c(1, 25, 6, 8,7),
                      noise= c(56, 67, 0, 0, 1), 
                      total_complaints_at_this_tract= c(89, 92, 9, 86, 17),
                      population_at_this_tract= c(34134, 103849, 345300, 43535, 2143767)
                    )

#   tract fire flood noise total_complaints_at_this_tract population_at_this_tract
#1  7354   32     1    56                            89                    34134
#2  5036    0    25    67                            92                   103849
#3 17406    3     6     0                             9                   345300
#4  5675   78     8     0                            86                    43535
#5  2354    9     7     1                            17                  2143767

So I wanted to apply the formula above to cells with complaints,tract_per_complaints[ ,2:4] only. So I had made a nested for loop which didn't work. Then applied the sweep() function like so:

sweep(tract_per_complaints[ ,c(2:4)], 1, FUN="/", tract_per_complaints$population_at_this_tract)

#wrong output
#            fire         flood           noise
#1 0.000937481690 0.00002929630 0.0016405929572
#2 0.000000000000 0.00024073414 0.0006451675028
#3 0.000008688097 0.00001737619 0.0000000000000
#4 0.001791661881 0.00018376019 0.0000000000000
#5 0.000004198217 0.00000326528 0.0000004664686

Unfortunately this was my best attempt with the problem. What can I try next? I have looked at these:

Normalization of only one column in data frame

Normalizing columns in R according to a formula

Need help implementing a function in R

How to use the 'sweep' function

halfer
  • 19,824
  • 17
  • 99
  • 186
CaseebRamos
  • 684
  • 3
  • 18

1 Answers1

1

I'm not sure sweep is the right tool here, as you don't have a summary statistic to sweep out yet. You can do this with apply:

apply(tract_per_complaints[ ,c(2:4)], 2, 
      FUN=function(v, p) { v * p / sum(v * p)}, 
      p = tract_per_complaints$population_at_this_tract)

Output:

          fire       flood     noise
[1,] 0.04401224 0.001701866 0.1735655
[2,] 0.00000000 0.129443578 0.6317793
[3,] 0.04174017 0.103296596 0.0000000
[4,] 0.13682627 0.017364677 0.0000000
[5,] 0.77742131 0.748193282 0.1946551

As an alternative, you could also look at dplyr::mutate_at.

Marius
  • 58,213
  • 16
  • 107
  • 105
  • This is 100X better than what I had done with my for loop. Thank you Marius! Appreciate your help buddy, and I will definitely look at the mutate_at function :) – CaseebRamos Feb 27 '20 at 02:27
  • Hello, I got it to work with dplyr too: `func <-function(v) { p = tract_per_complaints$population_at_this_tract; v * p / sum(v * p) }` then `tract_per_complaints <- dplyr::mutate_at(tract_per_complaints, .vars = vars(fire, flood, noise), func)` Thank you again for your great help friend! – CaseebRamos Feb 27 '20 at 02:40