What does the `order` argument mean in `tf.keras.utils.normalize()`?

Question

Consider the following code:

import numpy as np

A = np.array([[.8, .6], [.1, 0]])
B1 = tf.keras.utils.normalize(A, axis=0, order=1)
B2 = tf.keras.utils.normalize(A, axis=0, order=2)

print('A:')
print(A)
print('B1:')
print(B1)
print('B2:')
print(B2)

which returns

A:
[[0.8 0.6]
 [0.1 0. ]]
B1:
[[0.88888889 1.        ]
 [0.11111111 0.        ]]
B2:
[[0.99227788 1.        ]
 [0.12403473 0.        ]]

I understand how B1 is computed via order=1 such that each entry in A is divided by the sum of the elements in its column. For example, 0.8 becomes 0.8/(0.8+0.1) = 0.888. However, I just can't figure out how order=2 produces B2 nor can I find any documentation about it.

According to [this answer](https://stackoverflow.com/questions/52571752/what-is-the-purpose-of-keras-utils-normalize/52571871#52571871) it calls numpy [linalg.norm()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html) under the hood, you can see the "notes" section of that page for a description of the `order` parameter — G. Anderson, Sep 30 '19 at 20:02

Manoj Mohan · Accepted Answer · 2019-10-01T06:15:32.010

However, I just can't figure out how order=2 produces B2 nor can I find any documentation about it.

order=1 means L1 norm while order=2 means L2 norm. For L2 norm, You need to take the square root after summing the individual squares. Which elements to square depends on the axis.

Keras

A = np.array([[.8, .6], [.1, 0]])
B2 = tf.keras.utils.normalize(A, axis=0, order=2)
print(B2)

array([[0.99227788, 1.        ],
       [0.12403473, 0.        ]])

Manual

B2_manual = np.zeros((2,2))
B2_manual[0][0] = 0.8/np.sqrt(0.8 ** 2 + 0.1 ** 2)
B2_manual[1][0] = 0.1/np.sqrt(0.8 ** 2 + 0.1 ** 2)
B2_manual[0][1] = 0.6/np.sqrt(0.6 ** 2 + 0 ** 2)
B2_manual[1][1] =  0 /np.sqrt(0.6 ** 2 + 0 ** 2)
print(B2_manual)

array([[0.99227788, 1.        ],
       [0.12403473, 0.        ]])

You can look up the different types of Norm here: https://en.wikipedia.org/wiki/Norm_(mathematics) Worked examples: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html

score 0 · Answer 2 · answered Sep 30 '19 at 20:01

0

Order 1 normalize the input such that the sum of the absolute value of all element is 1 (L1 norm of the input equals 1). Order 2 normalize the input such that the sum of the squared value of all element is 1 (L2 norm of the input equals 1).

answered Sep 30 '19 at 20:01

valeriored83

141
2

score 0 · Answer 3 · answered Sep 30 '19 at 20:04

Passing order 2 in the order parameter, means you will be applying Tikhonov regularization commonly known as L2 or Ridge. L1 and L2 are different regularization techniques, both with pros and cons you can read in detail here in wikipedia and here in kaggle. The approach for L2 is to solve the standard equation for regresison, when calculating residual sum of squares adding an extra term λβTβ which is the square of the transposed matrix Beta (that's why it is called L2, because of the sqaure).

What does the `order` argument mean in `tf.keras.utils.normalize()`?

3 Answers3

Linked