tl;dr
Transforms are relative to the reference box which is set by the transform-box
property which defaults to border-box
. The border box is bounded by the border. As a result the scale transformation will scale everything within and including the border (border, padding, content box).
Long answer
I'm going to start my answer off with the following example:
button {
padding: 10px;
margin: 10px;
}
.scaled {
transform: scale(2);
}
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width">
</head>
<body>
<button class="default">Button</button>
<button class="scaled">Button</button>
</body>
</html>
There are two buttons. Both are given 10px padding and 10px margin (as well as default browser styles). The second is scaled by a factor of 2.
In Firefox 58 the first button renders with the following properties:

We'll focus on the properties and values of the horizontal axis:
34.7167 // Content box width
10 // Padding left
10 // Padding right
7 // Border left
7 + // Border right
------------
68.7167
------------
10 // Margin left
10 // Margin right
If you total everything but the margin you'll get the element width. The margin is excluded. This is due to the browser defining the element box as everything inside and including the border. This behaviour is specified by the box-sizing
rule set to border-box
.
So how does this play into the scale
transform function not scaling the margin?
Transformations occur relative to a reference box. The reference box is defined by the transform-box
property which by default has the value border-box
. As a result the border, padding and content box dimensions will be scaled by 2 while the margin is not.
This behaviour can be observed in the second button, which is scaled:

Again, looking at the properties and values of the horizontal axis:
34.7167 * 2 = 69.4334 // Content box width
10 * 2 = 20 // Padding left
10 * 2 = 20 // Padding right
7 * 2 = 14 // Border left
7 * 2 = 14 + // Border right
-----------------------------
68.7167 * 2 = 137.4334
-----------------------------
10 // Margin left
10 // Margin right
Further reading