Short answer: yes, it's better to declare any variable that's purely internal to your controller as a javascript variable ("var") rather than adding it to an Angular scope.
Angular scope objects provide many nice features that allow them to act as the model in an model-view-*(MV*) architecture (for instance data binding). Or said another way, as the Angular guide to scopes says "Scope is the glue between application controller and the view".
It's best to only put objects that you need in your model, that you need bound both to the DOM and your controller/services/..., on the scope as those features do come at a performance cost, as you point out. And it can also be confusing to other's who read your code if your scopes are "cluttered" with variables that aren't actually part of your model.
Here's the specific features of scopes from Angular scope docs:
Scopes provide APIs ($watch) to observe model mutations.
Scopes provide APIs ($apply) to propagate any model changes through
the system into the view from outside of the "Angular realm"
(controllers, services, Angular event handlers).
Scopes can be nested to limit access to the properties of application
components while providing access to shared model properties. Nested
scopes are either "child scopes" or "isolate scopes". A "child scope"
(prototypically) inherits properties from its parent scope. An
"isolate scope" does not. See isolated scopes for more information.
Scopes provide context against which expressions are evaluated. For
example {{username}} expression is meaningless, unless it is evaluated
against a specific scope which defines the username property.