The answer to your first question is yes. It is recommended that you should absolutely change all your character encoding Attributes for all your HTML 5 documents.
This is because it is the current HTML5 Standard according to W3C. I would change all of the pages in any given site based on this reason alone as a standardization of all markup rendering is inevitable.
This can easily be done on any editor that has a find/replace feature. Simply use the feature to find in every document the term
<meta charset="windows-1252">
and replace it with
<meta charset="utf-8"/>
The UTF-8 character encoding should be able to handle your math characters but if it doesn't simply leave your original charset as is. And the rest of your pages with text only you will want to change to UTF-8. Here is W3Schools position on your char encoding.
The HTML5 specification encourages web developers to use the UTF-8
character set, which covers almost all of the characters and symbols
in the world!
--W3Schools.com
If size is an issue, again you will only leave those documents that have the special math character requirements with the original encoding if they don't render correctly and I don't think it will effect your browser load time enough to damage your SEO.
If you have many pages with math symbols then this could be a problem if your looking for a popular site or for business, if not the size is so small the problem with file size seems mute.
For the other documents you should still change the encoding for them as UTF-8 even if you have a BOM.
If you have a UTF-8 byte-order mark (BOM) at the start of your file
then recent browser versions other than Internet Explorer 10 or 11
will use that to determine that the encoding of your page is UTF-8. It
has a higher precedence than any other declaration, including the HTTP
header.
You could skip the meta encoding declaration if you have a BOM, but we
recommend that you keep it, since it helps people looking at the
source code to ascertain what the encoding of the page is.
--w3.org
Good luck and happy coding! :-)