December has 31 days so when you subtract 1 month, you get 31 November which doesn't exist, so it rolls over to 1 December.
You can test the date (day in month) to see if it's the same, and if not, set the date to 0 so it goes to the last day of the previous month.
Also, setDate modifies the Date object so no need to create a new one:
function subtractMonth(date, months) {
let d = date.getDate();
date.setMonth(date.getMonth() - months);
if (date.getDate() != d) {
date.setDate(0);
}
return date;
}
let d = new Date(2020, 11, 31); // 31 Dec 2020
console.log(subtractMonth(d, 1).toString()); // 30 Nov 2020
This has side effects so that sequentially subtracting 2 months may give a different result to subtracting 2 months in one go.
Also in regard to new Date('12/31/2020')
, see see Why does Date.parse give incorrect results?
PS
I answered this before I remembered that there were plenty of questions about adding months that also cover subtracting. So I marked this question as a duplicate and rather than delete this answer, left it for posterity.
If you wish to vote for an answer, please go to one of the duplicates and vote for an answer there. :-)