You should use parentheses () to specify that you want to match either the left or right side of the pipe "|" symbol, rather than square brackets. Square brackets actually match character sets (i.e., [grapes] would match a single instance of either "g", "r", "a", "p", "e", or "s", while (grapes|apples) would match either "grapes" or "apples").
Also, another thing you're missing is an indication of "quantity". In other words, once you match a space (\s), how many spaces should it look for? In your example, it's only matching a single space character. You probably want to match as many consecutive spaces as exist on the left and right of the string. For that, you need to append * (match zero or more) or + (match one or more) immediately after the character \s.
So, to rewrite your regex:
var input = " first second ";
var trimmed = input.replace(/(^\s+|\s+$)/g, '');
console.log(trimmed);
You can copy and paste those lines into your JavaScript console to see that you get the desired output. The regex here literally says, "match either one or more space characters from the start of the string, or one or more space characters immediately followed by the end of a string." Then the replace function takes that match and replaces it with ''.