I came across this recently and the issue is to do with calling a function within the second argument of the replace function. The $1 is only valid if called directly in a string literal, not as a string argument to a function.
Another issue is that $('title').text().replace(regexp, (parseInt("$1")+1).toString())
will return a string. You need to pass the value you want to assign the text to as a function argument of the text()
function like you have done in your else block.
Try this:
title = $('title').text();
regexp = /^(\d+)/
if (numToReplace = title.match(regexp))
$(title).text(title.replace(regexp, (parseInt(numToReplace[1])+1).toString()))
else
$('title').text('1 New Messages | Default')
And a JSFiddle: http://jsfiddle.net/KFW4G/