Replace string in php which uses $ without referencing a variable

Question

I worded the title very badly, but you'll understand what I mean.

Here's a piece of code:

$value = $row['value'];
$fieldValue = str_replace("$value", $value, $existsValue);

I want to replace all instances of the literal "$value" within a string with the value of the variable $value.

So for example the string might be "Hello, my name is $value."

So I want to replace $value with the real value that is stored in the variable.