Same problem in Gimp...
And when I look at your image, the displace map exactly overlays the source image, and you are using a negative displacement, two hints that you have fallen victim to a very common misconception.
The misconception
Displace maps would describe what happens to the source pixels; in other words, the map would tell how to move the matching source pixel. This is wrong, and there are at least two problems with this assumed behavior:
- this is ambiguous, what do you do when several source pixels are projected to the same result pixel?
- what happens to pixels that are not the destination of a source pixel?
The reality (*)
The map tells where the corresponding result pixel comes from (which is unambiguous, and every pixel in the result has a source...).
A corollary is that the displace map shape is usually different from the source, you cannot deduce it simply from the source.
Your problem
If you look at your map up close:
- The background is >127, so positive but your displacement is negative so pixels in the result are pixel to the left of this in the source
- The ellipse is <127, so negative but your displacement is negative so pixels in the result are pixel to the right of this in the source
So in:

- In the green area, result pixels are copied fro the tip of the green arrow so you get a copy shifted left of the ellipse
- in the blue area, result pixels are copied from the tip of the blue arrow, so are a copy of the background.
- In the red area, result pixels are copied from the tip of the red arrow so you get a shifted right copy of the ellipse.
So you get ellipse-background-ellipse and you see a gap.
If you want to stretch that part, the background and the ellipse should be moved by roughly the same amount, so the displace map cannot have a hard edge, in should on the contrary be uniform (or have a very slow gradient)(this is an application of the corollary above).
However....
Going back to the initial problem, what are you trying to achieve, because I don't think the displace map is a solution... (ask a new question if so).
(*) At least in Gimp, but I can reasonably assume that it works like other software