0

Using the example at x86 assembler: floating point compare, I wrote the following NASM code, which compares two doubles and selects the smaller of the two:

fld qword[a]
fld qword[b]
fcomip
fstp qword [float_temp_var] ; to clear stack
jge minmax1_next
movsd xmm0,[b]
movsd [minval],xmm0
jmp minmax1_out
minmax1_next:
movsd xmm0,[a]
movsd [minval],xmm0
minmax1_out:

The only problem is that the results are not consistent. Sometimes it identifies the smaller value correctly and sometimes it does not. According to the post at the url above, I don't need to store or pop the flags register, so I don't know what I've done wrong. Research hasn't revealed the problem.

Michael Petch
  • 46,082
  • 8
  • 107
  • 198
RTC222
  • 2,025
  • 1
  • 20
  • 53
  • Did you read the question you linked to for code or did you read the answer? The question you linked to had a bug. They used signed comparisons which is a mistake you made here. – Michael Petch May 05 '18 at 21:32
  • 2
    I'm going to go out on a limb and flag it as a duplicate of the question you linked to, given that it's the exact same issue with the same solution. – Jeroen Mostert May 05 '18 at 21:34
  • Since you are already using SIMD (SSE2) instructions any reason you don't use [SIMD comparison](http://www.felixcloutier.com/x86/CMPSD.html) and stop using the x87 FPU instructions? – Michael Petch May 05 '18 at 21:43
  • @Michael Petch, thanks. I will change to unsigned comparisons, and I will also try the SIMD comparison. I would rather use SIMD than x87. I don't think I duplicate the question linked to because its solution doesn't work here, and you say it had the same error (unsigned vs signed). I will post the results as soon as I have them available. Many thanks again. – RTC222 May 05 '18 at 21:57
  • 1
    Hint: [What is the instruction that gives branchless FP min and max on x86?](//stackoverflow.com/q/40196817): you can much more easily do this with `movsd xmm0, [a]` / `minsd xmm0, [b]`. (Or load both into regs so you can use `minsd` and `maxsd`.) Or use `comisd xmm0, xmm1` to set flags the same way `fcomi` does. – Peter Cordes May 05 '18 at 21:58
  • I'm changing that to unsigned now. I updated the edit above to clarify. – RTC222 May 05 '18 at 22:05
  • @RTC222: You're using `jge` instead of `jae`. That's what the answer on the linked duplicate says you need, which is why I agreed with the other two close voters and closed this question as a duplicate. Re-read it carefully and double-check your assumptions about what works and what doesn't. – Peter Cordes May 05 '18 at 22:07

1 Answers1

0

You need to use jae instead of jge. See this question Difference between JA and JG in assembly for the difference between these two instructions. Although floating point comparison is signed, it sets the flag in the same way as an unsigned comparison, so you need to use the unsigned branch instructions.

prl
  • 11,716
  • 2
  • 13
  • 31
  • The answer given by both you and Michael Petch above were the solution. The problem is signed vs unsigned. Many thanks to all who replied. – RTC222 May 05 '18 at 22:14