I had a similar problem and found a solution for myself, drawing in fact from your question for inspiration. As you suspect, it is possible that your gdbserver freezes when running on the same core because one of your application threads is using all the core's cycles and gdbserver is not allowed to run because its priority is too low.
For my particular needs, I am using gdb for an application employing realtime scheduling running on the local machine, and I don't mind if gdb runs on the same core, but I want to be able to debug my program with all the priorities of the application threads respected. For me, what made things work was expanding the gdb command to this more complex construction:
taskset -c 3 chrt 99 gdb
The chrt command added to your taskset command switches to the SCHED_RR policy and runs gdb at the specified priority. My threads being debugged run themselves with lesser priority, and so I assume they only run when gdb is not running.
I was having a problem before. I think, when I requested gdb to resume execution after gdb had suspended execution at a breakpoint or so, that one thread would start running before a higher-priority thread was resumed and thus it was not always the thread that I expected to be running that was in fact running. For me, the above command seems to fix everything -- I assume because the application threads can only run when gdb has finished everything it needs to do in order to resume the program. So, I guess the command line that would be applicable in your case if you wanted to try this out would be:
taskset -c 3 chrt 99 gdbserver :1234 ./app.out
Note: so this would lock gdbserver to a certain core, but likely your real time threads would (or probably could) be running at a lesser priority and so gdbserver would not freeze on you.