To debug a Web Start application in Linux, create a shell script ~/bin/javaws-debug.sh with the javaws invocation in debug mode as described above:
~/bin/javaws-debug.sh:
#!/bin/sh
export JAVAWS_TRACE_NATIVE=1
export JAVAWS_VM_ARGS="-Xdebug -Xnoagent -Djava.compiler=NONE
-Xrunjdwp:transport=dt_socket,address=8989,server=y,suspend=n"
javaws "$@"
Then, in your browser, choose that script as the application to invoke on jnlp files.
For example, in Firefox, go to Edit→Preferences→Applications, Content Type: Java Web Start, and choose "Use Other" in Action and pick the script from the "Select Helper Application" dialog. In Chrome, you need to alter Linux system settings. In KDE, go to System Settings→File Associations, Known Types: application:x-java-jnlp-file, add a new Application, pick ~/bin/javaws-debug.sh from the "Choose Application for application/x-java-jnlp-file" dialog.
Once your browser is configured, Java Web Start application will start using your wrapper, which will enable debugger to connect on port 8989.