With an eye on non-repudiation, how can one truly ensure that the
document viewer, or signing application, or both have not been
replaced by malicious code?
The security of the user machine is essential for everything concern security. If you can't trust your operating system and installed applications, then every signature mechanism could be broken or tricked.
Even if the binaries were signed, there doesn't seem to be a way to
prevent the code from being modified in memory. Even if there way,
couldn't the attacker simply modify both the viewer and the signing
application?
Yes he can.
The security of your computer could be evaluated considering the weakest of your security mechanism. Build up strong cryptographic effort, isn't useful at all if your operating system has been compromised.
EDIT:
So how do I ensure the operating system has not been compromised?
Well, you could never be absolutely certain that your OS is secure. There are many different things to take in consideration:
- how many people have access to that operating system?
- which operating system are you talking about?
- which tasks are performed by users of that OS?
- Is that OS connected to Internet? If yes, how is organized your network infrastructure?
Can I boot the operating system and my application off a read-only
drive?
I don't know about that. Maybe you can. But if someone can exploit a service running on your machine (for example a server application with root privileges), than read-only drive won't save you.
What do you recommend?
First of all try to evaluate the level of security that your system really need. Probably you can't achieve perfect security. Then try to get a good compromise.
Remember that security and usability could become a trade-off. If you enforce your security policy, usability of your system may decrease.
For what concern my personal advices:
- Use a open-source operating system. Otherwise you can't be secure of what's really happening inside your machine.
- Try to configure a restrictive firewall. Deny the network access to every service except which ones you really need.
- If your system is used by many people (specially if they don't know anything about security), try to teach them some security principles. Human are often the weakest part to exploit (es. social engineering).
- Install only open-source software or software that you really trust. Keep your system updated.