It looks like it's impossible to prevent determined attackers from modifying one's process code/data. I'm hoping that its at least possible to detect such tampering.
Under Windows, is it possible to listen for DLL injections, WriteProcessMemory and CreateRemoteThread into the current process?
Under Linux, is it possible to listen for LD_PRELOAD and the DR rootkit?
with some really involved code you could be able to detect those... It all depends on how determined the attacker is... IF they are really determined then they would use some rootkit approach - in that case your app can do nothing about it (no detection, no stopping as long as the attackers know what they are doing)...
another approach could be to try to do some hashing of your segments in memory while running but that would account for some snake oil since the hashing code itself would present an entry point to circumvent this method.
Executing your code inside a self-built VM which in turn communicates with the rest of the system through a hypervisor... the hypervisor has to be made the boot loader for the system of course so that the OS is just a "child" of your hypervisor... should do the trick...although you would have to write all that yourself and make sure it has no exploitable weakness (pretty sure noone can do that for such a complex piece of software)...
not sure what you are up against but as long as the HW+SW your code is running on is not directly under your full control there is always a way to do the things you mention and with a bit of planning avoid detection too...
OR is this "only" about protection from software piracy/reversing ? IF so then there are some measures, even some 100% secure ones though it all is about balance of security versus usability...
Huh, how do you tell if LD_PRELOAD is malicious or not? What about ptrace? /dev/[k]mem? What about when one process plants a malicious plugin or something similar in the folder of another process's config directory? What about shared memory / IPC tampering?
First of all, this is a waste of time and complete nonsense to actually sell as a legitimate product. Not sure what you're trying to do. If it's antivirus, anticheat for a game, or DRM, then this is futile. The only thing you can do is run the process as another user, preventing other processes from modifying it in the first place. If that's not good enough, too bad, Linux isn't a secure operating system and never will be.
In theory, it's impossible to detect a process's memory being tampered with. In practice, it depends on how you define detect, and what kind of false positives and false negatives you care about.
If you know the normal behavior of a program is not to modify itself, you know exactly what segments of memory are meant to be static, and you know that there aren't any legitimate 3rd party programs on your PC that tamper with the program, then you may be able to detect memory tampering pretty easily.
The most general solution is to hook the OSs interprocess memory modification mechanisms like you said. This works as long as the enemy process doesn't have enough privileges to remove your hook or make certain OS calls that bypass your hook.
You can also just scan the entire process over and over, checksumming the memory using a secure hashing algorithm. Then again if the enemy process has privileges to modify your scanner, you lose.
So yeah, if the process hasn't the privileges to subvert your scanner, why would it have the privileges to modify the process you care about? Sounds like antivirus/anticheat/DRM to me.
