Security and prevention of vulnerability in a computer depends on the cooperation of several components: (1) the computer hardware, (2) the operating system, and (3) the application programs. Since security and prevention of vulnerability require additional time and money most uses of computers, especially personal computers back in the 1980s and 1990s had a fairly open architecture.
For instance, an IBM PC or clone using an Intel 8080 or 8086 would be running the MSDOS operating system or some variant. Many applications would modify various areas and tables within the BIOS area as well as the MSDOS operating system. This was possible because the architecture of the CPU presented memory as a single shared resource and the operating system, which was quite simple and straightforward, provided minimal security and protection against vulnerabilities. The hardware did not support virtual memory or address translation so all applications running had to be well behaved and exercise self restraint. There were accepted guidelines about how to chain interrupt handlers so that multiple applications would be able to coexist in the same physical memory area and be notified of events or interrupts for which they were looking.
This Wikpedia article, x86 Memory Segmentation, describes the evolution, from a memory addressing perspective, of the Intel x86 processor family from the 8086 to later versions such as the 80286 which began to introduce hardware support for address translation.
However modern CPUs of any complexity provide a multitude of mechanisms for security to support virtual memory and the management of virtual memory isolating applications. With the original IBM PC and its clones, all applications shared the the physical memory. Modern CPUs provide a virtual memory space for each application with the CPU translating individual memory accesses within the virtual memory space into actual physical memory locations. The result is that applications run in their own virtual memory space and it requires operating system mechanisms to allow cooperating applications, each running in its own virtual memory space, to have a shared memory region.
The goal of these mechanisms is to isolate individual applications as much as possible so that if they do something foolish, the only application that is affected is the application itself and not other applications that may be running. The most important effort is to ensure that applications can not affect the operating system so that the operating system stays up and running even if an individual application crashes.
There is nothing to stop an individual application from modifying itself. The idea is that if an application wants to modify itself then it should be allowed to do so so long as the consequences are limited to the application itself. In some cases regions of virtual memory may be marked as read only which would cause an exception if an attempt is made to modify the data or program code in the read only area.
Another use of read only areas is from this paper by Thompson, UNIX Implementation, in which the read only area or text section in UNIX terms, is a loaded once and shared between multiple processes. This is an interesting, historical paper providing a thumbnail sketch of one of the most influential monolithic (not kernel based) operating systems around, UNIX, which became quite successful and which was imitated by a number of other operating systems. This stackoverflow question/answer, Unix/Linux Loader Process, provides some additional details about the loader and how a process loads.
However anyone who has dealt with a buffer overrun problem has seen that an application can modify its own memory area, usually with catastrophic results on the application.
Take a look at the Intel 64 and IA-32 Architectures manuals for details about the Intel architecture.
Malware and viruses are applications that use various vulnerabilities and exploits within operating systems. Stuxnet is a famous example (see IEEE Spectrum The Story of Stuxnet as well as this Wired article How Digital Dectives Deciphered Stuxnet, the Most Menacing Malware in History) as is Flame, both of which seem to be cyberwarfare applications.
Also see this Wired.com article on a USB malware, Why the Security of USB is Fundamentally Broken which outlines a malware attack through a modification of USB firmware. There is a link to some of the work being done by the NSA in the United States.
Here is a simple example showing what can be done in C using C-style casting. This example will compile with Visual Studio 2005. If you run this in the debugger, you will see that it will hit the int 3 instruction to cause a break point hit.
int jjFunc (void)
{
int i = 3;
return i;
}
// 0xCC is an int 3 instruction for causing a break point.
unsigned char xxFunc [] = {0xcc,0,0,0,0};
int _tmain(int argc, _TCHAR* argv[])
{
int (*xx)(void) = jjFunc;
char *p = (char *)xx;
// point to my tiny program in memory
xx = (int (*)(void))&xxFunc[0];
xx ();
jjFunc ();
*p = 0;
jjFunc();
return 0;
}
