DEP, NX bit and others
This is preliminary, but I have almost-decided to do some research about what is probably most commonly known as DEP (Data Execution Prevention).
I'll point you to the Wikipedia entry for more information (and probably more technically accurate), but I'll give you a dumbed-down personalized version here.
Data Execution Prevention was introduced in Windows XP Service Pack 2, my favorite service pack btw, although the idea of DEP has been around for much longer than that. The idea of DEP is to stop programs, malicious or not, from executing code from a non-executable region of memory such as the default heap, stack or some memory pool pages; it can also prevent the program from writing executable code to these areas. A malicious program can take advantage of the way that an Operating System (Windows) handles exceptions improperly and then call up some code running in those special areas such as the default heap or stack and some special memory pools. DEP detects these anomalies and kills the process via standard operating system murdering tools.
The famous Blaster and Sasser worms used this method to inject code into (probably the stack) memory and executed it. From the small amount of research that I have done already, it appears as if the Blaster work took advantage of the RPC (Remote procedure call) in Windows. The same story goes for the Sasser worm, which was the result of a buffer overflow in the LSASS (Local Security Authority Subsystem Service).
A simple explanation of what a buffer overflow is this: A legitimate program will have a fixed size of temporary storage in memory (a buffer). This buffer is filled and emptied when the program needs to use it, a buffer overflow happens when more data that was expected enters the buffer and flows in to an adjacent memory space. This extra information could be a malicious program that is pushed into an executable area, then executed shortly after.
For an analogy, you have a company owned storage unit and you are constantly filling it with things you want to store and taking things out you are finished storing. You are also allowing a few employees to put things inside the storage unit when you need them to. One day a malicious employee puts more things in the storage unit than you have room for, and instead of it not fitting (like would happen in real life) it spills over into the next guys unit. Since computers are kind of dumb and obedient, they only do the things we tell them to, no matter how crazy it sounds, I'll say that the guy who owns the unit next to yours is very gullible and will do anything you tell him to. The things that spill over into his unit are some plans that tell him to immediately give your malicious employee every penny he has. Then he does it.
A lot of the time a buffer overflow will cause the original program to crash, or another program whose memory contents you've just overwritten, or both. DEP and its variants are supposed to stop this from happening, and while it isn't a perfect option it is better than nothing.
If my opinion changes over time or I find that I am wrong in anything I posted here, I'll try to come back and fix it, but I don't make any promises. Although, I've tried to make this as accurate as possible by doing some exploratory research.
As for some references to find out more:
http://support.microsoft.com/kb/875352 (A detailed description of the Data Execution Prevention (DEP) feature in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003)
http://www.watchguard.com/infocenter/editorial/135136.asp (Foundations: What Are Buffer Overflows?)
http://en.wikipedia.org/wiki/Buffer_overflow
http://en.wikipedia.org/wiki/Sasser_(computer_worm)
http://en.wikipedia.org/wiki/Blaster_worm
http://en.wikipedia.org/wiki/NX_bit
http://en.wikipedia.org/wiki/Exception_handling
http://en.wikipedia.org/wiki/Memory_pool
http://en.wikipedia.org/wiki/Stack_(data_structure)