Phil Zimmerman once asked 'Why don't you send your paper mail on postcards?' Would you send a love letter or will on a postcard, in an unsealed envelope? It may seem rediculous, sure, but this is exactly what people are doing every day. In fact, with email, this is exactly what we're doing with our everyday communications.
Most people are sending around email on the web that is horribly insecure. You may, incorrectly assume that since you have to log in with a password to read your email, and I have to log in to read the email you sent me, that this is a secure channel. Would you care if someone started reading your email, without your knowledge or consent? This writeup, then, is for you. It is mostly an introduction to cryptographic concepts for non-mathematics enthusiasts.
What's wrong with sending an email from my system to yours if we both log in? The most obvious, people could sniff your password. Shoulder surfing, dictionary attacks or collecting all network traffic are common ways of figuring out someone's mail password. Even if your account was secure, however, that doesn't imply that your message was secure. When you send an email to me, it will traverse through anywhere between 5-30 routers/computers. Any one of these points along the way can read, and store if they wanted, the email in transit. If your traffic goes through a foreign router that logs data and invades your privacy, how would you properly protect your rights in that country, not being a citizen of it?
First, a bit of background. Public/Private keys are asymmetric, whatever is done with one can only be undone with the other. Therefore when you send information to me, you encrypt it with my public key. Only my private key can decrypt it. When I send you something, I do so with your public key. Also, I can sign something with my own private key. Since I am the only one that knows my private key, only I could have signed it, if you check and decrypt it with the public key (which anyone can do). The concept is simple.
So, if I have convinced you, what should you do? A good introduction to installing and setting up GPG on a windows machine is done rather well by Brendan Kidwell, with A Practical Introduction to GNU Privacy Guard. If you use Gmail, or your service provider's email, you can use Thunderbird, with the Enigmail extension. Setting up Thunderbird is the same as any other mail application, and here's help configuring enigmail.