MAGICAL GUIDE TO RUNNING SERVERS:
You want to allow about 128mb of RAM for the server itself, or 256mb for Bukkit (base). Add 32-64mb per player.
For each plugin you have, or each mod, add 32mb (aside from Forge, which doesn't seem to on its own effect memory usage at all). For very large mods that implement a lot of functionality (IC2, TFC, TC2, etc) you would probably want to add 64mb or even 128mb.
That will generally be enough, regardless of what anyone else tells you. Minecraft is only really memory hungry in that it needs to load the world - but if your server is running Vanilla or only slightly modded and uses over a gigabyte for less than 20 players, something is terribly, terribly wrong. Back in the days of Brohoster, we ran a vanilla server with 10 people on it for four hours with 192mb of RAM, and it ran pretty well. It slowed down when we all moved away from one another, due to needing to load more of the world, but when we were all clustered together it didn't lag at all. At least until we started TNTing the spawn, at which point it would stop for about five seconds and then everything would be gone.
tl;dr: RAM isn't really that important and odds are you have like 8-16GB of it anyway so go nuts
What's actually important when running a server is:
- CPU speed - the average desktop has this covered, but if you're running it on a 1.8GHz single core expect performance issues
- Disk speed - loading the world from a 5800RPM drive will be slow. From a 7200RPM drive, it will still be pretty slow. From a 10000RPM or 15000RPM drive, it will be alright, and an SSD will noticeably improve performance in almost all cases.
- Upload speed - So you think that a server just needs a gigabyte of memory? To do what? Load the world? How does the client find out about that world? If, like me, your upload speed is very slow, this is going to be the biggest obstacle. You could be running your server with 64GB of RAM on a 3GHz hex-core with a PCI-E SSD, but if you only have 5mbit upload, it's going to lag with ten people on it. Speedtest is your friend.
When you look at a Minecraft hosting provider, they'll often recommend a 512mb server for about 16 players. This is totally legitimate, as they have a 100mbit or gigabit uplink and SSDs to back that up, as well as often having private tunnels to reduce latency (less hops between you and the server)