Everyone knows that the fastest browser is Elinks (or links, lynx, w3m, etc...), of course. Hands down. It's text only though.
That might seem to be a huge downside, but I think I can make a case for having a text-based browser in the modern age. There are no popups, ads, or javascript. Every single webpage is very easy to read and there's very little eyestrain. You will never have to deal with webpages crashing. Surprisingly, a lot of stuff works without any pictures - I can check my email, read the news, browse wikipedia, and post on forums in a text based browser. The only viruses you will ever have to worry about are the ones that you download and run yourself - you don't have to worry about plugin exploits.
I generally keep one on hand as a fall-back browser. When my wireless connection is stuffty or I just need to look up a wikipedia article really quick and I don't want to deal with all the bullstuff on the web, I use Elinks.
Here is a picture:
http://elinks.cz/screenshots/google-news--2004-09-29.pngNote that elinks runs on linux only. I have a copy of ubuntu server that I run in a windowed virtual machine so that I can use elinks and a couple other applications on my laptop running windows. You can also run it using cygwin. Finally, there are a few text-based browsers for windows machines, w3m comes to mind. However, I think elinks handles frames, tables, and colors the best, and it also has mouse support so you can actually click in it if your terminal application supports mouse input.