I'm gonna throw this out here because it's something that I noticed;
If you use any high-resolution textures period, they will eat your framerate. I normally have a texture pack chock full of high res textures (x256 water/lava, various mobs, etc.) and they dragged my performance down. I switched to default and it's glorious, especially with Optifine.
Now the only changes I have are my simple terrain.png modifications.