Source code is the uncompiled (in this case C++) code to the engine, and Badspot can modify it however, and add any feature. But only if he wants to. So, to answer the question, "Can Blockland's rendering engine be expanded?" the question is yes. The question you should be asking, however, is does Badspot want to, and does he find the amount of work worth it, compared to any other features he might have planned?
Even if I bought a license to use torque engine he still can't give me the code?
Even if Badspot gave you the source code, you'd have to figure out your way around it, and probably code the changes yourself yourself, since I doubt exact resources exist for an outdated version. You'd then have to compile it, and distribute it to the entire community, a tricky task. Badspot has the auto updater to do this for him, you don't. And then when Badspot releases a new version, you'd have to do it all over again.
Plus I don't think you can even buy a license to the outdated version of TGE.