so do I just grind my sword on stuff like grineeer or something
Yeah. You need to gain affinity and level up the sword.
You can do this by fighting enemies. You should earn a bit more if you get kills with the sword, but you'll pick up affinity for all your equipped items (that aren't maxed out) for any kills (although apparently kills with abilities will give affinity only to your warframe).
Also, open containers and you can find Affinity drops, usually about 100. It's not much, but every little bit helps.
Just keep doing missions and fighting enemies. Survival, or Defense will be good for you, since you should have lots of enemies attack you.
Also, if you get stealth kills back to back, they increase the affinity gain of your next kill by +100%. So, get 5 stealth kills, and the next kill will give you +500% affinity.