The reason for having so few characters is probably just because they don't want it to look repetitive. They can't really write and record unique dialog for tons of named characters, I mean, there's already a lot. So their only option, then, would be a bunch of generic NPCs, like "villager" or "farmer," which gets boring.
On this point, I seriously hope that in whatever next game they produce, they just invest a lot of time/money into lots and lots and lots of voice acting, by lots of different voice actors.
I know it's more of a cosmetic thing, and therefore less important than them to spend the time on working on a new engine, or gameplay mechanics, but I'm just so so bored of hearing the same 10 voices again and again and again.
Skyrim worked pretty well, because they had quite a few more voice actors.
Fallout 3 and New Vegas seemed to just have the same few voice actors. It got to the point where any time you saw an old man or a child you already knew what he was going to sound like.
And it wasn't even so much multiple npcs repeating the same lines, just having identical voice actors.
The Elder Scrolls Online however is, generally speaking, much better for Voice actors in my opinion.
At least as far as human based races go. (The Khajiit and Argonians in particular seem to have fewer voice actors, or if there are multiple ones, they're pulling very similar voices).
It really improves the immersion a lot when you don't notice that everyone sounds the same.
