I don't think that they should have to label themselves with a big sticker saying "Contains GMO Products" or whatnot, but I do think that all food (GMO or traditional) should be labelled with the name of the farm they come from, or same way of letting a consumer trace that information.
That way those interested can search for that farm and know under what conditions it grows its food. And that can be any number of things. Are they GMO crops, do they use artificial fertilisers (or traditional manures, or none), pesticides, are they free-range livestock or battery farmed, to what extent do they selectively breed their livestock, their employment (are they fairtrade for example).
Of course it's not suggesting that using or not using any of those things makes a product better or worse, but it lets a consumer (with their own personal beliefs, opinions and tastes) choose to support the farm they want, and know what exactly they're getting.
Besides the cost of putting that label on their product, I don't see a downside for producers, and I see it as beneficial to the consumer, without fearmongering.