The only downside to picking up junk for crafting in Fallout 4, for me, is that whenever I see a unique or fancy looking item that I want in my house, I pick it up and then forget that I picked it up.
I did the same in Fallout 3/NV, however when I eventually got home and emptied my inventory, it would be in my Junk section, with very little else, because I didn't pick up all the random stuff just scattered about.
So I knew it was important, so I dropped it and decorated.
Now I'm in the habit of getting back home, and just pressing "Transfer all Junk" to one of the workbenches, and I've forgotten that I picked up that nice Crystal Decanter for my dining room table.
The one thing I suppose I would like is a "favourite" button for junk, so I could stamp it in my inventory, and it wouldn't be automatically transferred with other stuff.
That would mean I would have to stamp it right away when I pick it up, but I'm happy to do that.