I should think it has to be the case if copper is being scraped off the base of the bullet. I agree too that I've never seen a case that was chamfered properly scarf up the bullet base? That said the steel wool idea might work well but be sure none of the particles of steel wool get into your die or it'll scratch it. I sometimes use steel wool to remove the carbon from case necks but always brush inside the case necks after.
Finally and most importantly, if something is barking up the edge of the base of the bullet it will most definitely mess up your accuracy. You can tweak the nose a considerable amount with little effect but the edge at the base of the bullet is an altogether different deal. If it's scarfed it screws things up a lot.
Just my 2cents

.