Factory, there are "stepped" washers with rubber glued to them to allow for a little bit of movement. You don't need them in there, and most delete them for a firmer shifter feel. I would just tweak the outer cable mount, so it points a bit more towards the arm. No more problem.
I might have put a washer behind them, but it's really not necessary.