Have a look at the idle screw tip, and the seat (part of the carb body) which the tip sits in, either might damaged. The shape of the tip corresponds to the seat that it sits within, if the tip is damaged it will not seat or seal properly (or do it's job). Sometimes people screw them in so tight that they damage the seat so getting a precise mix is difficult or impossible.

Other than that not sure what else it could be. 50rpm fluctuation is bearable though? Until you can fix the issue.