I'm not sure then. It might be something else altogether. But a wiped cam lobe or lobes wouldn't be as noticeable as you think, the lobes are lifting as usual except they may lift less meaning they are closing earlier or opening later, the loss of material off the lobe peak would mean the chattering will begin due to the excess space that has developed. I think you might get some popping as well due to the valve timing being out of order.

The only other things that come to mind are timing and the springs binding. But the timing wouldn't make a chattering noise although it would affect performance.

Spring binding could affect performance but not sure if it would be that audible.

Not sure dude. When you run it at idle are you absolutely positive the noise is coming from the rocker assembly?