Seems you can't really get them off without them breaking. It's almost like they are moulded on there.

Don't know if this helps but I shortened my handbrake for better gearstick clearance on my CA18 conversion and ended up
going to Bunnings and buying a small alloy draw knob which fitted perfectly and looks more solid than the original plastic one.
However, I did use a tap and die set to create a thread on the rod that the knob fits on.