AlphDoti
If your engine is stock, then do not change the jet sizes away from stock sizes as these are a proven combination.
The primary throttle shaft is the one that will wear the most as it is in use ALL the time, while the secondary is used only occasionally.
Contrary to what some may say, the original Hitachi carb is a good, proven design carb, & it is ideal when used in stock applications.

If you have a vacuum leak, it would mess with the idle more than anything else. If this leak was at the "manifold to head" gasket, the leak would affect one cylinder more than the others & may even show up as a missfire at idle. Vacuum leaks often whistle if they are in the carb area. Check the tightness of all manifold & carb bolts.

A worn throttle shaft may well have the effect that you describe. So will a faulty accelerator pump, which you have checked. If the primary shaft is badly worn, you have four options.
Live with it
Fit a good used base or carb.
Have the base reconditioned with new bushings & shaft.
Fit a new base assembly.
The last three are my recomendations.

A blocked intermediate circuit will also give you "off idle" problems. This can be checked by simply screwing the idle speed screw progressively open.[not the idle mixture screw] What happens is that as the throttle slowly opens, & the rpm's rise, the idle circuit can no longer supply enough fuel, & the airflow is not yet enough to start to draw fuel from the main metering circuit.

There is a small orifice, just up from the idle port, just above the throttle plate when it is closed, & it starts to feed fuel once the throttle is open a little bit. This fills in the gap between idle speed fuel needs, & the time when the airflow is enough to draw fuel from the main metering circuit. If this intermediate circuit is blocked, the engine will lean out as you screw the idle speed adjustment in. & the engine will stumble. It may even stall. The accelerator pump will have no role to play when the throttle is opened as slowly as this, so it will not mask the symptoms.

Make sure that the engine is properly tuned, & all the idle settings are right on the money BEFORE doing this simple test. Also make absolutely sure that the fuel LEVEL is correct in the float bowl by inspecting the little window in the carb while the engine is idling. It should be in the middle of the window, level with the little dot marked in the glass. If it is not correct, then fix it before proceding further.

If this intermediate circuit is the problem, then pull the carb down & blow compressed air in the reverse direction through the port to clear it.

I hope this is of some help.
Chris