Sounds like you are connecting it to manifold vacuum & not ported vacuum. On the other hand, if it really is connected to the right fitting on the carb, then it would seem that the throttle plate is open far enough for the vacuum port to see manifold vacuum & start sucking in some advance, even at idle.

See if you can use a vacuum guage to help tune the carb & stabilise the idle. The higher the vacuum at a given engine speed, the better, as this indicates a 'best mixture' condition. I use a big old Redex tune up one from the 60's about 100mm [4"] in diameter. It sure helps my tune ups.