Sounds like the "reaction disc" is missing. It normally live on the end of the booster piston.

Has the booster/master cylinder been pulled apart lately?

If it has, pull off the booster and shake it. If something sound like it's bouncing around in the booster, that's what it will be. Carefully pull the piston and rubber seal out of the booster housing, if there is no rubber disc on the flat end of the piston. shake the booster around more until it falls out the hole. Re-assemble the booster with the reaction disc stuck to the end of the piston with a light smear of grease.

Cheers,
Chris

ps. A hole in the booster diaphragm normally results in a rise in idle speed, these are the only 2 things that go wrong with a booster.