Pigdog
Have you run a vacuum comparison between a stock engine [almost any engine] & a cammy dual sidedraught equiped engine?

My little 1200 GX would pull 18" Hg at cruise & a fair bit more at idle [900rpm] which is quite enough to make a booster perform at its best.

The hot engine will develop adequate vacuum when the throttle is closed at speed but the vacuum signal decays significantly as the RPM & the road speed decline. Some hot engines are flat out making 5" Hg at idle although these would be seen more as race engines.

If the booster in your car provides a significant benefit then it should remain there as the vacuum signal must be adequate, but I wonder just how much better it would work with even more vacuum as you would get from a stocker.

As I had said, the Weber equiped Valiant Chargers didn't have brake boosters because of the low vacuum signal, so Chrysler used a different master cylinder to try & address the problem.

For what it's worth, a small bore master cylinder will provide more hydraulic pressure at a given measure of input [pedal] pressure, but will have longer pedal travel in order to displace the required volume of brake fluid.
In order to get an unboosted system to work properly, the pedal travel may end up being excessive.

A larger bore cylinder will displace a greater volume of fluid for a given measure of pedal travel, but will generate less hydraulic pressure for the same input [pedal] pressure.
With this system the pedal travel can be brought back to acceptable limits but the pedal pressure required to provide adequate braking may end up being excessive.

It's all a matter of balance between the two.