You'll get a lesser mechanical advantage by using larger master cylinder.... Pressure=Force/Area.
ie/ More work req'd by you.

As you've increased the diameter of your brake pistons by a fair bit by going to the Silvia / R31 brakes, you're already getting a heap more mechanical advantage than stock.

with that in mind, I reckon a slight increase in your M/C size wouldn't hurt.

But - As any tradesman will tell you.... If it aint broke, don't fix it