I know what you mean.
I recently drained my tank to remove it, and besides not beng able to remove the tank due to stuck nuts, i had to unblock the drain bung with a screwdriver as the gunk on the bottom as soo hard, i had to use a hammer to break it up!!!!

Flushing out a bit of grit and fines is as easy as you said, some soap and a hose.

Any ideas on the caked stuff in my tank. perhaps let sleeping dogs lie as I havaent really had aproblem with dirty fuel.