Okay, I've discovered a couple of things since then:
1. I don't think its the bearings. I must've put 10k miles on this engine before I noticed the issues. I think it'd be knocking heavily by now.
2. It uses some oil, but only at startup. IE, if it's been sitting for a time, there will be a puff at startup, but it immediately clears and it doesn't burn a drop while driving. I think oil is leaking into the cylinders while it's parked.
3. It seems to have good compression. Fires up REALLY fast and runs perfectly smoothly (by A-series standards)
4. the metal in the oil is NON-MAGNETIC
Here is a pic again, so you can see the color of the shavings:

If it likely isn't the bearings, what else might it be? The oil leaking into the cylinders sounds like a damaged guide/seal. Perhaps I should have the head rebuilt? I'd like to get this thing in better shape so I can start doing the mods I want to do. What else might it be? All I can think of is the gears on the distributor or oil pump being eaten up.
The car in question.