Sounds like you've found your problem. A good thing, since running it like that would have gotten bad.
Okay, find out why?
When the cam bearings are installed, there's two holes in #4 bearing. IF the bearing is right, the bearing holes will be lined up with the two block ports. Yeah, run a stiff length of wire through that hole, to see if any junk comes back out with it. A little harder, push the wire in 'til it bottoms...turn the engine by hand, to see if you can feel the wire end, moving along the cam journal.
Bottom line...if the cam bearing is NOT right, cam needs to be pulled, and cam bearings re-done.
Toss a handful of common sense at it all, and go from there. Good find, before things went south!