It's possible. I do believe that some people don't check their lifter bores carefully, and burrs on them cause lifter problems that wipe cams. However, I also believe much of the problem is with lifters. You sound like the type that checked things carefully, so I would lean more to suspecting the lifters.
My father-in-law just had a 302 Z-28 Chevy engine wipe cam and a (solid) lifter on startup. He heard an increase in tapping and immediately shut it down. He has been building engines for a living since the late 1960's and cars he has worked on have set more than 9 NHRA records - the guy knows how to build and start an engine! Everything was done absolutely by the book, although he felt it would be fine to start up with 102# seated pressure springs. Sure enough, when we tore it apart, #7 intake lobe and lifter was wearing quickly. I suspect that the lifters were not properly shaped on the bottom to make them spin.
Anyway, lesson learned. The motor had to be torn apart, checked, cleaned, etc. and he will be replacing the lifters with ones that have an oiling hole on the bottom, plus we put very weak springs in the heads for startup.
