I might have missed it in this thread, but what do you plan on using the pole barn for? Assumptions here are parking and/or working on cars. If that's the case, I can tell you that using a floor jack on my paver driveway is a bit of a PITA as the wheels tend to catch on the joints, especially with any load on the jack.
Paver style may make a difference. Our driveway is a mix of approx. 6x6 and 6x9 pavers that are beveled slightly on the edges, which create a lot of small valleys to catch stuff, like jack wheels. Ours were made for driveways so make sure whatever you use will hold up to the load. My experience is that concrete patio "stones" will not and will eventually crack.
The outer perimeter of the driveway. was set with a concrete border to help keep them from sliding around, but after 14 yrs we are seeing separation where we turn around an island in the driveway and they are also sinking in a few areas. If you use pavers, I suggest sealing them. That will help keep the sand down, prevent weeds/grass from growing between them and help with cleaning up any spills, drips, etc.
As far as crawling under vehicles, the surface of our pavers is smoother than the regular concrete driveway we have around back, but not as smooth as the concrete garage floor.