Don't give up! I've had, and am still having, some of the exact drivability issues as you...right up to and including the snooty neighbor (my exhaust is pretty loud). If I felt like it was a "sensitive" time of day ie: 6am or after 10pm, I would warm it up for just a couple of minutes and then idle the thing down the street for a while.
Not necessarily for this particular issue, but here's what I've done: new AirGap intake, new Summit 600 cfm carb, converted to HEI, new plugs ( the HEI let's me gap em at .045"), new 3" exhaust w hi-flow cat and Magnaflow muff, new pulleys and brackets to eliminate the non-functioning a/c, and so on and so on, but those are the big strokes.
And yet...even at operating temp (180*), I get an off-idle stumble which eventually goes away, buuut...still.
It started this behavior when I set the idle mix for max vacuum. The timing was good, but I need to recheck/reset it. This car has recently been pressed into service as a daily driver, Almost 40 mi per day. So the days of ignoring these kinds of drivability issues are over.
Also, I have a bizarre, rhythmic vibration that comes on at 45 mph/1900 rpm which, naturally, is how I spend 90% of my daily commute: a two-lane 45 mph road for about 15 miles...rrruuummm rruuuummm, rrruuummm, rruuuummm...like driving over grooves cut perpendicular to the road. Give it a little gas and it speeds up, lift and it slows down. It speeds up and slows down not in proportion to wheel speed or engine rpm tho. Weird. Checked the tq converter bolts yesterday hoping to find one loose or missing but all there and tight. Anybody had something similar?