The job of the low beam headlamp has a contradiction at its core: let the driver see far and wide enough ahead for safety, while not glaring other drivers (who are located…ahead).
First: Do we need brighter headlamps? Yes. A
lot of people die because drivers can't see well enough at night. Mostly pedestrians; pedestrian deaths keep going up even as every other kind of traffic-related injury and death keeps going down. Do they have to be so painful to look at? No.
Glare is a complicated subject for a bunch of reasons. Glare light and seeing light aren't 1:1 in their effect on safety. The safety benefit of additional seeing light is much bigger than the safety threat from additional glare light—speaking in objective terms of the measured effect of increasing or reducing seeing light and increasing or reducing glare light (and all the various combinations of doing those things) on how likely a driver is to hit someone they couldn't see.
"OW, FRACK! GET THOSE DAMN LIGHTS OUT OF MY EYES, I CAN'T SEE!" is a common reaction to glare, but there's a big difference between the level of glare required to spur that reaction of discomfort and the (much higher) level of glare required to actually degrade a driver's seeing performance. Plus, it takes much less intensity to cause uncomfortable glare than it does to provide adequate seeing, and it takes a much bigger increase in intensity to get a usable increase in seeing than it does to get a squawkable increase in glare.
Traffic safety researchers and regulators have been arguing about headlamp standards for many decades, and it's easy to point to a beam standard and say "See, this standard requires more seeing light at this particular angle, so it's superior" (or "See, it allows less glare light at this other angle, so it's superior") and that's true as far as it goes, but it doesn't go very far. Compared to the U.S. regulation, the "European" (U.N./rest-of-world-except-North America) low beam specification allows much less light toward oncoming drivers
and calls for a much lower aim angle. This means much less glare, and much shorter seeing distance.
Despite these large differences, in many years of careful study nobody's demonstrated an actual, real safety advantage to the one system or the other. The reason is because low beams—even good ones—conforming to either standard are inadequate for the job we ask them to do. Low beams are inherently, geometrically limited in the seeing distance they can provide. It's a pretty simple matter using not-very-fancy math to figure out that the best possible low beam, aimed correctly and on a flat, straight road, is good for about 40 to 45 mph, tops, and that's a lot slower than we routinely drive at night. Let the headlamps be less than the best possible (most of them are), let them be aimed incorrectly (most of them are), let the road be not flat and/or not straight (most of them aren't) and the maximum speed drops fast for which the low beams can allow the driver to see an obstacle in time to avoid it.
There is (finally) a solution. It's called ADB (Adaptive Driving Beam) or GFHB (Glare-Free High Beam). It is a camera-driven system that keeps track of others on/near the road in front of the equipped car and actively, selectively shadows them out of what is otherwise a high beam light pattern. The equipped driver has high-beam seeing; everyone looking at the equipped car sees low-beam glare. It works very well, and it's legal just about everywhere in the world
except the United States, which (alone in the world) does not recognise the U.N. vehicle safety standards and instead has its own different ones.
Europe and the rest of the world has about a decade's experience with ADB, something like 15 million miles driven by equipped cars without any major "oops, we didn't foresee that" types of issues. American suppliers have ADB systems developed and ready to go as soon as the US Government approves it, but that's not going to happen in the foreseeable future.
It's really frustrating because there's no debate as there often is with safety stuff. Everybody is onside: the auto industry wants it, the safety researchers say we should have it, the consumer-focused organisations want it, and NHTSA, the US Government agency responsible for auto safety standards, says they want it. But if that's true, it's not evident. NHTSA asked the SAE Lighting Systems Group to "translate" the U.N. regulation into language compatible with the U.S. legal system (a valid request and necessary step). The SAE LSG did a very good, very prompt job of it. NHTSA then said "Eh, thanks but no thanks", rejected the SAE document, and issued their own proposal that basically says "You can put ADB on a vehicle as long as it's not an adaptive driving beam system" (essentially, NHTSA said sure, go ahead and shadow people's eyes out of the beam pattern, as long as the shadow is no less intense than a regular low beam would be at that same angle). And the system test they proposed is completely impractical: very difficult and expensive, and without a clear yes/no threshold—no automaker is going to spend the money to develop, test, and deploy the system with the risk of "Yeah, we know you tested it and it passed, but we've decided it failed, so you have to recall all the cars".
Meanwhile, the entire rest of the world seems to be doing just fine with one version or another of the standard NHTSA says is poopy and dumb and unworkable. Pretty much the same as NHTSA has been doing for a long time: "we're right and the whole rest of the world is wrong, la la la got our fingers in our ears la la la is too is too is too" like a spazzed-out six-year-old. But even if NHTSA decided to go ahead with a thoughtful standard, that's not likely to happen, either; the current U.S. Government is very anti-regulation, to the point of issuing regulation freezes, even though sometimes a regulation doesn't say "no, you can't", but says "Yes, go ahead". Agencies that try to issue regulations tend to get their heads (funds) cut off these days.
Anyhow, given the realities of the American market, even if NHTSA said OK tomorrow, it would be years and years before ADB would show up on anything but very expensive models. And it's not a retrofittable system, so it's not applicable to old iron like we talk about on a board like this one. So back to conventional low/high beam headlamps:
I already described how American regs allow relatively high levels of glare from low beams. Another factor, a big one, is that we don't give a damn about headlight aim on this continent, even though it's the single most important factor in how well a driver can see at night. It is really, really difficult to get a
proper headlamp aim job in North America. I've got a
page on the subject. If all headlamps were correctly aimed, driving at night would be a lot easier, safer, and more comfortable for everyone.
Also, the rest-of-world regs require self-levelling systems for high-output low beam headlamps. The lamps actually have a little stepper motor that tips the aim down in response to cargo or passenger load. The more advanced systems even compensate for acceleration squat and brake dive, and those systems make a solid safety improvement (bcuz duh—read about it
here) but they're not required in the U.S. despite the popularity of pickup trucks and SUVs that can and do carry a lot of cargo and passengers (and have long rear suspension travel for ground clearance and carrying capacity, which increases the amount of up-aim with vehicle load).
Also, in North America we don't spank people who monkey with their headlamps ("HID kits", "LED bulbs", and other such hacks; see
here). Most states don't periodically inspect cars any more, and those few that do don't really apply more than a quick glance at the lights. There's nothing in the federal regs requiring new cars' lights to be aimed correctly; the feds leave that as a matter for the states to control, which mostly they don't. The IIHS headlamp tests started in 2016 exert strong pressure on automakers to do a better job of aiming the lights on cars coming off the assembly line…at least until IIHS publishes headlight grades for a given model, then the automakers know they can slack off again without consequence.
Also, the marketers love and push the trend of headlights getting bluer and bluer and bluer, because they get to babble about "closer to natural daylight" (which it isn't, not in any real way) and each wave of bluer headlights makes all previous lights look old, thus applying pressure to buy a new car to keep up with the Joneses. But for any given intensity, bluer white light produces about 50% more discomfort glare than yellower white light, without any seeing benefit to go along with it. If the engineers rather than the marketeers were in charge, we could reduce glare while keeping seeing light the same, or keep glare the same while giving night drivers more glare, or some mix of both, by going back to a warm-white headlight colour.
So yes, headlight glare in the United States has been getting worse for years—that's real. And there are relatively easy things that could be done to make the situation a lot better, but none of them will likely happen any time soon.
On your own car: install the best headlamps you can reasonably put on ("reasonably" accounting for budget, amount of night driving you do with that car, etc). Don't assume any particular
kind of headlamp (H4, "E-code"/European spec, LED, etc) is necessarily better than any other
kind of headlamp (sealed beam, DOT, halogen, etc)—no matter what technology or technical standard is involved, good headlamps are better than bad headlamps, and there is an absolute mountain of junk on the market, all of which is hyped as an upgrade, so you have to shop very carefully. Use the right kind of bulbs, the kind your lamps were intended to use, and avoid bulbs with blue or purple glass on them. If your post-'83 car with plastic headlamp lenses has "cataracts" (lenses gone cloudy/yellow), move that to the top of the fix list—headlamp lens degradation is a major reason why pedestrians get hit to death. There are
ways of refurbishing the lenses if they're not too far gone; if they are, put in new OE headlamps if you can afford 'em—they're far better than aftermarket.
Have your headlamps (new or old) aimed correctly; keep trying til you find a shop that has and uses the right tools for that job. Keep your windshield clean inside and out; replace it if it's sandblasted. Don't use blue or blue-white LEDs in your dashboard lights, and whatever bulbs you have in them, keep your dashboard lights turned down as low as you can while still being able to read the gauges at a glance—this one's not as obvious, but
every single light source in your field of view contributes to disability glare (the kind that degrades your ability to see), even if it's much too dim to cause any discomfort.
And…now I gotta go eat again!