An often overlooked fact is that difficulty was artificially inflated due to the lack of quality-of-life features that we take for granted today.
Although I agree to an extent, I'm going to play the devil's advocate here and say that many of the feature we've gotten over the time don't have quality of life either, they're just annoying abstractions that exist almost entirely for the sake of the more inept players, who're more interested in simply advancing the plot then actually playing a game.
Example, the mini-map feature is, in a way, the bane of exploration in RPGs in my opinion. I'd much rather play a game that would require me to draw my own map, take my own notes etc in order to make sense of the game-world.
Then again, I didn't grow up in an age where the "internet can provide me everything"-mindset had yet been normalized. I don't expect kids born after the late 80's to ever enjoy playing a game that doesn't hold their hands 99% of the time.
@topic in general -
Like another here said - the difficulty of old games was a remnant of arcade game-design.
It's based around a concept of retrying until you learn the patterns of the game, and then succeeding.
The way I see it, difficulties really only exist in two forms - you've got the experience-based difficulty (the one based around learning patterns and retrying), and you've got the adaptation-based difficulty (how well a player manages to use his/her current set of skills to solve a problem, but where experience cannot be used because the problem changes with each play).
Puzzle-based Visual Novel games, Many NES arcade style games etc, fall clearly into the first category. Many FPS shooters, fighting games and games that spawn random levels/enemies fall into the latter.
Neither of these are bad approaches in and of themselves, and many games exists in a spectrum using a little bit of both.
Most RPG's for instance, have set bosses with set attack patterns, but also have random encounters.
Max Payne is a shooter/action game, but all the scenarios and shoot-outs are made up of completely predetermined enemy positions, but there is some room for variation.
There is a problem though - the experience approach usually requires some patience, and the adaptation approach requires the development of skills.
So what happens? Well, if you set the experience bar high, then you get a game where you have to repeat the same levels tons of times to succeed, and most modern kids don't have that kind of patience.
If you set the adaptation bar high, then you're looking at games demanding pro-gamer level reaction-times and eye-hand-coordination and that's just beyond most people anyways.
So what do modern game designers do? They do neither. They make -loads of auto-saves, so you don't have to replay long sections, but they retain a simple game-play with low adaptation difficulty, so everyone including your grandmother can clear the game in a day or two.
I don't think that's any better than old-school arcade-style design - in fact it's just another form of the same factor influencing development. You know what that factor is? It's
greed.
Old school difficulty was made to keep players locked to the arcade cabinet, spending money.
New game design is easy and accessible to everyone so nobody feels like a loser, with bright flashes and "rail-roading" to keep the attention-deficit youth of today glued to the screen, and finally the experience is coated with "incentives" for continued play by following what the industry has learned from online-gaming business models (e-cred, social media link-up, trophies and mini-transaction).
If old-school games are like alcohol, modern AAA games are like crack-cocain.