IMO, the simple truth is this as far as fighter evolution during World War II--the focus went from traditional agility and lightness to speed, climb rate and firepower. The reason for this, in simplest terms, was due to bomber/recon plane design. Bomber and recon planes realized that speed could be life. The result is that they got larger and more powerful (more powerful, often larger/heavier engines, and more fuel for those engines as well as a desire for increased range).
This meant that fighters had to get faster though similar means, and even with radar, the speeds of possible bomber and recon planes shortened intercept windows, hence speeds, climb rates and power rose, to the detriment of traditional low speed agility. However, even fighters did see the advantages of sacrificing that in favor of speed, agility at higher speeds, climb rate and firepower. Not to mention that as all planes got faster, firepower increased, due to the narrower windows of keeping on target in a maneuvering fight.
Of course, more and heavier machine guns and cannons did also mean more weight, which goes in concert with the power increases.
Right, I think all this is true.
This, of course, is a big generalization. However, even early on this trend was realized. The genesis of the Hawker Typhoon/Tempest began in 1938. Even the IJA/IJN realized at about the same time that things were starting to change, as they realized they needed fast point interceptors to protect territory. I do think that they (and some others as well) got caught out by how fast things changed.
This is where I part ways with you - slightly. I would say this - rather than the shift to high flying, heavy, fast, not necessarily super maneuverable planes being inevitable, some kind of innate progression, I would say it is directly tied to the changing nature of the war.
Like you said, the fighters follow the bombers and the recon planes.
In NW Europe, the Strategic bombing campaign got into gear in 1944. high flying, powerful, heavily armed daylight bombers needed heavy long range fighter escort that could fly fast and high. The Allies also had shorter range fighters, Typhoons, Tempests, Spitfires, and (normally high altitude) P-47s repurposed, and in Italy the old P-40s, which either did the CAS themselves as fighter bombers, and / or protected the frontal aviation over the tactical war. The Western Allies had the advantage here in the tactical war because the Luftwaffe was forced to commit most of what they had to defense against the Strategic bombing campaign.
In the Pacific, similarly, the B-29 raids started up in 1944. These are even faster, longer ranged, potentially higher flying, more heavily armed and more capable than the B-17s and B-24s flying over Europe. By this time, the tactical / naval war in the Pacific is winding down. There are still some big battles, but the USN now has the IJN badly outnumbered and their new tactical fighters, the F6F and F4U, have the older Japaense fighter types outmatched. The Japanese suddenly have to figure out how to contend with B-29s and their (new to the Theater) escorts like P-51Ds and late model P-38s. This is one of the reasons the IJA and IJN started trying to make some really fast and high flying planes, though they never managed to really make a good B-29 killer.
But the big deviation from this trend is the Soviet Union. They never did concentrate on super fast, super high flying, heavily armed but not as agile fighters. Instead, they focused on small, light, (comparatively) lightly armed, and very agile frontal air superiority fighters like the Yak-3 and the La 5FN / La 7. This is because for them, the main war was not a Strategic Bombing campaign, but it was (still) a tactical war. The fighters follow the bombers, the bombers follow the war.
That is also my point about 1941-1943, this is when the tipping point of the war came for the Allies, IMO, and we started to see the shift - except in Russia - toward the Strategic war rather than the tactical.
As proof that this was an adaptation to the war and not inevitable, I again point out the continued competition among 3rd and 4th Gen jets over maneuverability vs speed (and now with 5th Gen, stealth)
And as to the assertion that the Zero was a "1940" fighter, well, one, it first entered service that year, if we want to be fair and accurate. Secondly, if it was serving in the European Theater, it's performance, especially in terms of speed and even climb, would've been good for 1940, but by 1942, it would've been obsolescent, if not obsolete.
In the Pacific, the Zero was helped by the fact that, one, most western powers weren't fielding the "latest and greatest" like they were in Europe. Overmatch against the Zero in terms of performance didn't fully happen for the USN/USMC until the Corsair and Hellcat entered service. Also for the British Commonwealth, the Spitfire VIII didn't enter service until sometime in the second half of 1943--months after it entered service in the European Theater. Also for the USAAF, Allison powered P-51s and A-36 fighter/dive bombers didn't show up in the CBI until about the same time in numbers.
The Spit VIII was probably a match for an A6M2 or M3 when it arrived, but the F4U and F6F weren't delayed, they got there as soon as they could make them. For most of 1943 the F4F and older army types were still holding the line. The Allison P-51 and A-36 didn't have a good record in China / Burma, the dominant US type in the CBI continued to be the P-40. In the Pacific the most important army type was the P-38.
Some of this was also due to the Allies agreement/mentality of "defeat Germany first if possible" edict. And understandably so, since it was felt that Germany was the more dangerous foe at the time. How much of that was accurate and how much was still rooted in old prejudices/biases, is up for debate.
Fact is that Japan in some areas weren't far behind the Allies or even Germany, in some areas they were. As mentioned, Japan did realize that fast interceptors were needed as soon as 1938. Problem was that existing programs took priority, and the interceptors' development also had some problems--though they weren't unique in that, as several air arms had problems with developing the first 400+ mph fighters.
In theory it was "Germany First" but in fact the US put a huge amount of assets, like most of their Navy, into the Pacific. Precisely because the Japanese were such a tough fight and a direct threat to the nation.
Japan also had the issue of sort of having to play catch up with the western world in terms of industrialization. This wasn't for lack of smarts or talent. But until Admiral Perry showed up, Japan was largely isolated from the rest of the world, and they were decades behind much of the rest of the world as far as the Industrial Revolution.
Agreed
Also, as far as infantry, the IJA was one of the first armies to widely use machine guns for infantry support. Though, oddly, during World War II, they didn't widely field submachine guns, which were ideal for jungle warfare at the time.
So as far as the Japan vs the world deal, they did lag behind in several areas during World War II. Some of this was partially also to the IJN getting favor for a lot of raw materials, such as steel for warships vs the IJA for tanks, for example. But for Japan being an island nation who's defensive and offensive operations would likely be heavily naval based, this did make sense. However, I do think that we should give them a bit more credit from a talent and smarts standpoint. Though if you're limited with what you have to work with, you're limited with what you have to work with.
Japanese Army was quite effective, as you can see clearly from their smashing victories against the British in Malaya and the US in the Philippines. But they seem to have kind of frozen in time. obviously the navy was a higher priority for development, but as you noted there was an odd pause in the crucial year. A6M5 should have come out when the A6M3 did. J2M and N1K1, B6N and B7N etc. should have been developed much more quickly than they were. Same with Ki-67 bomber, Ki-44 (avalable, but not in enough numbers) and Ki-84 for the Army. There was a weird kind of complacency in the crucial turning point moments of the war that you also see with the Germans, IMO. This is when the British, Americans and Soviets were pushing hardest for new kit.