I think the big distinction is between heavy AA and the others. Heavy AA was dependent on fragmentation with time fuses (yes, later on proximity fuses) and had a relatively slow rate of fire. So they were not really usable at short range. In contrast, light and medium AA were in many ways similar in that they relied on hitting the target, and were effective down to point blank range.
Well, I think there is a distinction between all three, or four (128mm is kind of in a 'super-heavy' category all it's own [EDIT: I should say "land based" 128mm here]. Today there are still three categories: rapid firing 3" or 5" guns, medium caliber (like 30mm or 25mm chain guns) and light caliber (20mm phalanx / CIWS) type guns on major warships... plus typically long, medium and sometimes also very short range missile systems. And now some are using energy weapons as well I think.
Once again, I'm not disputing that. But that doesn't prove that 40mm wasn't even more dangerous.
If I was trying to shoot something or someone, I'd of course rather have a 20mm than a 12.7mm, and I'd rather have a 40mm than either of the smaller options. It's certainly going to have better range and hit harder (as will 37mm or 25mm). But I don't think that means the lighter gun mounts are categorically inferior across the board. For land army / vehicle based, I can tell you that the Bofors 40mm mount was not light or easy to deploy. The US tried to mount these on vehicles, during WW2 they finally built the M19 MGMC which was basically a twin Bofors mounted onto a Stuart tank chassis, too late for the war though they used them a bit in the Korean War. Then they had something called the M42 "Duster" which was only marginally successful, and then the infamously unsuccessful M247 "Sergeant York" also of the Vietnam era.
The British mounted a Bofors into a Crusader tank, which became the Crusader III, AA MK 1, but they seemed to prefer the III, AA MK II and Mk III, which had twin Oerlikon 20mm.
The US also had the MGMC M 15 which had two 12.7mm and a 37mm, and it worked, but they ended up phasing it out in favor of the M 16 which had the four 12.7mm. Later they developed the Vulcan 20mm which the US mounted on M113 APCs and LAV type wheeled vehicles I think. The Vulcan 20mm is a very impressive weapon, the problem I saw with it is that it uses up an entire APC full of ammunition in about 1 minute.
Later more modern AA vehicles had 23mm (like the very successful Russian ZU-23 etc. types), 25mm (like the Chinese Type 95), or 30mm (South Korean K30, Soviet 2K22), or 35mm (German Gepard and several others). Of course the missions have changed - the targets are largely helicopters, sometimes missiles, and now increasingly drones, though they will also still be used against low flying CAS aircraft like A-10s and SU-24s etc. There are also a few with bigger guns than a 40mm, like the Soviet ZSU -57 (with twin 57mm AA guns). We also have MANPAD missiles and quite effective medium and long range SAMs, but there is still a niche for gun defence, albeit a smaller one. Now days defense against drones, rockets, and guided missiles, is particularly vital and the Israeli's started down that (I think going to be prove very long) road with their TROPHY point defense system for tanks.
At very close range, a 40mm gun is not quite as quick to track and 'acquire' targets as a 20mm or some kind of machine gun. Maybe that is why it does not seem to have ultimately been the ultimate AAA gun over all other options. Or maybe it has to do with ammunition capacity or storage, or the extra weight. I don't know from the engineering side, but looking at the history it seems that as good as it was, the 40mm AA gun was phased out, often in favor of lighter guns such as 23, 25 and 30mm (35mm is fairly close though, still popular and probably comparable). I think the Drone era is going to see a lot of new changes being made.
I suspect things like production limitations etc kept the light AA in use instead of being replaced by the more effective medium AA. In other words, the choice wasn't between replacing light AA with medium AA, but between keeping light AA and nothing.
I don't think that was necessarily the case. I agree with the premise that 40mm Bofors guns were probably ideal air defense for larger ships in WW2 (though they may have also wanted a layer of at least some lighter guns). Smaller ships (destroyers down to PT boats) are definitely going to want some lighter AA guns as well.