Laws of physics were firmly with Spitfire's (Shenstone's actually?) choice of thin wing, that made possible good turn of speed even on moderate power. When Spitfire ended up a bit slower than expected, it was due to small details that added up - like external windscreen instead of internal, draggy type of antenna installed, protruding cannon barrels, not fully covered U/C, use of 6 exhaust stacks vs. 12, obsolete carb, unfaired rear mirror, sometimes sloppy fit finish etc.
All of these things were eventually adressed from late 1942 on, it was too bad that Mk. V was the worst offender in here, just in time the LW upped up the bar with their 400 mph fighters from second half of 1941. The Spitfire V was capable for 400+ mph, the Mk XI for 440+ mph (as fast as Merlin Mustang), as can be seen in the document provided by Aozora.