It's slightly unfair to compare the F-4 to the F-14 as some of the lessons learned from the Phantom went into the Tomcat (and the Eagle and Hornet too).
The F-4 was, as I understand, originally designed to down bombers from long range with missiles. Hence, it didn't need particularly good dogfight capability - and so confident were they that missiles would settle the issue BVR or at least beyond gun range, the Navy didn't even put a gun on it.
Indeed the good dogfight capabilities of the F-4 in its class was accidental, two powerful engines were fitted both for reliability and with the speed performance agenda of competing against Russian speed records (held before the Phantom by the Fishbed), but as it was discovered in Vietnam this also worked great for sustained manoeuvers because it had plenty of excess thrust, so could outfly the much more agile MiG-17 for example so long as you kept the speed up where his controls were heavy, which was easy to sustain in the Phantom through manoeuvers and even in the vertical. The trick was just keeping the fight close to transonic and the Phantom was a great dogfighter, it had problems against lighter craft if too much speed bled off like sticking with a rolling scissors instead of doing a yo-yo.
But this capability was never intentional, it had been designed with BVR combat in mind with only a limited CWC capability, it was thought extended, seat of the pants turn fights lasting several minutes were pretty much over because even if you got that close within the first few seconds a sidewinder should've taken care of things. But hindsight is 20/20 in terms of realising how low tech the seekers and avionics were back then.
But still, whilst it can't compete with any 4th gen fighter, against other 3rd gen contemporaries the Phantom is a very good dogfighter so long as the pilot is aware of situational disadvantages.
Vietnam was a wakeup call that dogfighting was still alive and well and that Phantoms were losing out to the much smaller and more agile MiGs. The lesson was not lost on either the USN or USAF, who have put a gun into every fighter they have used since. They have also made sure that all of thier fighters have the capability to dogfight.
Judging from what I've read so far the need for a gun was in part a product of poor reliability of AIMs at such an early stage of development where their planners and demonstrations under test conditions were quite a bit more hopeful, or unrealistic. Early sparrows had simple technical difficulties and sidewinders couldn't track in more than a 3g turn. In a lot of dogfights there was no available firing solution with the missiles of that period but a gun would've worked just fine, pilots cursed their absence and MiG's could fire more often during turn fights than a Phantom who really had to pick his opportunity to even get a firing solution at all.
The high early loss rate wasn't so much related to lack of an internal gun but the circumstances of the conflict, severe flight restrictions (ROE), the deadly SA-2, more AAA over Hanoi than used by Germany in WW2, plus at the early stages the NVAF were using their Fishbeds where later on they shepherded them and used them only under cover for hit and run (because they didn't have many), so MiG-17's were the main type later, MiG-21 early on which performs better than Phantoms at high altitude. And it was related to pilot training, local command air doctrine, and the fact the vast bulk of losses were USAF F-105's (more than 40/month being downed until 67) rather than Phantoms. The arrival of Colonel Olds, a WW2 ace and his pet project known as Operation Bolo really turned things around. This was well before the Phantom got its internal guns and pretty much chased the NVAF Fishbeds from the skies except for relatively isolated encounters.
Assuming that the F 14/15/18 had never been built, the Phantom would have continued to lose out to more agile Warsaw Pact types as a fighter - hence the construction of the F-14 and -15. And while it's huge payload would have kept it in business as a mud-mover for a while longer, I think the drive toward single-seat multi-role types like the F-16, Rafale, Eurofighter etc. would have spelt the end for the Phantom regardless of it's performance. Air forces always want the best bang for the least buck, and the F-4 could not stay competitive on both fronts much past the late 80s, IMHO.
The F-4E is matched I think by the Flogger-B but not as a weapons platform, and the MiG draws away as altitudes rise. From about 1977 the Flogger-G kind of mothballs the Phantom as obsolete with sheer thrust/weight and clean design, plus the radar set is now doppler with improved track mode and the R-23 are an updated version comparable with the AIM-7F. But by this point the F-16 is nearing deliveries which turns things back around.
The Phantom didn't really lose out to Warsaw Pact contemporaries, certainly not in downgraded export versions (most Soviet satellite nations were using MiG-21MF and later license produced "bis" into the 80's with smaller numbers of Flogger-B handed over after 1978 whilst the export Flogger-E was just a MiG-21 with swing wings, and exported MiG-21FL was just a first generation series production with a side hinge canopy fitted).
The Phantom has total parity until after 1977 and then only over Russian airspace it has problems against the Flogger-G. The whole idea though was that the Phantom would be replaced by the F-16 and Hornet in front line service by 1980, and you could've pushed that ahead a couple of years if you really needed to, but the Eagle sought to do it back in 1972 which was just premature in terms of strictly defined need.