The thing is, at least as often as not, the 'right' engine didn't work out, and quite often it languished in development hell for god knows how long, resulting in a promising and potentially useful aircraft either showing up too late or being cancelled, and wasting countless man hours and resources trying to overcome fundamental design flaws. This happened over and over and over again.
Once you have a proven engine that is free of major* design flaws and faults, it therefore makes sense not to abandon it too lightly.
Some of the best aircraft of the war were created by merging a proven engine that performs as needed to an existing airframe stuck with a less than ideal engine, the classical example being the P-51. Another is the LaGG-3 and the La-5.
I think the War Ministry, RR etc. were operating on assumptions which we can now see from hindsight were not entirely correct. Not all of the future designs were going to be big birds, and small fighters remained on the front line to the end of the war. In fact I'm sure they could have found more uses for Peregrines, at the very least you could put them on a lot of lighter aircraft which were still widely used and give them a performance boost thereby. Avro Anson for example. Blackburn Skua might have gotten a new lease on life. Even the Lysander might have been more useful with a Peregrine, it would have probably been faster.
If you could boost power a little bit I could see a Peregrine being a useful powerplant for a Blenheim or a Hampden.
All that is just speculation of course. The bottom line historical fact which has been repeated over and over but still bears emphasizing - is that the British wasted resources on aircraft that ultimately proved to have virtually zero impact on the war effort and in many cases cost more lives than they saved. Defiant has it's fans around here I know, but not many people really wanted to pilot one into combat after the first few sorties. Blackburn Roc was even worse. Henley was a waste of time - there were plenty of old obsolete planes that could act as target tugs you didn't need to build 200 more. Lysander was marginal at best, certainly there was no need to produce so many (over 1,700). Did they really need to keep making (over 4,000) Blenheims into 1943? And so on.
RR obviously had their reasons to cancel the Peregrine, but that could have been overriden. It's not some unforgivable Sin because every country made mistakes like that. But cancelling the Whirlwind was indeed a mistake, in hindsight.
*Peregrine had some minor problems but so far as I'm aware it was a fundamentally sound design.