In the engines designed in 1934-37 it was more a question of of capacity (displacement) vs rpm. Max boost was still pretty low in most engine.
The Kestrel used in the Bf 109 and Ju 87 prototypes used a max of 3.5lbs boost (1.24 ata?). Granted it was an in service engine and not a prototype.
R-R had Built the the 35 liter Condor from 1918 to 1926 or so ( and a diesel version in 1932) plus the Buzzard of 36.7 liters which they developed into the "R" racing engines. The Buzzard only weighed 1140lbs.
I don't know why R-R dropped to 27 liters for the Merlin but they certainly had experience and the capability to build larger engines. But Rolls had only been making aircraft engines for less than 20 years, the idea that the Merlin would still be in first line service 10-15 years in the future would probably have shocked the designers of the time. The Merlin being R-Rs 6th commercial (non racing) engine in under 20 years.
Design of the Merlin started in 1932 (original concepts envisaged an inverted vee as one possible layout) and there was the possibility that the 37 litre R series would be adapted to become the next big engine; as such 27 litres was a good "halfway house" between that and the the 21 litre Kestrel (later the Peregrine).
This was literal in that all R-R needed to do to go from 21 (Kestrel Bore: 5 in or 127 mm; Stroke: 5.5 in or 140 mm = 1,295.88 in³ (21.24 L)) to 27 litres was to increase the bore and stroke of each cylinder by roughly half an inch ie; Merlin: Bore: 5.4 in (137 mm) Stroke: 6.0 in (152 mm) 1,647 cu in (27 L): ditto to go up to 37 l Griffon Bore: 6 in (152.5 mm) Stroke: 6.6 in (167.6 mm) = 2,240 in3 (36.7 L). With their experience of the Rs Rolls-Royce knew they could get at least 1,000 hp out of a 27 l engine, with the added bonus that it would be reasonably compact for the power generated.
As it was Rolls-Royce decided to design a new 37 l engine to take advantage of the refinements made in the Merlin cf the R series and, in turn, added further refinements resulting in the Griffon.
With the Jumo 210 Junkers started their 21 series with a similar engine to the Kestrel; the Jumo 210 was Bore: 124 mm (4.88 in) Stroke: 136 mm (5.35 in) Displacement: 19.7 l (1,202 in³), but with the 211 the dimensions went up by roughly an inch to Bore: 150 mm (5.91 in)Stroke: 165 mm (6.5 in) = 34.99 l (2,135.2 in³)