Don't forget, the B24 has cowl "cheeks" which house intercooler and oil cooler, so drag differences are not going to be directly proportional to engine diameters. Also, for most piston aircraft, INTERNAL cooling drag accounts for between 20 and 40℅ of total airframe drag. Air entering the B17 cowling flows around one set of cylinders and baffles and exits, whereas in the B24 it has to find its way through another set of cylinders and their associated baffles.
Now check out their fuselages. The B24J is an oval torpedo with only a windshield and two turrets to break the flow of air. The B17G has a chin turret, cheek guns, a (streamlined) doghouse type crew compartment, and various bumps and protrusions here and there. OTOH, the speeds you see on spec sheets aren't truly representative of combat conditions, as most missions were flown at grossly overloaded weights. Cruising at overload weight, with the resulting higher AOA, increases induced drag on the B24's Davis wing proportionally more than the B17's wider chord, more tapered wing. I think you'd find that the speed differential in combat use was less than the spec sheets would lead you to believe.
Having (briefly) flown both aircraft "hands on" (albeit at light weights), I can vouch for the B17 as a sweet and steady flyer, and the B24 as a nervous pig. The B24's narrow wing chord makes for a relatively narrow CG range, and I suspect we were loaded toward the aft limit, which tends to make any plane a little less steady.
Cheers,
Wes