If the RN had been forced to fight without RAF air superiority, they may not have failed entirely to hold the KM back, but they would have struggled. RN AAA systems were of inferior quality in the early stages of the war , and the FAA's fighter force was composed of very second-rate machines, so defending major surface assets from air attack would have been very difficult. On the other hand, we have already discussed the fact that Luftwaffe anti-shipping tactics were not at their most advanced, so it is possible that the RN would have survived to attack the barges, which would almost certainly been a massacre of German infantry.
On a side note, I'm not sure subs would be all that effective in the Channel and the southern North Sea. Picture the scenario:
1. The LW has total air superiority, meaning surfaced movement (and therefore battery charging) can only take place at night.
2. These seas are VERY narrow. With a lot of traffic moving about, it would only be a matter of time before contact was made with a sub moving in the area. (Conversely, of course, this a benefit as it means you should have no trouble finding a target)
3. Which brings me neatly to the next question... DO subs have the right weapons for the targets involved? The barges and lighters were of extremely shallow draft. I have read that during WW1, German subs struggled to torpedo RN monitors because they were of shallow draft and torpedoes went right under them. I am no torpedo expert, but I would imagine that this might also apply to RN subs attacking lighters. If this is the case, you have no weapon - you can't surface to use your gun because the LW has air superiority and even a geriatric civilian vessel has a chance of successfully ramming a surfaced sub, never mind the KM escort.
4. Assuming you survive all of this, you cannot return to a base to re-arm and refit because all of your bases and depot ships are within reach of LW bombers.
So basically, IMHO, a sub operating under the conditions we have surmised wouldn't have much luck.