Getting the British to not get back into the fight once Germany was involved with Russia is another problem. A not occupied Britian was always going to be threat.
I don't think that's something that would've flown with the British public. Once they were out and peace was signed, that'd have been it. Furthermore the British government would've probably been led by David Lloyd George or some such person, and wouldn't have been looking to re-enter the war. The Britain of the 1940s wasn't the Britain of the 19th century.
How does Germany enforce demilitarizing Britian?
She doesn't.
In fact, Hitler had this whole idea about Germany dominating Continental Europe while Britain would dominate the seas. He specifically wanted to preserve the British Empire.
What can Germany offer to bring Britian to the peace table if they don't invade and the invasion was never going to work.
While Hitler never offered concrete terms to Britain (he made an appeal for peace to Britain and France in the
Reichstag in October 1939, and one to Britain in July 1940, which can be read in English
here and
here), accepting peace with Hitler through negotiations via Mussolini (who'd arranged Munich back in 1938 during the Sudeten Crisis), was a real question in May 1940, the pursuit of which was being spearheaded by the by-then-former PM Chamberlain, as well as Foreign Secretary Halifax. Halifax had been both the Conservative Party's and the King's first choice to succeed Chamberlain as PM, and would've been PM, had he not turned it down. His reasoning for turning it down was that he'd had to have given up his seat in the House of Lords, but his real reason was probably that he didn't want Churchill breathing down his neck. He probably also genuinely thought Churchill more suited for the position at that time (Germany had not yet attacked the Low Countries when Halifax made the decision). See also the
British War Cabinet Crisis of May 1940.
If Britain had agreed to negotiations, concrete terms probably could've been established, and perhaps some sort of German withdrawal from most of Western Europe conducted in exchange for XYZ (not dissimilar to the 1871
Treaty of Frankfurt).
However Churchill was completely opposed to any kind of peace with Hitler. I don't think he would've made peace even if the entire BEF had been captured at Dunkirk.