Thumpalumpacus
Major
Problem may have been that there wasn't one Japan, there may have been at least two and maybe 3. And the American diplomats were not negotiating with the Japanese Army/nationalists.
While they weren't negotiating with IJA, IJA very much had the final word on whether an agreement would be reached or not. If the government reached an American agreement that IJA didn't like, all they had to do was resign the Minister of the Army and the government would collapse. Because no government could be formed without the Army and Navy appointing their own ministers from the active duty officer corps, they could prevent the formation of any government that did not do what they wanted.
What face saving could be done to justify giving back that territory?
Nothing the US would, or perhaps could, offer.