Maybe once, or twice, but eventually those slow, vulnerable planes will run into fighters. The comparison is similar to the notion of an SBD carrying a torpedo but not being able to land with one - it's not realistically feasible. The Albacore, Swordfish and TBD were all basically sitting ducks to a real fighter, or even a semi-fighter like an F1M. I think this is especially true for the hypermanueverable Japanese aircraft which could easily slow down to biplane speeds and tear them up. SBD not so much - it had the combination of factors which enabled it to survive such encounters, and even score air to air victories over Japanese bombers and recon aircraft. The Pacific Theater was not a 'permissive' environment in the sense of fighter opposition.
One thing occurred to me, from what I understand the TBD did not have self-sealing fuel tanks, which was a big problem. Is that true? Did the Swordfish or Albacore have them? The SBD did.
However, Swordfish being able to hit targets at night could have been very useful for example at Guadalcanal when the Japanese night attacks, bombardment sorties and supply runs were coming in. I think that was a real asset, even with the limited range. And I'll grant you, being able to use Albacore's as dive bombers makes them a bit more useful there as well. Ultimately they adapted the PBY for this same mission. I just can't really envision the Swordfish or the Albacore operating effectively during daylight hours anywhere near Japanese fighters, and given the fact that they (Japanese fighter aircraft) had two or three times the range, that put serious limits on the feasibility of their use, at least during the day.
I think in general, a carrier launched torpedo bomber was a very difficult design problem to solve in WW2, and the effective designs really didn't seem to emerge until near the end of the war (too late to play any significant role) and the beginning of the jet age when even promising designs like the B7N, Douglas BTD, Mrtin AM Mauler, or A1 Skyraider were rapidly becoming obsolescent (even if they did still find a role for the AD / A1 ). In the late 30's or even early 40's, it was barely possible to design such an aircraft and nobody fully rose to the challenge.