I find myself a bit conflicted now. I always brought them back to the cart "corral". My mom, while she was still with us, used the carts in the same fashion.
Don't feel too bad.
I could never convince her that the bag boys constantly patrolled the parking lot, herding up the carts left out there and bringing them back up to the store entrance.