Come when called
To help you feel more at ease when you first let your cat out, spend the time he is confined teaching him to come when called.
Initially, while in the same room, call his name and gently shake a packet of his favourite treats or the container you keep his food in, then immediately feed him some.They need to be very yummy so your cat will want to come to find and eat the special treat.
Once he runs to you from across the room to get his treat, start increasing the distance you call him from until you can call him to you from any room in the house.
When you let him out, he will still respond to your call if you continue to reward him.You now have a cat that will come when called.