Toxoplasmosis is a legitimate concern. It can be contracted from contact with your cat’s fecal material while cleaning the litterbox.
However, they make these things called gloves. They protect your hands nicely, along with a good wash after with soap and water.
You are as much at risk, or more, of toxoplasmosis from eating undercooked meat. One of the most common causes is gardening… you are in your garden, in the dirt, where neighborhood animals have defecated. You can also get it from unwashed produce that others picked.
Basic hygiene regarding your hands, and good food prep practices in the kitchen will protect you from toxoplasmosis.