The yellow is likely the silicone of the mask skirt degrading. Usually due to long exposure to sunlight. You may get some of the surface stuff off, but it tends to discolour right through the silicone so there will still be some showing no matter what you use to clean it with.
Mild dish soap and rinse well is what most use. The white spots could be mold. Do what you can with that tooth brush and soap on them.