I was actually just reading theprepared’s article about water storage containers and learned about push-pin vents. Is this what yours looks like? (taken from that article)

I personally don’t have a good answer for you or experience with these, but I can offer my suggestion from looking at these pictures.
I do not think that the teflon tape would help out too much. I would be scared that the pin would push the tape down into the container to be lost forever. Teflon tape works by adding a bit of material between multiple threads of a screw and the other part(not sure what it is called). If you add teflon tape to this plug, there is really only one thread, the plastic of the container.
What I would do is push that plug in as far as it can go, with a little bit of turning pressure too. Then tip the container on it’s side and see if it will leak. Notice the flow rate. Then add a bit of teflon tape and do the same and see if the leak stops or is slightly better.
Teflon tape should be safe, as it is used in plumbing and other water containers.
If you do not get into your container very often and are just using this for long term storage, I would place a piece of teflon tape on the plug, push it in as hard as I can, then take a hot glue gun and put a bead of hot glue around the opening and the plug. Then maybe a piece of duct tape over all of that. The hot glue won’t be hot enough to melt the plastic, and can come off super easy just by pulling it off with your fingers. It will hold things in place well, and maybe seal up any gaps a little.
I’d love to hear about your results and what you end up doing!