Hello Fish will be fine if nutrient level is normal. It works in the same way as salt. High levels of salt will upset fish, low levels can be beneficial by keeping bacteria and fin rot down. If evaporation makes the pond much stronger fish could be sick. There is no advantage to using nutrients except to plants grown in the pond. Algae may grow more strongly.
Perlite itself is an abrasive mineral, and if allowed to be eaten by fish it could make them sick or hurt them. I would recommend a strainer such as shade cloth, fly screen mesh or similar to strain out any Perlite that comes through. Perlite is not poisonous.