You should read a little about it and I think you would change your mind.

The adverse effects of psilocybin are well known and documented and we know for sure that it’s not physically dangerous.

For the trip part we mostly know how it work and how interact and it can be healing (or bad trip depending on the context).

Still you may have have digestive adverse effects, not because of poisoning but because the human gut cannot digest crude mushrooms.

But you will not be physically sick per se like you would be with a bacteria or a toxin, your immune system will not fight anything.

Oh, I've done more than my fair share of substances across the hallucinogen class, with a preference for Shrooms or LSD

I just was under the impression that from a 'medical(?)' point of view, the trip is nothing more than food poisoning

That's probably based on some specific definition of food poisoning where you get effects from the food. It's semantics rather than saying anything useful about what's going on.