Just like the users physical and mental well being, we must also protect the users immersion. Breaking this immersion may not necessarily do damage to the user, but can result in a lesser quality experience.
For this reason it is important to have a quite space, and keep interruptions of the experience to a minimum.
Barriers to stop people / objects from entering the VR space are recommended. A closed room for VR-only users & facilitators is even better.