This is where "Lights out management" is usefull. Almost all my servers are HPs with iLO - no need for physical access. You get separate network port (independed from OS) via witch server can be powered on/off and accessed via virtual serial port.