You're right because the entire enclave was on the oil rig just like how the US military is in the white house.
They lost access to the petroleum which meant that they couldn't repair their equipment or conduct science as quickly as the NCR and Brotherhood of Steel, so they fell behind technologically and eventually succumbed to attrition. Then you had the fact that they had no allies at all, and that they were hunted down and killed by the Brotherhood, the NCR, and the Master's Army.