Both adjacent and nearby owners and renters can object to construction they don't know about, i.e., no permit posted. This is risky for those who are doing work WITHOUT permit as they risk a stop-work order and all kinds of problems and fines then. Other reasons: noise, messiness, unsafe conditions.
Also even if work is permitted, if work is being done is disruptive outside of agreed-upon hours neighbors may complain.
San Francisco is a special place (densely populated one at that) where everyone has a say; oftentimes passionately so. Once a complaint is lodged things can get tied-up administratively and relationships can deteriorate rapidly. Compromise is a better solution if possible. So, whether you're being disruptive or being disrupted.