Has the seller's lender approved the sale and provided you with a short sale agreement notice? Typically, if the short sale addendum to the purchase contract was used, that is the official time when your inspection period starts. Prior to that time, if you don't have the seller's lender's approval, you can choose to cancel by notifying the seller in writing of your decision to do so. That would also depend on if you have any earnest money that was agreed upon to be non-refundable for a specified period of time prior to the seller's lender's approval. Usually 60 days.
Depending on where you at in the short sale transaction process, you may be able to cancel the contract and receive a full earnest money refund. Alternatively, if you are not concerned about losing your earnest money, you can choose to cancel the transaction at any time.
Any of the agents on here will not be able to advise you, especially since we haven't seen your contract. It's best that you explain to your buyer's agent about your concerns of the property location ask them to help you cancel the contract and get your earnest money refunded to you. After that point, as long as you haven't signed a buyer broker exclusive agreement with your agent, you should be free to choose another agent to work with.
If you have any other questions feel free to contact me directly.