If you are 3 weeks from closing, then you probably have not received a mortgage commitment from your lender. Talk to your loan officer about it. If you are rejected by your lender, then you can get out of the contract without losing your EMD. Is this a condo? There is also a contingency to review condo docs that could allow you to get out of the contract without loss of deposit.
If you are already past that, talk to your agent and/or your lawyer about what other escape clauses are still available to you in the contract. If you have no other escape clauses you can invoke, then you only risk the deposit monies you have given.
I would advise you to act quickly. If you can't complete the sale, it's only fair to the seller to let them know as soon as possible, as their house is now off the market for you. The more time that it is off market "for nothing", the less likely they will be to work with you.