What you did was not bank fraud and it is done all the time. Unfortunately the purchase was not structured correctly. You should be the one making the payments to the lender and not the old owner. When you do a seller finance transaction correctly you actually close in a title company and you are the one who holds insurance on the property. While most loans are not assumable at the same time there is not a lender out there who will call a note due if they are getting paid every month as agreed. They could but they never do.
You should contact an attorney as was advised because what the person who sold the home to you is doing is fraud. Seller financing is a good way for people to get out of a bad situation. Good for the seller because they are avoiding foreclosure or a short sale and good for the buyer who cannot qualify for a loan at the current time.
Unfortunately if you do not structure everything correctly this can happen. On the reverse if you were not making the payment the same would be true. So now you need to consult a real estate attorney to see what you options now are.