Well the seller sounds pretty tough, you already lost round one of negotiations. Are you sure you want to be going against this guy by yourself?
Really though, if you have had an agent working for you, now it appears that you are kicking the agent to curb and you do not seem to care if they get paid. What if your employer had you do a bunch of work, found a loop hole and then found that they could get away without paying you? You would think they were a monster. I do not know how much work the agent did for you. If they agent spent a lot of time with you, the right thing is to want them to be paid for their work.
People have a false idea and do not understand how hard agents work. Agents should be paid well if for no other reason that they are at risk of not getting paid. Sure people love to avoid paying agents, they are under the false assumption that their job is easy. That is exactly why agents rarely show these "for sale buy owners." These are the greedy rascals not the agents.
If you are buying a home for sale by owner, my guess is they want full market value, without you having the protection of your inspections and disclosures required by law. Well if you are not getting the protection of an agent, you should get a discount. Are you getting that discount?
Then you are being cheated.
What happens when you get into the house and find out there is mold, a leaky roof, plumbing problems etc.? Do you think you will be able to call the seller, tell them about it and have them write a check to fix it?
The fact is that with the internet you do not need an agent to find a house. A trained dog could find a house. You need one to negotiate on your behalf.
I say it may not be a good idea to negotiate with the seller yourself, he is already kicking your butt. Go to your agent have them write up an offer with the finders fee written into the agreement.