Simply put, no - it's not a fair price. However, if he is using part of that fee to help buy down your rate, not all of it would be going to him, in which case the fee would be okay.
I would take the GFE he gave you and use it to shop your loan, rate and terms. If you'd like, you can email me your credit score, loan parameters (purchase price, loan amount, loan type) and I will email you back another Godd Faith Estimate. I don't even need to see the one he gave you. I would prefer you use it to help get yourself a lower rate and/or fees. If your broker doesn't budge, then you know he's only concerned about profit and not getting you the best loan possible.
Feel free to contact me. I promise you I will not contact you further unless you request it.