You are not toast. Based on what you're sharing with us, you won't have a pricing hit either. If you're taking a loan of 150K and putting down 100K, that would mean that your Loan to Value or percentage of the purchase price being financed is 60%. So long as you keep it there, you won't have pricing hits and it would matter if your score was 850. You'd get the same rate.
In the web reference below, I'm including the chart from Fannie Mae that shows what these mysterious pricing hits are. It's actually more likely that you'll have a pricing hit for a California condo because a lot of lenders have those.
Also, now that we know this, I don't think it's advisable to mess around with your credit content. Even common sense changes can have unintended consequences. For instance, sometimes when you pay off a collection a credit score can temporarily drop. This happens because even though you might be paying it off, an aged date associated with the collection can get refreshed to an up to date one and that can temporarily hurt believe it or not. It's stupid but I've seen it sink clients. Also, if you drop below 660 you'll have a pricing hit regardless of your down payment so stay put. You're good where you are.
Let me know if I can help you with your loan. There's a lot of other good people here too. You asked your question of a good group.