Usually points are attached to "teaser rates". The potential buyer is drawn to the Lender by the lower rate and doesn't realize that he/she is paying for it. If you do not want to pay points your lender will most likely be willing to offer you a "zero point" rate, which of course will be higher. You should then do the arithmetic to figure out when your buy-down becomes cost effective. You will be saving $x per month through the lower rate. Add the total savings and divide by the dollar value of the points to determine how many years will elapse before the points are amortized. If you plan to stay in the home longer than that period, the buy down may be attractive.
However, in a low rate environment like we have presently a buy down is probably not justified. My guess is that you have a greater need for the money now rather than the savings over the long term.
Hope that help,