Points translate into a "buy-down" of the interest rate. In exchange for getting more money up front, the lender takes a lower rate over the term of the loan. To the Bank, money now is worth more than money in the future.
Usually points are attached to "teaser rates". The potential buyer is drawn to the Lender by the lower rate and doesn't realize that he/she is paying for it. If you do not want to pay points your lender will most likely be willing to offer you a "zero point" rate, which of course will be higher. You should then do the arithmetic to figure out when your buy-down becomes cost effective. You will be saving $x per month through the lower rate. Add the total savings and divide by the dollar value of the points to determine how many years will elapse before the points are amortized. If you plan to stay in the home longer than that period, the buy down may be attractive.
However, in a low rate environment like we have presently a buy down is probably not justified. My guess is that you have a greater need for the money now rather than the savings over the long term.