If you really like the area and are concerned about the schools, there are nice private schools in Madison. I don't know if I would call too many schools in the Madison area rough. We moved back from Portland, Oregon and the any school in the Madison area is better than the best schools there.
Another thing I suggest to my clients is to my a priority list of everything from # of bathrooms to distance from schools. Many times that will lead you in the right direction.
I know how important schools are in making a home purchase decision.
Each person has different criteria for what they consider important in a school; diversity, test scores, class size, IEP programs, etc.
greatschools.net is a resource for some of that information.
When you visit Madison, I would encourage you to go visit the schools and take a tour. Another option would be to phone them to ask your specific questions and ask for PTO contacts to use for references.