Moving from warm sunny california back into the bitter cold and heavy snow of new york? Yup, I think that's crazy.
Be aware, property and income taxes are very high now in NY and NJ. If you buy a condo look out for HOA fees. I saw one person say they were looking at $750 a month for those. Those fees come if you live there every month, or never show up.
if you choose to move to the bronx in the summer, and go back to cali in winter, it might be ok.
The important thing is to look before you buy. Come back, spend a month or 3 renting. After that time, you can decide if it is still as you remember, or if you no longer wish to live there. The bronx could have changed, you may have changed. Either or both (both did happen) may make you feel it is a great idea, or an idea whose time has passed.