Returning from Port Authority before 5 is fine. Between 5-6.30 the lines for buses 113 and 114 are crazy and the organization at PA not very good. You often have to wait for more than one bus to get on and there is a lot of tension in the air. Not a great experience at all.
But there are an increasing number of direct trains to NY Penn and frequent trains to Newark Penn.
I drive to Harrison and take the Path to the World Trade stop for lower Manhattan. I do this because of the flexibility getting home later in the evening. It takes between 20 minutes and an hour depending on traffic to get to Harrison, the same back in the evening. About 25 minutes to WTC from Harrison. There's plenty of parking in Harrison and it costs between 8-11 bucks a day.