Your code is correct.
If you are indeed directly using the millisecond value that you get from the server, then I suspect that something is coded incorrectly on the server side.
Keep in mind that Dates represent instants in time, and do not contain time zones.
Current "milliseconds" represents, as the documentation says, "the difference, measured in milliseconds, between the current time and midnight, January 1, 1970 UTC(coordinated universal time)" - thus, the offset from the "epoch date" in milliseconds.
Dates don't have timezones until you format them for a particular zone - they just contain this millisecond offset.
So, if you are getting a time other than the one that you expect, either the server is sending the incorrect value, or PrettyTime is inferring the incorrect local timezone when you call format(). PrettyTime appears to use the currently configured Locale, so that's probably not the issue.
I suggest that you test whether the server time is the problem by testing PrettyTime with a locally generated date, something like this:
Date now = new Date();
System.out.println(p.format(now));
If you really want to convince yourself that everything is ok client-side, try this:
Date now = new Date();
long msec = now.getTime();
Date theDate = new Date(msec);
System.out.println(p.format(theDate));
also, to find out what the server is actually sending you, https://currentmillis.com can show you the current time in millis, and convert millis to human-readable formats.