You actually kinda answered your question in the comments. Unless you want to work with all complexity of Calendar, calculate holidays, week days, time zones, etc then simple long will serve your. Calendar takes a lot of memory (instead of 8 bytes for long) because it has to serve all these conversions, transformations, comparisons, leniency, field manipulation and other functions based on locales. It has a lot fields staring with ERA, DAY_OF_WEEK, DAY_OF_WEEK_IN_MONTH, etc. All of them are used in some actions so performance of the class is much worse then simple long comparison.
Of course you cannot compare to other Classes. Each class serves its own needs and could be much more complex and expensive then Calendar.