How can I properly split imbalanced dataset to train and test set?

Question

I have a flight delay dataset and try to split the set to train and test set before sampling. On-time cases are about 80% of total data and delayed cases are about 20% of that.

Normally in machine learning ratio of train and test set size is 8:2. But the data is too imbalanced. So considering extreme case, most of train data are on-time cases and most of test data are delayed cases and accuracy will be poor.

So my question is How can I properly split imbalanced dataset to train and test set??

Answer 1

Probably just by playing with ratio of train and test you might not get the correct prediction and results.

if you are working on imbalanced dataset, you should try re-sampling technique to get better results. In case of imbalanced datasets the classifier always "predicts" the most common class without performing any analysis of the features.

Also use different metric for performance measurement such as F1 Score etc in case of imbalanced data set

Please go through the below link, it will give you more clarity.

What is the correct procedure to split the Data sets for classification problem?

Cleveland heart disease dataset - can’t describe the class

Answer 2

Start from 50/50 and go on changing the sets as 60/40, 70/30, 80/20, 90/10. declare all the results and come to some conclusion. In one of my work on Flight delays prediction project, I used 60/40 database and got 86.8 % accuracy using MLP NN.

Answer 3

There are two approaches that you can take.

1. A simple one: no preprocessing of the dataset but careful sampling of the dataset so that both classes are represented in the same proportion in the test and train subsets. You can do it by splitting by class first and then randomly sampling from both sets.

   import sklearn
   
   XclassA = dataX[0]  # TODO: change to split by class
   XclassB = dataX[1]
   YclassA = dataY[0]
   YclassB = dataY[1]
   
   XclassA_train, XclassA_test, YclassA_train, YclassA_test = sklearn.model_selection.train_test_split(XclassA, YclassA, test_size=0.2, random_state=42)
   
   
   XclassB_train, XclassB_test, YclassB_train, YclassB_test = sklearn.model_selection.train_test_split(XclassB, YclassB, test_size=0.2, random_state=42)
   
   Xclass_train = XclassA_train + XclassB_train
   Yclass_train = YclassA_train + YclassB_train
   

2. A more involved, and arguably better one, you can try first to balance your dataset. For that you can use one of many techniques (under-, over-sampling, SMOTE, AdaSYN, Tomek links, etc.). I recommend you review the methods of imbalanced-learn package. Having done balancing you can use the ordinary test/train split using typical methods without any additional intermediary steps.

The second approach is better not only from the perspective of splitting the data but also from the speed and even ability to train a model (which for heavily imbalanced datasets is not guaranteed to work).