- Use of appropriate evaluation metrics<\/li><\/ol>\n\n\n\n
Rather than being part of a method to directly solve the imbalanced dataset problem, it can be said that it is the first step to accurately interpret and understand the currently trained model and apply the solution that will be described later. For example, suppose we have a problem of binary classification of labels 0 and 1, and the proportion of samples belonging to label 0 to the entire dataset is 99% and the proportion of samples belonging to label 1 is 1%. If the trained model classifies all data as 0, the accuracy of this model will be 99%. Although this accuracy is not an incorrect indicator, can this 99% performance indicator properly tell the performance of this model? In general, we want to classify 1s correctly, not 0s in these data. If so, this indicator would not be worthwhile. Therefore, it is recommended to use the following evaluation index [1], which can see not only accuracy but also various aspects.<\/p>\n\n\n\n
![\"\"](\"https:\/\/smilegate.ai\/wp-content\/uploads\/2021\/06\/\uc2a4\ud06c\ub9b0\uc0f7-2021-06-24-\uc624\ud6c4-7.22.47-1024x222.png\")
<\/figure>\n\n\n\n- Precision: True Positive \/ (True Positive + False Positive); The percentage of what the model classifies as true that are actually true (fit)<\/li>
- Recall: True Positive \/ (True Positive + False Negative); Proportion of (fit) the model predicted to be true out of what is actually true<\/li>
- F1 score: 2 * Precision * Recall \/ (Precision + Recall); Harmonic mean of precision and recall<\/li>
- ACU: refers to the area under the ROC curve, a graph showing the performance of a classification model at all thresholds<\/li><\/ul>\n\n\n\n
Among the above indicators, it can be seen that the precision and recall for the label with a small ratio are very low. In other words, it has not been well-trained for that label.<\/p>\n\n\n\n
<\/div>\n\n\n\n- Sampling the training dataset<\/li><\/ol>\n\n\n\n
If it is determined that the model has not been trained well through the above evaluation indicators, this is the first strategy that can be applied simply. You can solve the imbalance problem by matching the ratio between labels.<\/p>\n\n\n\n
- Under-sampling [2]: Creating a balanced dataset by reducing the number of samples that occupy a large proportion, a method that can be used when the amount of data is sufficient.<\/li>
- Over-sampling [3]: Creating a balanced dataset by increasing the number of samples occupying a small proportion, a method that can be used when the amount of data is insufficient. (ex. repetition, bootstrapping, SMOTE, ROSE)<\/li><\/ul>\n\n\n\n
A proper combination of the above two methods [4] will lead to a better sampler. In addition, the Imbalanced Dataset Sampler[5] applicable to PyTorch, which is commonly used as a deep learning framework, has been released, so it would be good to refer to it. However, if the data belonging to the label you want to classify is extremely small and not enough to learn, the sampling method alone may not be able to solve it.<\/p>\n\n\n\n
<\/div>\n\n\n\n- Data Augmentation<\/li><\/ol>\n\n\n\n
This method can be used when the number of data is extremely small. However, whether and how to apply this data augmentation method will differ depending on the task depending on the domain. <\/p>\n\n\n\n
- Image Augmentation<\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/smilegate.ai\/wp-content\/uploads\/2021\/06\/Enlarge-your-Dataset.png\")
<\/figure>\n\n\n\n- Text Augmentation<\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/smilegate.ai\/wp-content\/uploads\/2021\/06\/Operation-Sentence-1.png\")
<\/figure>\n\n\n\nThe above example is a data augmentation technique that is universally applied to image and text data. If you use the samples with a label with a low ratio rather than a label with a high ratio already, you can increase the performance by augmenting them.<\/p>\n\n\n\n
<\/div>\n\n\n\n- Using loss functions for unbalanced datasets<\/li><\/ol>\n\n\n\n
Finally, to use a loss function that fits unbalanced data. One of the most popular loss functions for unbalanced data is Focal Loss[6]. In addition to this, various loss functions exist, and there is also LADE Loss[7], which will be introduced in CVPR 2021.<\/p>\n\n\n\n
- Focal Loss: Solved by weighting the loss value based on the classification error<\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/smilegate.ai\/wp-content\/uploads\/2021\/06\/FL.png\")
<\/figure>\n\n\n\nLADE Loss: Solved by distributing the label distribution of the unbalanced training data to the label distribution of the target data.<\/p>\n\n\n\n
![\"\"](\"https:\/\/smilegate.ai\/wp-content\/uploads\/2021\/06\/\uc2a4\ud06c\ub9b0\uc0f7-2021-06-24-\uc624\ud6c4-7.20.54.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/smilegate.ai\/wp-content\/uploads\/2021\/06\/\uc2a4\ud06c\ub9b0\uc0f7-2021-06-24-\uc624\ud6c4-7.21.14.png\")
<\/figure>\n\n\n\nA more fundamental solution than the described method is to acquire more datasets. However, the cost of collecting the appropriate dataset and labeling is significant. If the amount of data set used for the problem you are trying to solve is not sufficient enough to train the machine learning model, all of the above methods may not solve it. If the dataset is sufficient, but highly imbalanced, the above method may be applied to drive performance. The last method, the method using the loss function, will be shared as code and I will finish the post.<\/p>\n\n\n\n
https:\/\/github.com\/Joonsun-Hwang\/imbalance-loss-test\/blob\/main\/Loss%20Test.ipynb<\/a><\/p>\n\n\n\n
<\/div>\n\n\n\n
\n\n\n\n[1] https:\/\/en.wikipedia.org\/wiki\/Precision_and_recall<\/a>
[2] https:\/\/imbalanced-learn.org\/stable\/under_sampling.html<\/a>
[3] https:\/\/imbalanced-learn.org\/stable\/over_sampling.html<\/a>
[4] https:\/\/imbalanced-learn.org\/stable\/combine.html<\/a>
[5] https:\/\/github.com\/ufoym\/imbalanced-dataset-sampler<\/a>
[6] https:\/\/arxiv.org\/abs\/1708.02002<\/a>
[7] https:\/\/github.com\/hyperconnect\/LADE<\/a><\/p>\n<\/span><\/div>","protected":false},"excerpt":{"rendered":"[\uc11c\ube44\uc2a4\uac1c\ubc1c\ud300 \ud669\uc900\uc120] \uae30\uacc4\ud559\uc2b5 \ubaa8\ub378\uc744 \uc9c0\ub3c4 \ud559\uc2b5\ud560 \ub54c \ub77c\ubca8\uac04 \ub370\uc774\ud130\uc758 \uac1c\uc218\uac00 \ubd88\uade0\ud615\ud55c \ub370\uc774\ud130\uc14b\uc744 \ud6c8\ub828 \ub370\uc774\ud130\ub85c \uc0bc\uc744 \uacbd\uc6b0, \ube44\uc728\uc774 \uc791\uc740 \ub77c\ubca8\uc5d0 \uc18d\ud55c \uc0d8\ud50c\ub4e4\uc5d0 \ub300\ud55c \ud559\uc2b5\uc774 \uc798 \uc774\ub8e8\uc5b4\uc9c0\uc9c0 \uc54a\ub294 \ud604\uc0c1\uc744 \uacaa\uac8c \ub429\ub2c8\ub2e4. \ub2e8\uc21c\ud788 \uc0d8\ud50c\uc758 \uac1c\uc218 \uc790\uccb4\uac00 \uc801\ub2e4\uba74 \ub2f9\uc5f0\ud788 \ud559\uc2b5\uc774 \uc798 \uc774\ub8e8\uc5b4\uc9c0\uc9c0 \uc54a\uc744 \uac83\uc774\uba70, \uc0d8\ud50c\uc774 \ucda9\ubd84\ud788 \ud559\uc2b5\ud560\ub9cc\ud07c\uc740 \uc788\ub2e4\uace0 \ud558\ub354\ub77c\ub3c4, \ube44\uc728 \ucc28\uc774\uac00 \uadf9\uc2ec\ud558\ub2e4\uba74 \ubaa8\ub378\uc740 \ud3b8\ud5a5\uc131\uc744 \uac00\uc9c0\uac8c \ub420 \uac83\uc785\ub2c8\ub2e4. \ud2b9\ud788, \uc608\ub97c…<\/p>\n
<\/span><\/div>","protected":false},"author":1,"featured_media":60406,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[19,196,197],"tags":[],"class_list":["post-60394","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech04","category-code","category-data","category-19","category-196","category-197","description-off"],"_links":{"self":[{"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/posts\/60394","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/comments?post=60394"}],"version-history":[{"count":6,"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/posts\/60394\/revisions"}],"predecessor-version":[{"id":60411,"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/posts\/60394\/revisions\/60411"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/media\/60406"}],"wp:attachment":[{"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/media?parent=60394"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/categories?post=60394"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/smilegate.ai\/en\/wp-json\/wp\/v2\/tags?post=60394"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Focal Loss: Solved by weighting the loss value based on the classification error<\/li><\/ul>\n\n\n\n
- Using loss functions for unbalanced datasets<\/li><\/ol>\n\n\n\n
- Text Augmentation<\/li><\/ul>\n\n\n\n
- Image Augmentation<\/li><\/ul>\n\n\n\n
- Data Augmentation<\/li><\/ol>\n\n\n\n
- Sampling the training dataset<\/li><\/ol>\n\n\n\n