In language, unsupervised learning algorithms that rely on word prediction (like GPT-2 and BERT) have been extremely successful, achieving top performance on a wide array of language tasks. However, our results suggest that when faced with a new domain where the correct model priors are unknown, a large GPT-2 can learn excellent features without the need for domain-specific architectural design choices. As a consequence, we require significantly more compute in order to produce features competitive with those from top unsupervised convolutional nets. To highlight the potential of generative sequence modeling as a general purpose unsupervised learning algorithm, we deliberately use the same transformer architecture as GPT-2 in language. On JFT (300M images with 18K classes), achieved a result of We only show ImageNet linear probe accuracy for iGPT-XL since otherĮxperiments did not finish before we needed to transition to different Logistic regression on learned features (linear probe) As further proof, features from the model achieve state-of-the-art performance on a number of classification datasets and near state-of-the-art unsupervised accuracy on ImageNet. This is evidenced by the diverse range of coherent image samples it generates, even without the guidance of human provided labels. When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category. Transformer models like BERT and GPT-2 are domain agnostic, meaning that they can be directly applied to 1-D sequences of any form. Our work aims to understand and bridge this gap. However, the same broad class of models has not been successful in producing strong features for image classification. Recently, it has seen incredible success in language, as transformer models like BERT, GPT-2, RoBERTa, T5, and other variants have achieved top performance on a wide array of language tasks. Unsupervised and self-supervised learning, or learning without human-labeled data, is a longstanding challenge of machine learning.
0 kommentar(er)
