摘要： 去年NLP領域最火的莫過於BERT了，得益於數據規模和計算力的提升，BERT在大規模語料上預訓練（Masked Language Model + Next Sentence Prediction）之後可以很好地從訓練語料中捕獲豐富的語義信息，對各項任務瘋狂屠榜。我們在對BERT進行微調之後可以很好地適用到自己的任務上。如果想深入了解BERT的運行機制，就需要去仔細地研讀一下BERT的源碼。今天這篇文章我們來看看在BERT提出大半年之後，又有哪些基於BERT的有趣的研究。
摘要： In self-supervised learning, an AI technique where the training data is automatically labeled by a feature extractor, the said extractor not uncommonly exploits low-level features (known as “shortcuts”) that cause it to ignore useful representations. In search of a technique that might help to remove those shortcuts autonomously, researchers at Google Brain developed a framework — a “lens” — that makes changes enabling self-supervised models to outperform those trained in a conventional fashion.
