AI Briefingdistilbert-base-uncased: 40% Lighter than BERT: Performing NLP Tasks with 67M Parameters
distilbert/distilbert-base-uncased
About the project
This is a Transformer-based language model that reduces size and increases inference speed by distilling knowledge from the BERT base model. It consists of 67M parameters, which is 40% of the original BERT, and was pre-trained on BookCorpus and English Wikipedia data.
It provides Fill-Mask functionality for predicting masked words by default. It is suitable for tasks requiring full context understanding, such as sentence classification, token classification, and question answering, but is not recommended for text generation. It is an uncased version specialized for English processing.
It achieves high performance on the GLUE benchmark, recording scores such as 91.3 on the SST-2 task and 89.2 on QNLI, maintaining a significant portion of the original model's capabilities. It supports major frameworks including PyTorch, TensorFlow, and JAX, and is available for commercial use under the Apache 2.0 license.
distilbert/distilbert-base-uncased
The original page has no description.
fill-mask
This introduction was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report errors, attribution issues, or removal requests via Contact.