Meet MosaicBERT: A BERT-Style Encoder Architecture and Training Recipe that is Empirically Optimized for Fast Pretraining

BERT is a language mannequin which was launched by Google in 2018. It’s primarily based on the transformer structure and is thought for its vital enchancment over earlier state-of-the-art fashions. As such, it has been the powerhouse of quite a few pure language processing (NLP) functions since its inception, and even within the age of huge language fashions (LLMs), BERT-style encoder fashions are utilized in duties like vector embeddings and retrieval augmented technology (RAG). Nonetheless, previously half a decade, many vital developments have been made with different kinds of architectures and coaching configurations which have but to be integrated into BERT.

On this analysis paper, the authors have proven that pace optimizations may be integrated into the BERT structure and coaching recipe. For this, they’ve launched an optimized framework known as MosaicBERT that improves the pretraining pace and accuracy of the basic BERT structure, which has traditionally been computationally costly to coach.

To construct MosaicBERT, the researchers used totally different architectural selections corresponding to FlashAttention, ALiBi, coaching with dynamic unpadding, low-precision LayerNorm, and Gated Linear Items.

• The flashAttention layer reduces the variety of learn/write operations between the GPU’s long-term and short-term reminiscence.
• ALiBi encodes place data by way of the eye operation, eliminating the place embeddings and performing as an oblique speedup technique.
• The researchers modified the LayerNorm modules to run in bfloat16 precision as a substitute of float32, which reduces the quantity of knowledge that must be loaded from reminiscence from 4 bytes per factor to 2 bytes.
• Lastly, the Gated Linear Items improves the Pareto efficiency throughout all timescales.

The researchers pretrained BERT-Base and MosaicBERT-Base for 70,000 steps of batch measurement 4096 after which finetuned them on the GLUE benchmark suite. BERT-Base reached a mean GLUE rating of 83.2% in 11.5 hours, whereas MosaicBERT achieved the identical accuracy in round 4.6 hours on the identical {hardware}, highlighting the numerous speedup. MosaicBERT additionally outperforms the BERT mannequin in 4 out of eight GLUE duties throughout the coaching length.

The massive variant of MosaicBERT additionally had a big speedup over the BERT variant, reaching a mean GLUE rating of 83.2 in 15.85 hours in comparison with 23.35 hours taken by BERT-Massive. Each the variants of MosaicBERT are Pareto Optimum relative to the corresponding BERT fashions. The outcomes additionally present that the efficiency of BERT-Massive surpasses the bottom mannequin solely after intensive coaching.

In conclusion, the authors of this analysis paper have improved the pretraining pace and accuracy of the BERT mannequin utilizing a mixture of architectural selections corresponding to FlashAttention, ALiBi, low-precision LayerNorm, and Gated Linear Items. Each the mannequin variants had a big speedup in comparison with their BERT counterparts by reaching the identical GLUE rating in much less time on the identical {hardware}. The authors hope their work will assist researchers pre-train BERT fashions sooner and cheaper, in the end enabling them to construct higher fashions.

Take a look at the Paper. All credit score for this analysis goes to the researchers of this mission. Additionally, don’t neglect to comply with us on Twitter. Be a part of our 35k+ ML SubReddit, 41k+ Fb Neighborhood, Discord Channel, and LinkedIn Group.

When you like our work, you’ll love our publication..