Microsoft’s deep learning language model Turing-NLG outperforms GPT-2
Microsoft has announced its deep learning language model Turing-NLG, and its impressive 17 billion parameters make it the largest language model to date. While it is not publicly available, a demo version has been released to a small group for testing purposes. Let’s see if more parameters mean better results, compared to OpenAI’s GPT-2 and NVIDIA’s Megatron-LM.
Turing Natural Language Generation, short Turing-NLG, is a new deep learning language model developed by Microsoft. It was announced on the Microsoft Research Blog.
The transformer-based model Turing-NLG can not only generate texts and summarize documents, but also respond to questions. Microsoft views this as an important task in NLP and sees it as its goal “to respond as directly, accurately, and fluently as humans can in any situation.”
In the release post, there’s even a summary created by the model itself:
Turing Natural Language Generation (T-NLG) is a 17 billion parameter language model by Microsoft that outperforms the state of the art on many downstream NLP tasks. We present a demo of the model, including its freeform generation, question answering, and summarization capabilities, to academics for feedback and research purposes. <|endoftext|>
Comparison with other language models
With 17 billion parameters, Microsoft claims T-NLG to be the largest language model ever published.
In this graphic, Microsoft demonstrates the number of parameters compared to other language models like OpenAI’s full 1.5 billion model of GPT-2, which caused a stir last year and was released in November, and Google’s ground-breaking model BERT:
Here you can see how Turing-NLG outperformed both OpenAI’s full 1.5 billion model of GPT-2 as well as NVIDIA’s Megatron-LM, which has 8.3 billion parameters:
So far, Turing-NLG has not been publicly released, and Microsoft has not announced whether this is planned. The company does, however, state that work created in Project Turing, which Turing-NLG is a part of, is “being integrated into multiple Microsoft products including Bing, Office, and Xbox.”
See the Microsoft Research Blog for further information on Turing-NLG.