The panelists discuss DevOps buzzwords and when and where they might have value for organizations seeking performance improvements.
Researchers at Google have developed a new deep-learning model called BigBird that allows Transformer neural networks to process sequences up to 8x longer than previously possible. Networks based on this model achieved new state-of-the-art performance levels on natural-language processing and genomics tasks.
BigBird is a new self-attention model that reduces the neural-network complexity of Transformers, allowing for training and inference using longer input sequences. By increasing sequence length up to 8x, the team wasable to achieve new state-of-the-art performance on several NLP tasks, including question-answering and document summarization.The team also used BigBird to develop a new application for Transformer models in genomic sequence representations, improving accuracy over previous models by 5 percentage points.
The Transformer has become the neural-network architecture of choice for sequence learning, especially in the NLP domain.
BigBird is a new self-attention scheme that has complexity of O, which allows for sequence lengths of up to 4,096 items. Instead of each item attending to every other item, BigBird combines three smaller attention mechanisms. First is random attention, which links each item with a small constant number of other items, chosen randomly.
He noted that although the experiments in the paper used a sequence length of 4,096, the model could handle much larger sequences of up to 16k.
Original article