Energy and Policy Considerations for Deep Learning in NLP

This study provides an in-depth analysis of the environmental and financial costs associated with training large neural network models for NLP. It quantifies the energy consumption and carbon emissions involved in training NLP models, showing that training state-of-the-art models such as BERT, GPT-2, and ELMo can result in substantial carbon emissions. For example, training a large NLP model with neural architecture search (NAS) can emit over 626,000 pounds of CO₂, equivalent to the lifetime emissions of several cars. Training modern NLP models is both computationally expensive and environmentally damaging. The authors report that training these models requires extensive GPU and TPU usage, which draws significant power. This paper highlights the disparity between the environmental costs of developing models and the performance improvements that are often minimal. 

The authors compare CO₂ emissions from training various models, such as the Transformer, BERT, ELMo, and GPT-2, alongside typical emissions benchmarks like air travel or human energy consumption. The results show that training some models results in carbon emissions comparable to transcontinental flights. The analysis also explores the energy sources used in different countries and cloud computing services. For instance, cloud providers like Google and Microsoft use a relatively high percentage of renewable energy, while other major services and regions rely more on coal and natural gas, exacerbating the environmental impact.