Meta, a leading developer of foundational open-source large language models like Llama, anticipates a significant increase in computing power needed for future model training.
During Meta’s second-quarter earnings call, Mark Zuckerberg revealed that training Llama 4 will require ten times more computing power than Llama 3. Despite this, he emphasized the importance of building the capacity to keep up with competitors.
“The computing power needed for Llama 4 will be nearly ten times what we used for Llama 3, and future models will need even more,” Zuckerberg said. “It’s challenging to predict the trend for future generations, but I’d rather build capacity early than risk being too late, given the long lead times for new projects.”
Meta launched Llama 3 with 80 billion parameters in April and recently released an upgraded version, Llama 3.1 405B, with 405 billion parameters, making it Meta’s largest open-source model to date.
Meta’s CFO, Susan Li, mentioned that the company is planning new data center projects and building capacity for future AI model training, which is expected to increase capital expenditures in 2025. Li emphasized the importance of scalable infrastructure that can adapt to various needs, ensuring that Meta remains flexible and competitive.
Training large language models is expensive. Meta’s capital expenditures rose 33% to $8.5 billion in Q2 2024, up from $6.4 billion the previous year, driven by investments in servers, data centers, and network infrastructure. This investment is crucial as Meta aims to stay ahead in the AI race.
According to a report, OpenAI spends $3 billion on training models and an additional $4 billion on discounted server rentals from Microsoft. This highlights the immense financial commitment required in the AI industry.
“As we expand our generative AI training capacity, we’re building our infrastructure to be flexible. This will allow us to allocate training capacity to generative AI inference or our core ranking and recommendation work when it’s more valuable,” Li said during the call.
Meta also discussed its consumer-facing Meta AI usage, noting that India is the largest market for its chatbot. However, Li mentioned that the company doesn’t expect significant revenue contributions from Gen AI products soon. Despite this, the focus remains on enhancing the technology and infrastructure to support future growth and capabilities.
