Snowflake | Generative AI | Large Language Model

Snowflake, has consistently been a trailblazer in the realm of artificial intelligence. Today, they unveiled their flagship generative AI model, Snowflake Arctic, marking a significant milestone for the team and signaling their foray into the generative artificial intelligence domain.

Snowflake Arctic is crafted with a distinct emphasis on transforming enterprise AI, with the goal of redefining the parameters of cost-effective training and openness. This model establishes a fresh benchmark for large language models (LLMs), with a pledge to reshape the AI landscape for enterprise applications.

As the newest creation from the esteemed Snowflake AI Research Team, Snowflake Arctic marks a pivotal juncture in Snowflake’s progression into the domain of generative artificial intelligence. Customized to meet the demands of enterprises, this model prioritizes efficiency, intelligence, and openness, poised to make a substantial impact within the AI community and beyond.

A Costly Dilemma

Snowflake acknowledged the challenges of developing high-quality enterprise intelligence using Large Language Models (LLMs), which has historically been a demanding and costly undertaking. The traditional training and inference processes for LLMs have necessitated substantial resource investments, often amounting to tens or hundreds of millions of dollars. Over the years, researchers have grappled with the difficulties of efficiently training and inferring LLMs within these constraints.

The Birth of Snowflake Arctic

Snowflake states that their Snowflake AI Research team pioneered groundbreaking systems such as ZeRO, DeepSpeed, PagedAttention/vLLM, and LLM360. These innovations significantly reduced the cost of LLM training and inference, making LLMs more accessible and cost-effective for the community. This led to the birth of Snowflake Arctic, a top-tier enterprise-focused LLM that revolutionizes cost-effective training and openness.

Efficiency and Openness

In its recent announcement, Snowflake praises Arctic for its outstanding efficiency and openness. The platform exhibits exceptional proficiency in handling enterprise tasks such as SQL generation, coding, and instruction following benchmarks, even surpassing open-source models trained with significantly higher compute budgets. This accomplishment sets a new benchmark for cost-effective training, empowering Snowflake customers to create high-quality custom models for their enterprise needs at a reduced cost. Additionally, the Apache 2.0 license provides unrestricted access to weights and code, ensuring that Snowflake Arctic is genuinely open and accessible to all.

Availability and Accessibility

Snowflake Arctic is now accessible through various channels, including Hugging Face, Snowflake Cortex, Amazon Web Services (AWS), Microsoft Azure, NVIDIA API catalog, Lamini, Perplexity, Replicate, and Together. This availability ensures that top-tier enterprise intelligence is accessible at an incredibly low training cost.

The Enterprise Intelligence Metric

At Snowflake, a consistent pattern in AI needs and use cases from enterprise customers has been observed. Snowflake Arctic is designed to meet these needs, excelling at SQL, code, complex instruction following, and the ability to produce grounded answers. This amalgamation of abilities is captured into a single metric called enterprise intelligence, representing the model’s prowess.

The Unique Architecture

Snowflake Arctic attains exceptional training efficiency through its distinctive Dense-MoE Hybrid transformer architecture. This innovative approach merges a 10B dense transformer model with a residual 128×3.66B MoE MLP, resulting in a total of 480B parameters, with 17B active parameters chosen using a top-2 gating. Inference efficiency is equally critical, and Snowflake Arctic represents a significant advancement in MoE model scale, employing more experts and total parameters than any other open-sourced auto-regressive MoE model.

The Commitment to Openness

Built upon the collective experiences of a diverse team and major insights from the community, Snowflake Arctic embodies the spirit of open collaboration. The commitment to a truly open ecosystem goes beyond open weights and code, extending to open research insights and open source recipes.

Key Points and Future Developments

• Snowflake Arctic is releasing model checkpoints for both the base and instruct-tuned versions under an Apache 2.0 license, allowing for their free use in research, prototypes, and products.
• The model’s LoRA-based fine-tuning pipeline, in collaboration with NVIDIA TensorRT-LLM and vLLM, is paving the way for efficient model tuning and inference implementations.
• Snowflake Arctic’s continuous development includes the expansion of its attention window to support unlimited sequence generation capability in the near future.

Key Advantages

• Efficiently Intelligent: Snowflake Arctic excels at enterprise tasks such as SQL generation, coding, and instruction following benchmarks, setting a new baseline for cost-effective training and enabling Snowflake customers to create high-quality custom models for their enterprise needs at a low cost.
• Truly Open: The model is available under the Apache 2.0 license, providing ungated access to weights and code. Additionally, Snowflake is open sourcing all of its data recipes and research insights, furthering the frontiers of what open source AI can do.

About Snowflake Arctic

Snowflake Arctic is a significant milestone for Snowflake, marking its entry into the generative artificial intelligence arena. The model is designed to compete with other leading LLMs and is aimed at providing industry-leading intelligence and efficiency in a truly open way to the AI community. Snowflake Arctic is the result of the collective experiences of the Snowflake AI Research Team and major insights and learnings from the community. It is built upon open-source code and research insights, reflecting Snowflake’s commitment to a truly open ecosystem.