Salesforce has unveiled a new language model benchmark tailored specifically for evaluating AI models in customer relationship management (CRM) tasks. This benchmark aims to provide businesses with a comprehensive assessment framework focused on CRM use cases, including sales and service applications.
Developed by Salesforce’s AI research team, the CRM benchmark assesses AI systems across four critical metrics: accuracy, cost, speed, and trust and safety. Unlike previous benchmarks that may not prioritize metrics relevant to enterprise needs, such as operational costs and trustworthiness, Salesforce’s test is designed to help businesses make strategic decisions about deploying AI systems in CRM environments.
“Business organizations are increasingly leveraging AI to drive growth, efficiency, and personalized customer experiences,” said Clara Shih, CEO of Salesforce AI. “This benchmark is a dynamic framework empowering companies to evaluate AI models with a focus on balancing accuracy, cost-effectiveness, speed, and trust.”
The benchmark includes a public leaderboard where model owners can compare their AI systems' performance. At launch, OpenAI’s GPT-4 Turbo leads in accuracy for CRM tasks, while Anthropic’s Claude 3 Haiku ranks as the most cost-effective model. French AI startup Mistral’s Mixtral 8x7B tops the list for speed, showcasing the efficiency of smaller language models.
In terms of trust and safety, Google’s Gemini Pro 1.5 emerged as the highest-rated model with a safety score of 91%. Meta’s Llama 3 models also scored highly in trustworthiness, reflecting their robust safety measures.
Salesforce plans to expand the benchmark by incorporating additional CRM use cases and offering support for fine-tuned models. This initiative aims to assist enterprise leaders in optimizing their AI strategies for CRM applications, emphasizing performance, accuracy, responsibility, and cost-effectiveness.
“As AI technology evolves, it’s crucial for businesses to find the right balance of performance and responsibility to unlock AI’s full potential in driving business growth,” noted Silvio Savarese, Executive Vice President and Chief Scientist at Salesforce AI Research. “Salesforce’s CRM Language Model Benchmark represents a significant advancement in AI assessment, accelerating the adoption of next-generation AI solutions tailored for CRM-specific challenges.”
The introduction of this benchmark underscores Salesforce’s commitment to enhancing AI capabilities in CRM and advancing the industry’s standards for evaluating AI performance in business contexts.
