What if I told you that you can save 60% or more off of the associated fee of your LLM API spending without compromising on accuracy? Surprisingly, now you may.

Large Language Models (LLMs) are actually a part of our on a regular basis lives. Firms use the technology to automate processes, improve customer experiences, construct higher products, get monetary savings, and more.

Hosting your individual LLMs may be very difficult. They provide broad capabilities but are sometimes expensive to run. They often require complex infrastructure and large amounts of information. Cost and complexity are why you employ prompt engineering. It’s possible you’ll even use retrieval-augmented generation (RAG) to enhance context and reduce hallucinations. With each techniques, you offload running LLMs to the likes of OpenAI, Cohere, or Google. Yet, scaling LLM adoption to latest use cases, especially with the newest powerful models, can drive up a latest cost that was previously unaccounted for. Weaker models could also be cheaper, but are you able to trust them with complex questions? Now, latest research shows us easy methods to get monetary savings and get nearly as good, sometimes higher, LLM results.

Get to Know LLM Cascades

Within the seek for lower LLM costs, researchers turned to the concept of LLM Cascades. In the dead of night ages, before the launch of ChatGPT, a team from Google and The University of Toronto defined this term as programs that use probability calculations to get the most effective results using multiple LLMs.

More recently, the FrugalGPT paper defined cascades as sending a user query to an inventory of LLMs, one after the opposite, from weaker to stronger LLMs, until the reply is sweet enough. FrugalGPT Cascades uses a dedicated model to find out when the reply is sweet enough against a top quality threshold.

A recent paper titled ‘Large Language Model Cascades With Mixture of Thought Representations for Cost-Efficient Reasoning’ from George Mason University, Microsoft, and Virginia Tech offers an alternate: a function that may determine whether the reply is sweet enough without fine-tuning one other model.

Mixture of Thought LLM Cascades

As an alternative of using several LLMs, ‘Mixture of thought’ (MoT) reasoning uses just two — GPT 3.5 Turbo and GPT 4. The previous model is thought to be the ‘weaker’ LLM, while the latter is the ‘strong’ LLM. The authors harnessed LLM ‘answer consistency’ to flag whether an LLM’s response is sweet enough. LLMs produce consistent answers to similar prompts once they are confident the answers are correct. Due to this fact, when weaker LLM answers are consistent, there isn’t a must call the stronger LLM. Conversely, these LLMs produce inconsistent answers once they lack confidence. That’s once you need a stronger LLM to reply the prompt. (Note: you need to use a weaker/stronger LLM pair of your alternative as well.)

The prompts themselves use few-shot in-context prompting to enhance LLM answer quality. Such prompts guide the LLM’s response by giving examples of comparable questions and answers.

To enhance model reasoning and simplify consistency measurement, the researchers introduce a latest prompting technique for reasoning tasks by ‘mixing’ two prompting techniques:

• Chain of Thought (CoT) Prompting encourages LLMs to generate intermediate steps or reasonings before arriving at a final answer. Generating these steps helps the model improve complicated task results. It also increases answer accuracy.
• Program of Thought (PoT) extends Chain of Thought prompting and uses the model’s output as a latest input for further prompts. Prompts using this system often request the model to reply with code as a substitute of human language.

The paper also introduces two methods to find out answer consistency:

• Voting: This method samples multiple answers from LLM queries with similar prompts or by various the response temperature option. It then measures how similar the LLM’s answers are to one another. The reply that agrees essentially the most with all the opposite answers is assumed to be correct. The team also defined a versatile ‘threshold’ value that aligns answer consistency and budget constraints.
• Verification: This approach compares the LLM’s most consistent answers across two distinct thought representations (e.g., CoT and PoT). The algorithm accepts the weaker LLM’s answer if the 2 prompt responses are equivalent.

Since voting requires multiple prompts, it could be more suitable when a budget exists to guide the edge number.

The Bottom Line: Mixture of Thought Saves You Money

Let’s take a look at how much money the MoT technique saves and its impact on answer accuracy.

The researchers used the next sum to calculate prompt cost:

• The associated fee of prompting the weaker model (because we may prompt it several times)
• The associated fee of the reply evaluation process
• If the evaluation process rejects the reply, we add the associated fee of prompting the strong model

The outcomes were dramatic:

• Using MoT variants — combining voting and verification with CoT and PoT — can result in comparable performance at 40% of the associated fee of solely using GPT-4.
• In testing against the CREPE Q&A dataset, MoT outperformed GPT-4 at 47% of its cost.
• Mixing PoT and CoT improves decision-making in comparison with using one in all the techniques alone.
• Increasing the edge when using the voting method didn’t significantly impact quality despite the extra cost.
• The consistency model proved itself in reliably identifying correct LLM answers. It successfully predicted when to resort to using the strong model to acquire the optimal results.

Hosting and managing Large Language Models (LLMs) in-house comes with significant challenges. They bring about complexity, high costs, and the necessity for extensive infrastructure and data resources. In consequence, LLMs present substantial hurdles for organizations in search of to harness their broad capabilities. That will lead you to show to hosted LLMs. Yet, this approach presents firms with unexpected cost increases and budget challenges as they expand to latest use cases. That is especially evident when integrating the newest powerful models. To avoid that fate, you face a latest dilemma: Are you able to trust weaker, more cost-effective models? Are you able to overcome concerns about their accuracy in handling complex questions?

LLM Cascades with Mixture of Thought (MoT) offers two significant steps forward:

1. Substantial cost savings over exclusively using the newest models.
2. Demonstrable results on par with the newest models.

This breakthrough provides organizations with a practical and efficient approach to navigating the fragile balance between the powerful capabilities of LLMs and the imperative to administer costs effectively.

Domino Staff Software Engineer Subir Mansukhani contributed to this post.