Give tools to your LLMs using JavaScript

We’ve got recently been working on Agents.js at huggingface.js. It’s a brand new library for giving tool access to LLMs from JavaScript in either the browser or the server. It ships with just a few multi-modal tools out of the box and might easily be prolonged along with your own tools and language models.

Installation

Getting began could be very easy, you possibly can grab the library from npm with the next:

npm install @huggingface/agents

Usage

The library exposes the HfAgent object which is the entry point to the library. You possibly can instantiate it like this:

import { HfAgent } from "@huggingface/agents";

const HF_ACCESS_TOKEN = "hf_..."; 

const agent = latest HfAgent(HF_ACCESS_TOKEN);

Afterward, using the agent is straightforward. You give it a plain-text command and it would return some messages.

const code = await agent.generateCode(
  "Draw an image of a rubber duck with a top hat, then caption this picture."
);

which on this case generated the next code

async function generate() {
  const output = await textToImage("rubber duck with a top hat");
  message("We generate the duck picture", output);
  const caption = await imageToText(output);
  message("Now we caption the image", caption);
  return output;
}

Then the code will be evaluated as such:

const messages = await agent.evaluateCode(code);

The messages returned by the agent are objects with the next shape:

export interface Update {
    message: string;
    data: undefined | string | Blob;

where message is an info text and data can contain either a string or a blob. The blob will be used to display images or audio.

In the event you trust your environment (see warning), you can even run the code directly from the prompt with run :

const messages = await agent.run(
  "Draw an image of a rubber duck with a top hat, then caption this picture."
);

Usage warning

Currently using this library will mean evaluating arbitrary code within the browser (or in Node). It is a security risk and mustn’t be done in an untrusted environment. We recommend that you simply use generateCode and evaluateCode as an alternative of run in an effort to check what code you’re running.

Custom LLMs 💬

By default HfAgent will use OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 hosted Inference API because the LLM. This will be customized nevertheless.

When instancing your HfAgent you possibly can pass a custom LLM. A LLM on this context is any async function that takes a string input and returns a promise for a string. For instance if you might have an OpenAI API key you may make use of it like this:

import { Configuration, OpenAIApi } from "openai";

const HF_ACCESS_TOKEN = "hf_...";
const api = latest OpenAIApi(latest Configuration({ apiKey: "sk-..." }));

const llmOpenAI = async (prompt: string): Promise<string> => {
  return (
    (
      await api.createCompletion({
        model: "text-davinci-003",
        prompt: prompt,
        max_tokens: 1000,
      })
    ).data.decisions[0].text ?? ""
  );
};

const agent = latest HfAgent(HF_ACCESS_TOKEN, llmOpenAI);

Custom Tools 🛠️

Agents.js was designed to be easily expanded with custom tools & examples. For instance when you desired to add a tool that might translate text from English to German you may do it like this:

import type { Tool } from "@huggingface/agents/src/types";

const englishToGermanTool: Tool = {
  name: "englishToGerman",
  description:
    "Takes an input string in english and returns a german translation. ",
  examples: [
    {
      prompt: "translate the string 'hello world' to german",
      code: `const output = englishToGerman("hello world")`,
      tools: ["englishToGerman"],
    },
    {
      prompt:
        "translate the string 'The short brown fox jumps over the lazy dog` into german",
      code: `const output = englishToGerman("The short brown fox jumps over the lazy dog")`,
      tools: ["englishToGerman"],
    },
  ],
  call: async (input, inference) => {
    const data = await input;
    if (typeof data !== "string") {
      throw latest Error("Input should be a string");
    }
    const result = await inference.translation({
      model: "t5-base",
      inputs: input,
    });
    return result.translation_text;
  },
};

Now this tool will be added to the list of tools when initiating your agent.

import { HfAgent, LLMFromHub, defaultTools } from "@huggingface/agents";

const HF_ACCESS_TOKEN = "hf_...";

const agent = latest HfAgent(HF_ACCESS_TOKEN, LLMFromHub("hf_..."), [
  englishToGermanTool,
  ...defaultTools,
]);

Passing input files to the agent 🖼️

The agent also can take input files to pass along to the tools. You possibly can pass an optional FileList to generateCode and evaluateCode as such:

If you might have the next html:

<input id="fileItem" type="file" />

Then you definitely can do:

const agent = latest HfAgent(HF_ACCESS_TOKEN);
const files = document.getElementById("fileItem").files; 
const code = agent.generateCode(
  "Caption the image after which read the text out loud.",
  files
);

Which generated the next code when passing a picture:

async function generate(image) {
  const caption = await imageToText(image);
  message("First we caption the image", caption);
  const output = await textToSpeech(caption);
  message("Then we read the caption out loud", output);
  return output;
}

Demo 🎉

We have been working on a demo for Agents.js you can check out here. It’s powered by the identical Open Assistant 30B model that we use on HuggingChat and uses tools called from the hub. 🚀