Fork me on GitHub

LLM.js — Simple LLM library for Node.js

LLM.js

GitHub Repo stars NPM Downloads GitHub code size in bytes GitHub License


LLM.js is the fastest way to use Large Language Models in JavaScript (Node.js and Web). It’s a single simple interface to hundreds of popular LLMs:

await LLM("the color of the sky is", { model: "gpt-4" }); // blue

Features

Install

Install LLM.js from NPM:

npm install @themaximalist/llm.js

Setting up LLMs is easy—just make sure your API key is set in your environment

export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export MISTRAL_API_KEY=...
export GOOGLE_API_KEY=...
export GROQ_API_KEY=...
export TOGETHER_API_KEY=...
export PERPLEXITY_API_KEY=...
export DEEPSEEK_API_KEY=...

For local models like llamafile and Ollama, ensure an instance is running.

Usage

The simplest way to call LLM.js is as an async function.

const LLM = require("@themaximalist/llm.js");
await LLM("hello"); // Response: hi

This fires a one-off request, and doesn’t store any history.

Chat

Initialize an LLM instance to build up message history.

const llm = new LLM();
await llm.chat("what's the color of the sky in hex value?"); // #87CEEB
await llm.chat("what about at night time?"); // #222d5a

Streaming

Streaming provides a better user experience by returning results immediately, and it’s as simple as passing {stream: true} as an option.

const stream = await LLM("the color of the sky is", { stream: true });
for await (const message of stream) {
    process.stdout.write(message);
}

Sometimes it’s helpful to handle the stream in real-time and also process it once it’s all complete. For example, providing real-time streaming in chat, and then parsing out semantic code blocks at the end.

LLM.js makes this easy with an optional stream_handler option.

const colors = await LLM("what are the common colors of the sky as a flat json array?", {
  model: "gpt-4o-mini",
  stream: true,
  stream_handler: (c) => process.stdout.write(c),
  parser: LLM.parsers.json,
});
// ["blue", "gray", "white", "orange", "red", "pink", "purple", "black"]

Instead of the stream being returned as a generator, it’s passed to the stream_handler. The response from LLM.js is the entire response, which can be parsed or handled as normal.

JSON

LLM.js supports JSON schema for OpenAI and LLaMa. You can ask for JSON with any LLM model, but using JSON Schema will enforce the outputs.

const schema = {
    "type": "object",
    "properties": {
        "colors": { "type": "array", "items": { "type": "string" } }
    }
}

const obj = await LLM("what are the 3 primary colors in JSON format?", { schema, temperature: 0.1, service: "openai" });

Different formats are used by different models (JSON Schema, BNFS), so LLM.js converts between these automatically.

Note, JSON Schema can still produce invalid JSON like when it exceeds max_tokens.

System Prompts

Create agents that specialize at specific tasks using llm.system(input).

const llm = new LLM();
llm.system("You are a friendly chat bot.");
await llm.chat("what's the color of the sky in hex value?"); // Response: sky blue
await llm.chat("what about at night time?"); // Response: darker value (uses previous context to know we're asking for a color)

Note, OpenAI has suggested system prompts may not be as effective as user prompts, which LLM.js supports with llm.user(input).

Message History

LLM.js supports simple string prompts, but also full message history. This is especially helpful to guide LLMs in a more precise way.

await LLM([
    { role: "user", content: "remember the secret codeword is blue" },
    { role: "assistant", content: "OK I will remember" },
    { role: "user", content: "what is the secret codeword I just told you?" },
]); // Response: blue

The OpenAI message format is used, and converted on-the-fly for specific services that use a different format (like Google, Mixtral and LLaMa).

Switch LLMs

LLM.js supports most popular Large Lanuage Models, including

LLM.js can guess the LLM provider based on the model, or you can specify it explicitly.

// defaults to Llamafile
await LLM("the color of the sky is");

// OpenAI
await LLM("the color of the sky is", { model: "gpt-4o-mini" });

// Anthropic
await LLM("the color of the sky is", { model: "claude-3-5-sonnet-latest" });

// Mistral AI
await LLM("the color of the sky is", { model: "mistral-tiny" });

// Groq needs an specific service
await LLM("the color of the sky is", { service: "groq", model: "mixtral-8x7b-32768" });

// Google
await LLM("the color of the sky is", { model: "gemini-pro" });

// Ollama
await LLM("the color of the sky is", { model: "llama2:7b" });

// Together
await LLM("the color of the sky is", { service: "together", model: "meta-llama/Llama-3-70b-chat-hf" });

// DeepSeek
await LLM("the color of the sky is", { service: "deepseek", model: "deepseek-chat" });

// Can optionally set service to be specific
await LLM("the color of the sky is", { service: "openai", model: "o1-preview" });

Being able to quickly switch between LLMs prevents you from getting locked in.

Parsers

LLM.js ships with a few helpful parsers that work with every LLM. These are separate from the typical JSON formatting with tool and schema that some LLMs (like from OpenAI) support.

JSON Parsing

const colors = await LLM("Please return the primary colors in a JSON array", {
  parser: LLM.parsers.json
});
// ["red", "green", "blue"]

Markdown Code Block Parsing

const story = await LLM("Please return a story wrapped in a Markdown story code block", {
  parser: LLM.parsers.codeBlock("story")
});
// A long time ago...

XML Parsing

const code = await LLM("Please write a simple website, and put the code inside of a <WEBSITE></WEBSITE> xml tag" {
  parser: LLM.parsers.xml("WEBSITE")                       
});
// <html>....

Note: OpenAI works best with Markdown and JSON, while Anthropic works best with XML tags.

API

The LLM.js API provides a simple interface to dozens of Large Language Models.

new LLM(input, {        // Input can be string or message history array
  service: "openai",    // LLM service provider
  model: "gpt-4",       // Specific model
  max_tokens: 100,      // Maximum response length
  temperature: 1.0,     // "Creativity" of model
  seed: 1000,           // Stable starting point
  stream: false,        // Respond in real-time
  stream_handler: null, // Optional function to handle stream
  schema: { ... },      // JSON Schema
  tool: { ...  },       // Tool selection
  parser: null,         // Content parser
});

The same API is supported in the short-hand interface of LLM.js—calling it as a function:

await LLM(input, options);

Input (required)

Options

All config parameters are optional. Some config options are only available on certain models, and are specified below.

Public Variables

Methods

async send(options=<object>)

Sends the current Message History to the current LLM with specified options. These local options will override the global default options.

Response will be automatically added to Message History.

await llm.send(options);

async chat(input=<string>, options=<object>)

Adds the input to the current Message History and calls send with the current override options.

Returns the response directly to the user, while updating Message History.

const response = await llm.chat("hello");
console.log(response); // hi

abort()

Aborts an ongoing stream. Throws an AbortError.

user(input=<string>)

Adds a message from user to Message History.

llm.user("My favorite color is blue. Remember that");

system(input=<string>)

Adds a message from system to Message History. This is typically the first message.

llm.system("You are a friendly AI chat bot...");

assistant(input=<string>)

Adds a message from assistant to Message History. This is typically a response from the AI, or a way to steer a future response.

llm.user("My favorite color is blue. Remember that");
llm.assistant("OK, I will remember your favorite color is blue.");

Static Variables

Static Methods

serviceForModel(model)

Return the LLM service for a particular model.

LLM.serviceForModel("gpt-4o-mini"); // openai

modelForService(service)

Return the default LLM for a service.

LLM.modelForService("openai"); // gpt-4o-mini
LLM.modelForService(LLM.OPENAI); // gpt-4o-mini

Response

LLM.js returns results from llm.send() and llm.chat(), typically the string content from the LLM completing your prompt.

await LLM("hello"); // "hi"

But when you use schema and tools — LLM.js will typically return a JSON object.

const tool = {
    "name": "generate_primary_colors",
    "description": "Generates the primary colors",
    "parameters": {
        "type": "object",
        "properties": {
            "colors": {
                "type": "array",
                "items": { "type": "string" }
            }
        },
        "required": ["colors"]
    }
};

await LLM("what are the 3 primary colors in physics?");
// { colors: ["red", "green", "blue"] }

await LLM("what are the 3 primary colors in painting?");
// { colors: ["red", "yellow", "blue"] }

And by passing {stream: true} in options, LLM.js will return a generator and start yielding results immediately.

const stream = await LLM("Once upon a time", { stream: true });
for await (const message of stream) {
    process.stdout.write(message);
}

The response is based on what you ask the LLM to do, and LLM.js always tries to do the obviously right thing.

Message History

The Message History API in LLM.js is the exact same as the OpenAI message history format.

await LLM([
    { role: "user", content: "remember the secret codeword is blue" },
    { role: "assistant", content: "OK I will remember" },
    { role: "user", content: "what is the secret codeword I just told you?" },
]); // Response: blue

Options

LLM Command

LLM.js provides a useful llm command for your shell. llm is a convenient way to call dozens of LLMs and access the full power of LLM.js without programming.

Access it globally by installing from NPM

npm install @themaximalist/llm.js -g

Then you can call the llm command from anywhere in your terminal.

> llm the color of the sky is
blue

Messages are streamed back in real time, so everything is really fast.

You can also initiate a --chat to remember message history and continue your conversation (Ctrl-C to quit).

> llm remember the codeword is blue. say ok if you understand --chat
OK, I understand.

> what is the codeword?
The codeword is blue.

Or easily change the LLM on the fly:

> llm the color of the sky is --model claude-3-haiku-20240307
blue

See help with llm --help

Usage: llm [options] [input]

Large Language Model library for OpenAI, Google, Anthropic, Mistral, Groq and LLaMa

Arguments:
  input                       Input to send to LLM service

Options:
  -V, --version               output the version number
  -m, --model <model>         Completion Model (default: llamafile)
  -s, --system <prompt>       System prompt (default: "I am a friendly accurate English speaking chat bot") (default: "I am a friendly accurate English speaking chat bot")
  -t, --temperature <number>  Model temperature (default 0.8) (default: 0.8)
  -c, --chat                  Chat Mode
  -h, --help                  display help for command

Debug

LLM.js and llm use the debug npm module with the llm.js namespace.

View debug logs by setting the DEBUG environment variable.

> DEBUG=llm.js* llm the color of the sky is
# debug logs
blue
> export DEBUG=llm.js*
> llm the color of the sky is
# debug logs
blue

Examples

LLM.js has lots of tests which can serve as a guide for seeing how it’s used.

Changelog

LLM.js has been under heavy development while LLMs are rapidly changing. We’ve started to settle on a stable interface, and will document changes here.

Projects

LLM.js is currently used in the following projects:

License

MIT

Author

Created by The Maximalist, see our open-source projects.