Skip to main content

LiteLLM

LiteLLM is Python SDK that allows you to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq].

This guide shows you how to instrument LiteLLM applications using Agenta's observability features.

Installation​

Install the required packages:

pip install -U agenta litellm

Configure Environment Variables​

import os

os.environ["AGENTA_API_KEY"] = "YOUR_AGENTA_API_KEY"
os.environ["AGENTA_HOST"] = "https://cloud.agenta.ai"

Code Example​

import agenta as ag
import litellm
import asyncio

ag.init()

@ag.instrument()
async def agenerate_completion():
    litellm.callbacks = [ag.callbacks.litellm_handler()]

    # Define the messages for the chat completion
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a short story about AI Engineering."},
    ]
    temperature = 0.2
    max_tokens = 100

    # Make an asynchronous chat completion request
    chat_completion = await litellm.acompletion(
        model="gpt-3.5-turbo",
        messages=messages,
        temperature=temperature,
        max_tokens=max_tokens,
    )
    print(chat_completion)

# Run the asynchronous function
asyncio.run(agenerate_completion())

Explanation​

  • Initialize Agenta: ag.init() sets up the Agenta SDK.
  • Instrument the Function: The @ag.instrument() decorator wraps the agenerate_completion function, creating a parent span in Agenta.
  • Set Up Callback: litellm.callbacks = [ag.callbacks.litellm_handler()] sets the Agenta callback handler for LiteLLM. This enables LiteLLM to send trace data to Agenta.