DocsExperimentation

Experimentation (releases & versions)

You can track the effect of changes to your LLM app on metrics in Langfuse. This allows you to:

  • Run experiments (A/B tests) in production and measure the impact on costs, latencies and quality.
    • Example: “What is the impact of switching to a new model?”
  • Explain changes to metrics over time.
    • Example: “Why did latency in this chain increase?”

Releases

A release tracks the overall version of your application. Commonly it is set to the semantic version or git commit hash of your application.

The SDKs look for a release in the following order:

  1. SDK initialization
  2. Environment variable
  3. Automatically on popular platforms

SDK initialization

Decorators

from langfuse.decorators import langfuse_context, observe
 
@observe()
def fn():
    langfuse_context.update_current_trace(
        release="<release_tag>"
    )
 
fn()

Low-level SDK

from langfuse import Langfuse
 
langfuse = Langfuse(
  release="<release_tag>"
)

Via environment variable

The SDKs will look for a LANGFUSE_RELEASE environment variable. Use it to configure the release e.g. in your CI/CD pipeline.

LANGFUSE_RELEASE = "<release_tag>" # <- github sha or other identifier

If no other release is set, the Langfuse SDKs default to a set of known release environment variables.

Supported platforms include: Vercel, Heroku, Netlify. See the full list of support environment variables for JS/TS and Python.

Versions

The version parameter can be added to traces and all observation types (span, generation, event). Thereby, you can track the effect of a new version on the metrics of an object with a specific name using Langfuse analytics.

Decorators

from langfuse.decorators import langfuse_context, observe
 
@observe()
def fn():
    # trace level
    langfuse_context.update_current_trace(
        version="1.0",
    )
 
    # observation level
    langfuse_context.update_current_observation(
        version="1.0",
    )
fn()

Low-level SDK

langfuse.generation(
  name="guess-countries",
  version="1.0",
)

langfuse.trace(), langfuse.span() and langfuse.event() also take an optional version parameter.

Version parameter in Langfuse interface

Version on single generation

Was this page useful?

Questions? We're here to help

Subscribe to updates