Tracking Custom DSPy Optimizer

If you are building a custom DSPy optimizer, then LangWatch won’t support tracking it out of the box, but adding track to any custom optimizer is also very simple.

​

1. Initialize LangWatch DSPy with optimizer=None

Before the compilation step, explicitly provide None on the optimizer parameter to be able to track the steps manually:

langwatch.dspy.init(experiment="dspy-custom-optimizer-example", optimizer=None)

compiled_rag = my_awesome_optimizer.compile(RAG(), trainset=trainset)

​

2. Track the metric function

Either before instantiating your optimizer, or inside the compilation step, don’t forget to wrap the metric function with langwatch.dspy.track_metric so that it’s tracked:

metric = langwatch.dspy.track_metric(metric)

​

3. Track each step

Now at each step your optimizer progresses, call langwatch.dspy.log_step to capture the score at the current step index, optimizer info and predictors being used on this step evaluation:

langwatch.dspy.log_step(
    optimizer=DSPyOptimizer(
        name="MyAwesomeOptimizer",
        parameters={
            "hyperparam": 1,
        },
    ),
    index="1", # step index
    score=0.5,
    label="score",
    predictors=candidate_program.predictors(),
)

The LLM calls and examples being evaluated with be tracked automatically and logged in together with calling log_step.

​

Wrapping up

That’s it! You should see the steps of the optimizer in the LangWatch dashboard now.

For any questions or issues, feel free to contact our support, join our channel on Discord or open an issue on our GitHub.