​
Optimizing LLM Workflows in LangWatch

The Optimization Studio provides the power of DSPy optimizers to improve your LLM workflow performance. Starting from a basic setup with baseline performance, you can significantly enhance results through automated optimization techniques.

​
Getting Started with Optimization

To begin optimization:

  1. Set up your basic workflow with an LLM node
  2. Connect your dataset
  3. Add appropriate evaluators
  4. Click the “Optimize” button

​
Optimization Options (0:47)

The platform offers different optimization strategies such as:

  • Improving prompts and demonstrations with MIPROv2
  • Prompt-only optimization with MIPROv2
  • Demonstrations optimization with BootstrapFewShotWithRandomSearch

​
Configuration Settings (1:07)

Key optimization parameters include:

  • Number of prompts to generate
  • Number of demonstrations to bootstrap
  • Teacher LLM selection (can use a more powerful LLM to teach a cheaper one)
  • Optimization budget and constraints

​
Monitoring Optimization Progress (1:55)

During optimization:

  • View real-time progress in the optimization window
  • Monitor score improvements
  • Access detailed logs of the optimization process
  • Track cost and performance metrics

​
Understanding Results (2:14)

The optimization process typically shows:

  • Initial baseline performance
  • Progressive improvements
  • Final optimized results
  • Detailed breakdown of changes made

​
Applying and Managing Optimizations (2:29)

After optimization:

  • Apply optimized settings with one click
  • Review new instructions and demonstrations
  • Test individual examples
  • Run evaluation on test set to validate improvements

​
Advanced Optimization Strategies (4:11)

To further improve results:

  • Try different LLM models
  • Add prompting techniques (like chain of thought)
  • Combine multiple optimization approaches
  • Experiment with different demonstration sets

​
Cost Considerations (7:37)

Important factors to consider:

  • Optimization costs vs. inference costs
  • Trade-offs between model performance and expense
  • Tracking costs per call
  • Balancing quality and budget requirements

​
Best Practices (8:08)

For optimal results:

  1. Start with smaller datasets and lighter models
  2. Gradually increase complexity
  3. Monitor costs and performance metrics
  4. Test different model combinations
  5. Use optimization results to make informed decisions about model selection

​
Tips for Success

  • Begin with a clear baseline measurement
  • Use appropriate evaluators for your use case
  • Consider both quality and cost metrics
  • Iterate and experiment with different approaches
  • Keep track of optimization history for comparison

The Optimization Studio provides a systematic way to improve your LLM workflows, allowing you to find the optimal balance between performance and cost for your specific use case.