Documentation Index
Fetch the complete documentation index at: https://docs.getcalmo.com/llms.txt
Use this file to discover all available pages before exploring further.
Langfuse Integration
Connect Calmo to your Langfuse instance to enable LLM observability and prompt management through AI assistance. This integration provides access to 9 specialized tools across 3 categories for complete LLM monitoring and optimization workflows.Overview
The Langfuse integration transforms how your team handles LLM operations and observability by providing:- Intelligent Prompt Management - AI-powered prompt optimization and version control
- Advanced Trace Analysis - Comprehensive LLM execution tracing with performance insights
- Session Analytics - User activity analysis and conversation pattern recognition
- Cost Optimization - Token usage analysis and cost monitoring across LLM providers
- Quality Assurance - Automated evaluation and quality scoring of LLM outputs
- Safe Operations - All tools are read-only for secure observability without data modification
Key Capabilities
When connected, Calmo gains access to 9 Langfuse tools across 3 categories:| Category | Tools | Capability |
|---|---|---|
| Prompt Management | 2 tools | Manage and retrieve prompts with version control |
| Trace & Observations | 5 tools | Analyze LLM execution traces and performance |
| Session Analytics | 2 tools | Session analysis and user activity monitoring |
Prerequisites
- Langfuse instance (cloud or self-hosted) with API access
- Project access with appropriate permissions
- Calmo account with team or personal workspace
Setup Process
Step 1: Access Your Langfuse Instance
Locate Your Langfuse Configuration:- Navigate to your Langfuse dashboard
- Note your instance URL (e.g.,
https://cloud.langfuse.comor your self-hosted URL) - Ensure you have project access and API key generation permissions
Step 2: Generate Langfuse API Keys
Create Project API Keys:- Log in to your Langfuse instance
- Navigate to your target project
- Go to Settings → API Keys
- Click Create new API key pair
- Configure key settings:
- Description: “Calmo Integration”
- Permissions: Read-only (recommended for security)
- Copy both the Public Key and Secret Key immediately
- Public Key: Used for client-side operations and identification
- Secret Key: Used for server-side operations and authentication
- Base URL: Your Langfuse instance URL (cloud or self-hosted)
- Read access to prompts and prompt versions
- Read access to traces and observations
- Read access to sessions and user activity data
- Read access to project statistics and metadata
Step 3: Connect to Calmo
- Navigate to Integrations in your Calmo dashboard
- Click Langfuse integration
- Enter your Public Key
- Enter your Secret Key
- Enter your Base URL (defaults to
https://cloud.langfuse.com) - Configure tool permissions:
- ✅ All operations are read-only for security
- ✅ All tools enabled by default for comprehensive observability
- Test the connection using the built-in connection test
- Complete the integration setup
Tool Categories & Configuration
Prompt Management (Safe)
Default: Enabled - Essential for prompt optimization and version control- get-prompts - Get all prompts with metadata and version information
- get-prompt - Get a specific prompt with full version history and configuration
Trace & Observations (Safe)
Default: Enabled - Core LLM execution monitoring and analysis- get_trace - Get comprehensive trace information with all spans and metadata
- get-trace-details - Get detailed trace information including costs and performance
- fetch-traces - Fetch multiple traces with filtering and pagination
- fetch-observations - Fetch observations with advanced filtering capabilities
- fetch-observation - Fetch a single observation with complete details
Session Analytics (Safe)
Default: Enabled - User behavior and conversation analysis- fetch-sessions - Fetch user sessions with activity and conversation data
- analyze-user-activity - Analyze user activity patterns and engagement metrics
Team vs Personal Configuration
Team/Organization Setup
- Shared Langfuse project access across team members
- Organization-level LLM monitoring and governance policies
- Centralized prompt management and optimization workflows
- Team-wide cost monitoring and optimization insights
Personal Setup
- Individual Langfuse project connections
- Personal prompt experimentation and analysis
- Private LLM performance monitoring and debugging
- Individual cost tracking and optimization
Security & Best Practices
⚠️ Safety Recommendations
- Read-Only Access - All tools are read-only by design for maximum security
- Key Security - Use dedicated API keys with minimal required permissions
- Instance Security - Ensure Langfuse instance uses HTTPS and proper authentication
- Data Privacy - Review data retention and privacy policies for your Langfuse instance
- Access Monitoring - Monitor API usage and access patterns regularly
🔒 Permission Levels
| Risk Level | Operations | Recommendation |
|---|---|---|
| Low | All Langfuse operations (read-only by design) | ✅ Safe to enable |
Configuration Management
Updating Langfuse Connection
- Navigate to Integrations → Langfuse
- Click Edit Configuration
- Update public/secret keys or base URL as needed
- Test connection to verify changes
- Save configuration updates
Managing Multiple Projects
- Connect separate Langfuse projects for different applications
- Use different API keys for production vs development environments
- Configure project-specific monitoring and analysis workflows
- Maintain separate cost tracking per project
Advanced Features
Prompt Intelligence
- Version Analysis - Compare prompt versions and performance metrics
- Optimization Insights - AI-powered prompt improvement suggestions
- A/B Testing - Analyze prompt performance across different versions
- Template Management - Manage prompt templates and variables
Trace Analytics
- Performance Monitoring - Real-time LLM execution performance tracking
- Cost Analysis - Detailed token usage and cost breakdown by provider
- Error Detection - Automatic identification of failed or problematic traces
- Quality Scoring - Automated quality assessment of LLM outputs
Session Intelligence
- Conversation Flow - Analyze conversation patterns and user journeys
- Engagement Metrics - Track user engagement and session quality
- Behavioral Patterns - Identify common user behavior patterns
- Optimization Opportunities - Discover areas for conversation improvement
LLM Observability Workflows
Performance Monitoring
- Real-Time Tracking - Monitor LLM performance and latency in real-time
- Cost Optimization - Track and optimize token usage across different models
- Quality Assurance - Continuous monitoring of output quality and relevance
- Error Analysis - Identify and analyze LLM execution errors and failures
Prompt Development
- Iterative Optimization - Track prompt performance improvements over time
- Version Management - Manage prompt versions and deployment strategies
- A/B Testing - Compare different prompt approaches and configurations
- Best Practice Discovery - Identify high-performing prompt patterns
User Experience Analysis
- Conversation Quality - Analyze conversation effectiveness and user satisfaction
- Journey Mapping - Understand user interaction patterns and preferences
- Engagement Optimization - Optimize conversation flows for better engagement
- Personalization Insights - Discover opportunities for conversation personalization
Troubleshooting
Common Issues
Authentication Failed- Verify public and secret keys are correct and haven’t been revoked
- Check API key permissions in Langfuse project settings
- Ensure base URL is correct for your Langfuse instance
- Verify network connectivity to Langfuse instance
- Confirm API keys have access to the target project
- Check project permissions and user access levels
- Verify project exists and is accessible
- Review Langfuse logs for detailed error information
- Verify data exists in the specified time range
- Check trace IDs and observation IDs are correct
- Ensure data collection is active for your application
- Review data retention policies for your Langfuse instance
- Check network connectivity to Langfuse instance
- Verify Langfuse instance is responsive and healthy
- Consider increasing timeout settings if using self-hosted instance
- Review firewall and proxy settings
Getting Help
- Test Connection - Use the built-in connection test feature
- Update Credentials - Regenerate API keys if authentication issues persist
- Check Documentation - Refer to Langfuse official documentation for API setup
- Contact Support - Reach out to support@getcalmo.com for integration assistance
Data Types & Analysis
Prompt Data
- Prompt Configurations - Prompt templates, variables, and version information
- Performance Metrics - Prompt execution success rates and quality scores
- Version History - Complete audit trail of prompt changes and deployments
- Usage Analytics - Prompt usage patterns and frequency analysis
Trace Data
- Execution Traces - Complete LLM execution flows with timing and performance data
- Span Information - Individual operation details within traces
- Cost Metrics - Token usage and cost breakdown by model and provider
- Quality Scores - Automated and manual quality assessments
Observation Data
- LLM Outputs - Generated text and response quality metrics
- Input Analysis - Input processing and preparation statistics
- Model Performance - Model-specific performance and accuracy metrics
- Error Tracking - Failed operations and error classification
Session Data
- User Sessions - Complete user interaction sessions with conversation data
- Activity Patterns - User behavior patterns and engagement metrics
- Conversation Flows - Conversation structure and progression analysis
- Engagement Metrics - Session duration, interaction frequency, and satisfaction
Analytics Data
- Usage Statistics - Application usage patterns and adoption metrics
- Performance Trends - Long-term performance and quality trends
- Cost Optimization - Cost analysis and optimization recommendations
- Quality Insights - Quality improvement opportunities and best practices
For additional help with Langfuse integration, contact our support team at support@getcalmo.com.