Senior Data Scientist - ML Operations & Analytics

Responsibilities

• Build ML operations analytics tracking pipeline health, costs, latency, and quality metrics across all services
• Lead production LLM evaluation initiatives, designing custom metrics, quality monitoring and LLM observability in general
• Drive development efforts in collaboration with different ML teams based on analytics outputs
• Collaborate with our data annotation team on prompt engineering and evaluation dataset creation
• Help with curating the Filevine dataset
• Create actionable dashboards and insights for ML teams and leadership
• Help define SLAs, cost optimization and failure prevention strategies for our ML pipelines

Qualifications

• 3+ years in data science, ML operations, or analytics engineering
• Experience with LLM evaluation frameworks and quality metrics design
• Strong Python, SQL, and data visualization skills
• Knowledge of prompt optimization techniques
• Deep understanding of latest trends in LLMs and their production deployments…