Observability Engineer

About The Role & Team

IDC is building the next generation of AI-powered intelligence platforms that transform how technology decisions get made. Our platform re-imagines the way decision-makers discover and interact with trusted research and data, and is foundational to IDC's future.We are looking for an Observability Engineer to join the Quality & Evaluation team. You will build and maintain the observability platform that gives the team visibility into AI system performance, cost, and quality. Working under the direction of the senior evaluation engineer, you will build the dashboards, alerting systems, and experimentation infrastructure that enable the team to understand and improve the platform's AI pipeline.Your work makes it possible for everyone on the team to see what the system is doing, how much it costs, and whether it is getting better or worse. Without your contributions, the team is flying blind.

What You’ll Do

• Build and maintain the observability platform that provides visibility into AI system performance, cost, and quality
• Build dashboards that track key operational metrics: response latency, LLM token usage, cost per query, and error rates across the agent pipeline
• Implement A/B testing infrastructure that enables controlled experiments on prompt strategies and pipeline changes
• Extend existing tracing infrastructure to provide deeper visibility into per-component behavior
• Build alerting systems that surface anomalies in cost, latency, or quality metrics
• Contribute to evaluation infrastructure alongside the senior evaluation engineer
  
  

What You Bring

• 3+ years of experience building observability, monitoring, or data visualization systems
• Experience with metrics collection, dashboarding, and alerting (e.g., Prometheus, Grafana, Datadog)
• Understanding of distributed tracing concepts
• Proficiency in Python
• Experience building or contributing to A/B testing or experimentation platforms
• Comfort working with large volumes of telemetry data
  
  

Preferred Qualifications

• Experience with LLM-specific observability (e.g., Langfuse, LangSmith)
• Background in cost optimization or resource monitoring for ML/AI systems
• Familiarity with statistical methods for experiment analysis
• Experience building custom dashboards or internal tools
• Background in platform or infrastructure engineering
  
  

Why This Role Stands Out

At IDC, your work helps shape how the world understands technology and where it goes next. You collaborate with curious, high-caliber colleagues who value rigor, integrity, and shared success. As the premier global provider of trusted technology intelligence, IDC equips business and technology leaders with the evidence they need to make confident decisions. Our insights inform strategy, investment, and innovation across industries and regions.

Recognized by IIAR as Analyst Firm of the Year for five consecutive years, IDC sets the standard for credibility and impact. With more than 1,000 analysts worldwide and a truly global perspective, we combine deep expertise with practical relevance. Here, your ideas matter, your voice is heard, and your contributions provide the insights leaders rely on every day. It is meaningful work, backed by a culture that supports growth, collaboration, and long-term career development with a globally respected brand.