Staff Backend Engineer – Second Horizon (Grafana Labs)

About Grafana Labs

Grafana Labs is the company behind Grafana Cloud, the fully managed observability platform trusted by more than 10,000 organizations to ensure reliability, resolve incidents faster, and optimize telemetry at scale. We are a 100% remote company with team members across 40+ countries, backed by leading investors including Lightspeed Venture Partners, Sequoia Capital, GIC, Coatue, J.P. Morgan, CapitalG, and Lead Edge Capital.

Role Overview

This is a remote opportunity and we are looking for candidates from the U.S. or Canada. Residents of Quebec are not eligible for this role.

We recently started a skunkworks initiative with a mission to bring observability to the rest of the business – we are building an AI-native data intelligence system that gives agents reliable, governed access to enterprise context.

Key Responsibilities

  • Build the core backend services: Design, implement, test, and operate the first services for context ingestion, context indexing, retrieval orchestration, API access, source configuration, and system administration.
  • Create a scalable SaaS foundation: Help define and build the architecture for a multi-tenant service, including tenant isolation, usage tracking, quotas, audit logs, background jobs, and reliable service boundaries.
  • Power agent-facing retrieval workflows: Build APIs and service interfaces that allow AI agents, MCP tools, CLIs, and internal applications to retrieve relevant context, provenance, confidence signals, and warnings.
  • Work across product and infrastructure: Partner with the team to make practical tradeoffs between fast experimentation and long-term reliability.
  • Operate what you build: Instrument services with metrics, logs, traces, alerts, and dashboards.
  • Contribute to technical direction: Help shape the architecture, service boundaries, storage choices, API contracts, deployment patterns, and engineering practices.
  • Communicate effectively and take ownership with real, measurable impact.

Requirements

  • Experience with LLMs, prompt engineering, and building applications powered by GenAI.
  • Proven track record of delivering software that made it into production and is actively used by users.
  • Exposure to working in cloud-native environments (e.g., AWS, GCP, Azure).
  • Experience using observability tools to understand and troubleshoot system behavior.

Why You’ll Thrive at Grafana Labs

  • 100% Remote Global Culture
  • Scaling Organization
  • Transparent Communication
  • Innovation-Driven
  • Open Source Roots
  • Empowered Teams
  • Career Growth Pathways
  • Approachable Leadership
  • Balance is Key (30 days annual leave, 3 reserved for Shutdown Days)

Compensation

The estimated salary range for this position is $174,986 – $209,983 per year. Final offer amounts are determined by multiple factors including candidate experience and expertise and may vary from the amounts listed above.

Never Miss a New Job Opportunity

Keep up with newly posted worldwide jobs, remote openings, and career opportunities so you can take action at the right time.

Join Our WhatsApp Channel
Free job updates • Join anytime