Senior SRE Engineer (Observability Focus)

Responsibilities

• Own the full observability stack: metrics (VictoriaMetrics), logs (OpenSearch), and traces (OpenTelemetry) — from pipeline design to day-2 operations.
• Architect and run VictoriaMetrics cluster topology (vmstorage/vminsert/vmselect), including vmagent scraping, remote write configuration, vmalert rules, and cardinality control.
• Operate OpenSearch clusters: index lifecycle management (ISM), hot-warm-cold architecture, shard tuning, and ingest pipelines via Data Prepper.
• Build and maintain OTEL Collector pipelines — receivers, processors, exporters — and instrument services across Java, Python, and JS/TS stacks (auto and manual).
• Run Kafka as the telemetry transport layer (OTEL Collector → Kafka → backends), including topic design, partition strategy, consumer group lag monitoring, and throughput tuning for high-volume telemetry.
• Manage log shipping infrastructure using Fluent Bit, Vector, or Fluentd; define structured logging standards and field normalization across services.
• Build Grafana dashboards and alerting that engineers actually use — clear, actionable, with well-structured variables and thresholds.
• Work with platform and application teams to improve sampling strategies (head/tail), batching, and context propagation across distributed services.
• Contribute to incident response, post-mortems, and reliability improvements driven by observability signals.
• Mentor engineers on observability practices, tooling, and structured logging standards.

Requirements

• 6+ years in a DevOps, SRE, or platform engineering role, with at least 2 years focused on observability tooling at production scale.
• Deep hands-on experience with VictoriaMetrics (or Prometheus) — MetricsQL/PromQL, exporters, service discovery, remote write, downsampling, and retention management.
• Solid OpenSearch or Elasticsearch skills: cluster operations, Query DSL, ISM policies, and ingest pipeline design.
• Production experience with OpenTelemetry: Collector configuration, OTLP, context propagation, and instrumentation across multiple languages.
• Strong Kafka skills — producer/consumer patterns, consumer group management, Kafka Connect, Schema Registry, and JMX-based monitoring. Strimzi experience a plus if you've run Kafka on Kubernetes.
• Proficiency with log shippers (Fluent Bit, Vector, Fluentd) and structured log parsing/normalization.
• Working knowledge of Kubernetes (operators, Helm), Argo CD/GitOps, and Terraform/Ansible.
• Comfortable in a hybrid AWS + on-prem environment; solid understanding of networking as it applies to scraping and shipping pipelines.
• Scripting ability in Bash or Python for automation and tooling.
• Strong communication skills — you can explain observability tradeoffs clearly to engineers and non-engineers alike.
• English proficiency.