Skip to main content
Version: Next

Monitoring Remote Executor

Feature Availability
Self-Hosted DataHub
DataHub Cloud

Overview​

This guide covers all aspects of monitoring your Remote Executor deployment:

  1. File-based health checks
  2. UI-based health monitoring
  3. Advanced Prometheus metrics configuration

Health Checks​

File-Based Health Checks​

The Remote Executor uses file-based health checks that can be monitored by your container platform:

  • Liveness: /tmp/worker_liveness_heartbeat
  • Readiness: /tmp/worker_readiness_heartbeat

These files are automatically managed by the Remote Executor and can be used by Kubernetes liveness/readiness probes or ECS health checks.

UI-Based Health Monitoring​

Monitor Remote Executor health directly in the DataHub UI:

  1. Navigate to Data Sources > Executors
  2. View health information for each Pool:
    • Active Remote Executor instances
    • Last reported time for each executor
    • Status (Active/Stale)
    • Currently running Ingestion tasks and their details

Advanced: Prometheus Metrics​

The Remote Executor exposes metrics on port 9087/tcp in Prometheus/OpenMetrics format. Metrics can be collected by Prometheus stack or compatible agents, such as DataDog.

Metric Categories​

  1. Ingestion Metrics

    • datahub_executor_worker_ingestion_requests - Total jobs received
    • datahub_executor_worker_ingestion_errors - Failed jobs (v0.3.9+)

  2. Resource Metrics (v0.3.9+)

    • Memory: datahub_executor_memory_*
    • CPU: datahub_executor_cpu_*
    • Disk: datahub_executor_disk_*
    • Network: datahub_executor_net_*

Prometheus Configuration​

Example ServiceMonitor resource to allow scraping Remote Executor metrics in Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
  name: datahub-remote-executor
spec:
  endpoints:
    - port: metrics
  selector:
    matchLabels:
      app.kubernetes.io/name: datahub-remote-executor

Discovering Available Metrics​

  1. View metrics endpoint directly:

    curl http://your-executor:9087/metrics

  2. Read annotations in Prometheus UI

  3. Search datahub_executor_* in your monitoring system

note

Platform-specific metrics (e.g., container restarts) should be monitored through native tooling (CloudWatch for ECS, Kubernetes metrics for K8s).