Metrics

Metric endpoints, labels, recording rules, and queries for INFRA Victoria observability components.

INFRA monitors VictoriaMetrics, VictoriaLogs, VictoriaTraces, VMAlert, Alertmanager, Grafana, Nginx Exporter, and Blackbox Exporter. Targets come from roles/infra/templates/prometheus/infra.yml.j2; scrape jobs are defined in roles/infra/templates/victoria/prometheus.yml.

Component versions and enabled services affect the raw metric set. This page lists scrape targets, rules, and common metrics for the Victoria stack. See Monitoring System for site-wide counts.

Metric Definitions and Counts

Raw metrics: Produced by components or exporters. Read TYPE and HELP at /metrics for their types and meanings.
Recorded metrics: Computed from expressions by vmalert and queried in VictoriaMetrics. They have no independent TYPE metadata and cannot all be classified as Counter or Gauge.
Scrape status: up and scrape_* are generated by the collection system. up=1 indicates a successful scrape; database and application availability still require functional checks.

Histogram / Summary _bucket, _sum, and _count are different samples of the same metric family. Historical table rows, recording rules, and live time series count different objects and cannot be added together.

Scrape Endpoints and Labels

All targets below use job=infra and HTTP /metrics. The default target file is /infra/targets/infra/infra-<infra_seq>.yml.

Component	Default Port	`cls`	Instance Label `ins`
Nginx Exporter	9113	`nginx`	`nginx-<infra_seq>`
VictoriaMetrics	8428	`vmetrics`	`vmetrics-<infra_seq>`
VictoriaLogs	9428	`vlogs`	`vlogs-<infra_seq>`
VictoriaTraces	10428	`vtraces`	`vtraces-<infra_seq>`
VMAlert	8880	`vmalert`	`vmalert-<infra_seq>`
Alertmanager	9059	`alertmanager`	`alertmanager-<infra_seq>`
Blackbox Exporter	9115	`blackbox`	`blackbox-<infra_seq>`
Grafana	3000	`grafana`	`grafana-<infra_seq>`

Each target also carries ip and the scrape endpoint instance. The template does not inject the old type label; use cls to distinguish components. Targets are registered only for enabled components. VMAlert requires both vmalert_enabled and vmetrics_enabled; Nginx target registration depends on nginx_enabled.

Blackbox Exporter’s own metrics belong to job=infra. ICMP probe_* metrics produced through /probe belong to the separate job=ping. These are different target types.

Current Recording Rules

files/victoria/rules/infra.yml defines 2 recording rules, evaluated by vmalert and written back to VictoriaMetrics:

Metric	Source Expression	Meaning
`infra_up`	`up{job="infra"}`	Whether an infrastructure scrape endpoint is reachable
`agent_up`	`up{job!~"infra\|node\|etcd\|minio\|pgsql\|redis\|mysql"}`	Reachability of targets outside the excluded jobs

infra_up and agent_up are recording rules, absent from components’ raw /metrics output. The exclusion list above defines the scope of agent_up: it does not exclude jobs such as kafka, ping, or pgrds, and does not cover exporters already in node or pgsql. up=1 indicates only successful scraping. Assess functional service health with nginx_up, probe results, or component metrics.

The InfraDown and AgentDown alerts fire when their respective metrics stay below 1 for one minute. Both use severity CRIT. See Monitoring and Alerting for the rules.

Shared Liveness Recording Rules

In addition to module rules, roles/infra/templates/victoria/agent.yml generates 14 shared liveness rules, deployed to /infra/rules/agent.yml. The table retains template expressions; configuration variables render the ports. All derive from collector-generated up and indicate scrape success, not necessarily successful database queries, quorum, or application requests.

Recorded Metric	Template Expression
`node_up`	`up{instance=~".*:{{ node_exporter_port\|default('9100') }}"}`
`haproxy_up`	`up{instance=~".*:{{ haproxy_exporter_port\|default('9101') }}"}`
`keepalived_exporter_up`	`up{instance=~".*:{{ vip_exporter_port\|default('9650') }}"}`
`docker_up`	`up{instance=~".*:{{ docker_exporter_port\|default('9323') }}"}`
`vector_up`	`up{instance=~".*:{{ vector_port\|default('9598') }}"}`
`etcd_up`	`up{job="etcd"}`
`minio_up`	`up{job="minio"}`
`pg_exporter_agent_up`	`up{instance=~".*:{{ pg_exporter_port\|default('9630') }}"}`
`pgbouncer_exporter_agent_up`	`up{instance=~".*:{{ pgbouncer_exporter_port\|default('9631') }}"}`
`pgbackrest_exporter_agent_up`	`up{instance=~".*:{{ pgbackrest_exporter_port\|default('9854') }}"}`
`patroni_up`	`up{instance=~".*:{{ patroni_port\|default('8008') }}"}`
`mysql_exporter_up`	`up{job="mysql"}`
`kafka_up`	`up{job="kafka",role=~".+"}`
`kafka_exporter_up`	`up{job="kafka",role=""}`

These recorded metrics are also absent from raw /metrics endpoints. Rules matching ports do not restrict job, so port overrides must remain consistent between the Infra rule template and service configuration.

Key Dashboard Metrics

The following names come from queries in the current files/grafana/infra dashboards and provide troubleshooting entry points. Availability depends on the installed version’s /metrics output.

Component	Key Metrics	Purpose
Common process metrics	`process_cpu_seconds_total`, `process_resident_memory_bytes`, `process_start_time_seconds`	Process resources and start time
Victoria components	`vm_app_version`, `vm_app_uptime_seconds`	Version and uptime
VictoriaMetrics	`vm_rows_inserted_total`, `vm_rows`, `vm_data_size_bytes`, `vm_new_timeseries_created_total`	Ingestion, storage, and new series
VictoriaLogs	`vl_rows_ingested_total`, `vl_bytes_ingested_total`, `vl_data_size_bytes`, `vl_http_requests_total`	Log ingestion, capacity, and requests
VictoriaTraces	`vt_rows_ingested_total`, `vt_bytes_ingested_total`, `vt_storage_rows`, `vt_data_size_bytes`	Trace ingestion and storage
VMAlert	`vmalert_alerts_firing`, `vmalert_alerts_pending`, `vmalert_recording_rules_last_evaluation_samples`, `vmalert_alerting_rules_errors_total`	Alert state and rule evaluation
Alertmanager	`alertmanager_alerts`, `alertmanager_silences`, `alertmanager_notifications_total`, `alertmanager_notifications_failed_total`	Alert aggregation, silences, and notifications
Grafana	`grafana_build_info`, `grafana_http_request_duration_seconds_count`, `grafana_stat_active_sessions`	Version, HTTP requests, and sessions
Nginx	`nginx_up`, `nginx_http_requests_total`, `nginx_connections_active`, `nginx_connections_waiting`	Nginx status, traffic, and connections

These are raw metric names. Counters usually require rate() / increase() before analysis. Histogram / Summary _bucket, _sum, and _count are samples of one metric family, not unrelated families counted by row.

Get the Actual Metric List

Run these queries in VictoriaMetrics VMUI (default http://<infra-ip>:8428/vmui) or Grafana Explore:

up{job="infra"}
count by (cls) (up{job="infra"})
count by (__name__) ({job="infra"})

The first query checks each target’s scrape status, the second counts registered component endpoints, and the third lists visible series counts by metric name at the query time. To count distinct metric names, use count(count by (__name__) ({job="infra"})). This is still not an exporter metric-family count and cannot be added directly to other module counts.

Read raw types, HELP text, and labels from the target endpoint, for example:

curl -fsS http://127.0.0.1:8428/metrics
curl -fsS http://127.0.0.1:9428/metrics
curl -fsS http://127.0.0.1:8880/metrics