Detect when Vector is lagging

[ShahroZafar]

Is there a way using which we can somehow detect by how much the vector is falling behind reading events from a particular file?

Currently we want to use vector to use file source to push logs of a very critical application (Almost 0 log loss) to kafka and we want to get alerted if vector is lagging behind and is not keeping up with the load.
The deployment is in kubernetes (We are currently not using kubernetes_logs source as log enrichment doesn't matter and we want to keep the log entry size as small as possible)

[jszwedko]

My suggestion for this would be to add a timestamp to each event as it is written to the file and then you could measure the difference between the timestamp being read and "now" (in Vector this could be done using a remap transform to calculate the difference and then the log_to_metric transform to turn that into a metric).

[ShahroZafar]

Right. I will give it a try. Thanks

Also is there a way to get information about how many files are actively watched? I mean a way using which we can be sure that vector has not missed / skipped reading a file because the rotation is very very fast (due to load)

[pront]

For your follow-up, the file source exposes an open_files gauge. With the default internal-metrics namespace, it appears in Prometheus as vector_open_files, with component_id identifying the file source.

This exposes Vector's internal metrics:

sources:
  internal:
    type: internal_metrics
    scrape_interval_secs: 1

sinks:
  metrics:
    type: prometheus_exporter
    inputs: [internal]
    address: "0.0.0.0:9598"

I checked this on Vector 0.60.0: the gauge reported 2 for two open files, then 3 when another matching file appeared.

The count cannot prove that every file was discovered. A file can appear and disappear between discovery scans. Keep rotated files available long enough to be discovered and drained, include the relevant rotated paths, and review glob_minimum_cooldown_ms alongside the rotation guidance.

Jesse's timestamp approach measures the age of events that actually reach Vector. Use the application's written timestamp, with synchronized clocks; the file source's default .timestamp records ingestion time. An age metric alone also cannot detect a completely stalled stream, so monitor event throughput or an application heartbeat alongside it.