Add block size to head block cut log message (#5771)

* Add block size to head block cut log message

This change adds a size field to the 'head block cut' log message
in the ingester. This improves observability by showing the actual
size of the block when it's cut, making it easier to understand
whether blocks are cut due to size or duration limits.

Before:
level=info msg="head block cut. enqueueing flush op" tenant=CashwalkAPI block=0e5151dd-4a5f-4fbe-b24e-d946f6faf894

After:
level=info msg="head block cut. enqueueing flush op" tenant=CashwalkAPI block=0e5151dd-4a5f-4fbe-b24e-d946f6faf894 size=134217728

This is particularly useful for debugging OOMKilled issues caused by
concurrent flushing of oversized blocks, as it provides immediate
visibility into block sizes that correlate with memory pressure.

Signed-off-by: backend-minho <[email protected]>

* Add block size to head block cut log messages

This change adds a size field to the log messages when cutting head
blocks in both ingester and livestore modules. This improves
observability by showing the actual size of blocks when they are cut,
making it easier to understand whether blocks are cut due to size or
duration limits.

Changes:
- ingester: Add size to 'head block cut. enqueueing flush op' message
- livestore: Add size to 'queueing wal block for completion' message

Example output (ingester):
level=info msg="head block cut. enqueueing flush op" block=0e5151dd-4a5f-4fbe-b24e-d946f6faf894 size=134217728

This is particularly useful for debugging OOMKilled issues caused by
concurrent flushing of oversized blocks, as it provides immediate
visibility into block sizes that correlate with memory pressure.

Signed-off-by: MinhoJJang <[email protected]>

* Enhance logging for flush operations in ingester

This update improves the logging for flush operations by adding a message when enqueueing a flush operation in the `cutOneInstanceToWal` method. Additionally, the log message for cutting head blocks has been simplified by removing the "enqueueing flush op" part, focusing on the block cut event itself.

Changes:
- ingester: Log message enhancement for flush operation enqueueing
- ingester: Simplify log message for head block cut

This enhancement aids in better tracking of flush operations and improves log clarity.

Signed-off-by: [MinhoJJang] <[email protected]>

---------

Signed-off-by: backend-minho <[email protected]>
Signed-off-by: MinhoJJang <[email protected]>
Signed-off-by: [MinhoJJang] <[email protected]>
Co-authored-by: MinhoJJang <[email protected]>

Author: MinhoJJang

modules/ingester/flush.go

@@ -193,7 +193,7 @@ func (i *Ingester) cutOneInstanceToWal(instance *instance, immediate bool) {
 	}
 
 	if blockID != uuid.Nil {
-		level.Info(log.Logger).Log("msg", "head block cut. enqueueing flush op", "tenant", instance.instanceID, "block", blockID)
+		level.Info(log.Logger).Log("msg", "enqueueing flush op", "tenant", instance.instanceID, "block", blockID)
 		// jitter to help when flushing many instances at the same time
 		// no jitter if immediate (initiated via /flush handler for example)
 		i.enqueue(&flushOp{

modules/ingester/instance.go

@@ -287,6 +287,10 @@ func (i *instance) CutBlockIfReady(maxBlockLifetime time.Duration, maxBlockBytes
 		}
 
 		completingBlock := i.headBlock
+		blockID := (uuid.UUID)(completingBlock.BlockMeta().BlockID)
+		blockSize := completingBlock.DataLength()
+
+		level.Info(i.logger).Log("msg", "head block cut", "block", blockID, "size", blockSize)
 
 		// Now that we are adding a new block take the blocks mutex.
 		// A warning about deadlocks!!  This area does a hard-acquire of both mutexes.
@@ -304,7 +308,7 @@ func (i *instance) CutBlockIfReady(maxBlockLifetime time.Duration, maxBlockBytes
 			return uuid.Nil, fmt.Errorf("failed to resetHeadBlock: %w", err)
 		}
 
-		return (uuid.UUID)(completingBlock.BlockMeta().BlockID), nil
+		return blockID, nil
 	}
 
 	return uuid.Nil, nil

modules/livestore/instance.go

@@ -396,9 +396,10 @@ func (i *instance) cutBlocks(immediate bool) (uuid.UUID, error) {
 	}
 
 	id := (uuid.UUID)(i.headBlock.BlockMeta().BlockID)
+	blockSize := i.headBlock.DataLength()
 	i.walBlocks[id] = i.headBlock
 
-	level.Info(i.logger).Log("msg", "queueing wal block for completion", "block", id.String())
+	level.Info(i.logger).Log("msg", "queueing wal block for completion", "block", id.String(), "size", blockSize)
 
 	err = i.resetHeadBlock()
 	if err != nil {