[TPU Offload]Manage staging buffer in HBM

[juncgu-google]

Description

Add a staging buffer manager to book-keep the # of used staging blocks (page / chunk) for swap- in / out. This should be helpful in mitigating the OOM issue when kv cache offloading is handling very large of save / load blocks.

1. staging buffer size is controlled by env var: TPU_OFFLOAD_STAGING_BUFFER_TOKENS=8192. we will find a way to specify the buffer size in GB instead of in tokens.
2. when there are insufficient staging buffers for the incoming save / load operations, we will drop the extra save / load blocks.

Tests

pytest -sv tests/distrbuted/cpu_offloading_scheduler_test.py

Checklist

Before submitting this PR, please make sure:

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have made or will make corresponding changes to any relevant documentation.

[github-actions]

Description

Start with a short description of what the PR does and how this is a change from
the past.

The rest of the description includes relevant details and context, examples:

• why is this change being made,
• the problem being solved and any relevant context,
• why this is a good solution,
• some information about the specific implementation,
• shortcomings of the solution and possible future improvements.

If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure:

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have made or will make corresponding changes to any relevant documentation.

[saikat-royc]

aligned with the high level appraoch on the block tracking, reporting and staging buffer budgeting. let's also add unit tests for added confidence in the PR

[saikat-royc]

nice work on the unit tests!

[juncgu-google]

Thanks for the reviews!