Add hardlink support #1954
Open
Add hardlink support #1954
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ChengyuZhu6
force-pushed
the
hardlink
branch
3 times, most recently
from
ff5ecf3 to
3e921b6
Compare
ktock
reviewed
Feb 3, 2025
ChengyuZhu6
force-pushed
the
hardlink
branch
2 times, most recently
from
b879c95 to
457b45b
Compare
ChengyuZhu6
force-pushed
the
hardlink
branch
2 times, most recently
from
8829e66 to
92702af
Compare
Member
|
Needs rebase |
ChengyuZhu6
force-pushed
the
hardlink
branch
from
92702af to
1d5fd08
Compare
Member
Author
Done. |
ChengyuZhu6
force-pushed
the
hardlink
branch
from
1d5fd08 to
8694c9a
Compare
Member
Author
|
Just retrigger ci, no code change. |
Member
Author
|
@ktock I conducted experiments with several basic images, converting them to the estargz format and running them in containers with a simple 'echo "hello"' command. These tests used only background threads of stargz to pull images to the local machine. By measuring the overall memory and disk usage, I observed that implementing hardlinks resulted in a 20-30% reduction in both memory consumption and disk space requirements. 
|
Member
Author
|
When working with different versions of an image, implementing hardlinks can achieve a memory and disk deduplication effect of nearly 50%. I verified this by conducting tests with various development versions of Golang images. 
|
Member
Author
|
kindly ping @ktock |
ChengyuZhu6
force-pushed
the
hardlink
branch
3 times, most recently
from
efbcc75 to
01b70ae
Compare
ChengyuZhu6
force-pushed
the
hardlink
branch
3 times, most recently
from
1d0858b to
756bb86
Compare
Member
Author
|
cc @ktock |
|
We're very interested by this PR, is there any plan for review @ktock ? |
Member
|
Needs rebase |
ktock
requested changes
Jul 16, 2025
| } | ||
|
|
||
| // ChunkDigest option allows specifying a chunk digest for the cache | ||
| func ChunkDigest(digest string) Option { |
ChengyuZhu6
force-pushed
the
hardlink
branch
4 times, most recently
from
9669477 to
349ad77
Compare
Member
Author
ktock
reviewed
Jul 28, 2025
Member
|
CI failure in |
ChengyuZhu6
force-pushed
the
hardlink
branch
5 times, most recently
from
170ded5 to
d62a0aa
Compare
ChengyuZhu6
force-pushed
the
hardlink
branch
6 times, most recently
from
e85935b to
f39d6a2
Compare
Member
Author
|
ping @ktock |
ktock
reviewed
Aug 21, 2025
ChengyuZhu6
force-pushed
the
hardlink
branch
2 times, most recently
from
3351d93 to
e90577f
Compare
ChengyuZhu6
force-pushed
the
hardlink
branch
2 times, most recently
from
6d9d2bb to
c3fa985
Compare
ChengyuZhu6
force-pushed
the
hardlink
branch
from
c3fa985 to
e376da9
Compare
Implement hardlink management for Stargz Snapshotter cache to reduce
storage usage by deduplicating identical content chunks.
Implementation:
- Store canonical files in {root}/hardlinks/ directory
- Use nlink-based garbage collection (remove when nlink == 1)
- Track chunk digests with LRU cache for efficient lookups
- Create hardlinks from canonical files to cache locations
Configuration:
- Add HardlinkRoot option for hardlink storage directory
API:
- Enroll(): Register canonical files by digest
- Get(): Retrieve canonical file paths
- Add(): Create hardlinks to target locations
Signed-off-by: ChengyuZhu6 <[email protected]>
ChengyuZhu6
force-pushed
the
hardlink
branch
from
e376da9 to
c0aac13
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Propose the implementation of a hardlink feature in the caching mechanism to optimize memory usage, improve performance and save disk space.
Fixes: #1953