Epic: context & storage#753
Draft
atravitz wants to merge 13 commits into
Draft
Conversation
* feat(warehouse): initial re-implementation of StorageManager * feat(warehouse): initial implementation with test * feat(warehouse): add support for new context object * feat(warehouse): experimental implementation of executing a DAG * feat!: Replace Context with new implementation BREAKING CHANGE: This completely breaks existing `Context` to leverage our new context management. * test(warehouse): add a context test * test: cleanup test_protocolunit.py * refactor(StorageManager)!: make _convert_to_namespace a static method * test(ProtocolDAG): fix ProtocolDAG tests * test(Protocol): cleanup testing for transformation and protocol * test(Protocol): remove reliance on mixin in two cases since StorageManger isn't tokenizable * chore: remove commented out code * Revert "chore: remove commented out code" This reverts commit 7e6bb03. * chore: fix comments and cleanup code * fix: satisfy typing * refactor(StorageManager)!: make _convert_to_namespace, convert_to_namespace * docs: storagemanager * docs: add docstrings to Context object * Update gufe/protocols/protocolunit.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * fix: indents from code review web ui * fix: remove TODO * chore: rename scratch_path to scratch_dir * chore: remove dead comment * chore: remove redudndant fixtures * refactor: rename convert_to_namespace to append_to_namespace * docs: added news * Apply suggestions from code review on news item Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * fix docs env build by mocking zstandard * test: add a context test for validating cleanup --------- Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io>
* feat: return namespaced handles so other untis can ingest these handles * test: add a test to validate * Update gufe/storage/storagemanager.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> --------- Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com>
… backend (#745) * refactor: split out transfer functionality * test: split out transfer functionality tests Signed-off-by: Ethan Holz <ethan.holz@omsf.io> --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #753 +/- ##
==========================================
- Coverage 98.56% 98.44% -0.13%
==========================================
Files 43 45 +2
Lines 2797 2832 +35
==========================================
+ Hits 2757 2788 +31
- Misses 40 44 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
* docs: add information on storage Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * docs: initial draft on context * docs: improve context docs * docs: migrate how to write a custom implementation * doc: update with comments from review Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * fix docs build * docs: Add information on changes in feat/return-storage-handles This adds new information on how storage is handled, and how storage gets passed around by units. * docs: add information on pre-namespaced objects from other branch * docs: add suggestion to back link to Context docs * docs: add becomes in migration guide per suggestion * docs: add link to protocol how to * docs: add more informaton on StorageManagers * docs: clean up storage code example * Apply suggestions from code review Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * fix typo * Apply suggestions from code review Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs: add more context on context * fix note formatting * move serialization constraints to the end * Apply suggestions from code review Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Apply suggestion from @atravitz Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * fix formatting for docs build * Apply suggestions from code review Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * clean up --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io>
atravitz
force-pushed
the
feat/warehouse
branch
from
8a24e9a to
23137fa
Compare
dotsdl
reviewed
Apr 10, 2026
dotsdl
left a comment
dotsdl
left a comment
Member
There was a problem hiding this comment.
This is looking excellent! Did a quick review of the new concept docs; just a couple notes I think should be addressed.
Will give the rest of the PR a thorough review at a later time! Can't wait to use this in alchemiscale to finally support result file retention!
|
|
||
| .. code-block:: python | ||
|
|
||
| path = ctx.shared.scratch_dir / "myfile.dat" |
Member
There was a problem hiding this comment.
Suggested change
| path = ctx.shared.scratch_dir / "myfile.dat" | |
| path = ctx.shared / "myfile.dat" |
Comment on lines
+158
to
+161
| 3. **Handle ctx.permanent.** There was no equivalent in the legacy API. | ||
| Decide which results must persist between DAG executions and write them via | ||
| ``ctx.permanent``. For migration you can start by mirroring whatever used | ||
| to live in ``ctx.shared`` and refine later. |
Member
There was a problem hiding this comment.
I don't think we should recommend that protocol authors switch from shared to permanent by default, as this could result in large in-DAG intermediate files getting held by execution engines for no good reason.
The recommendation here should be to switch to permanent any files that must survive DAG execution for the purposes of continuation or understanding results later; the default should not be keep everything.
Member
There was a problem hiding this comment.
To elaborate: in alchemiscale, we plan to retain permanent files by default (unless opted out of by the Task creator) for DAGs, and only retain shared during the duration of DAG execution (for continuation if interrupted) or retain shared indenfinitely if opted into by the Task creator (for debugging problematic cases).
This was referenced Jun 4, 2026
|
🚨 API breaking changes detected! 🚨 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes to how we handle contexts and storage.
This introduces breaking changes that will require updates to openfe protocols and exorcist (as described in openfe #1833).
Whether we do this in 2.0 or a highly-advertised 1.x is TBD.
Implemented by @ethanholz - this epic is to track the work as a long-lived branch.
Tips
Since this will create a commit, it is best to make this comment when you are finished with your work.
Checklist
newsentryDevelopers certificate of origin