Add the Pipeline Projects sections#11
Merged
dimitri-yatsenko merged 19 commits intodatajoint:mainfrom Dec 15, 2025
Merged
Conversation
Introduces a comprehensive guide for organizing DataJoint data pipeline projects. Covers: - Pipeline as Git repository and Python package structure - One-to-one mapping of database schemas to Python modules - The required `schema` property in each module - DAG structure at both table and module levels - pyproject.toml configuration for object storage - Module dependency patterns and best practices
…ect-chapter-TrpJb Add data pipeline project organization chapter
- Use src/workflow/ as standard path for pipeline code - Add LICENSE file to project structure - Add notebooks/, docs/, docker/ directories - Include Docker and docker-compose examples - Add figures from SPECS (pipeline-illustration.png, schema-illustration.png) - Update pyproject.toml with build-system and src layout config
…ect-chapter-TrpJb Update Pipeline Projects chapter with standard structure
…reate Schemas - Move 070-pipeline-project.md from Operations to Design as 000-pipeline-project.md - Streamline Create Schemas chapter by removing duplicated multi-schema content - Add reference from Create Schemas to Pipeline Projects for project organization - Fix table link in Create Schemas (010-table.ipynb -> 015-table.ipynb) - Pipeline Projects now serves as the introduction to the Design section
- Simple scripts/notebooks are the starting point for learning - Full project structure (src/workflow/) is for production deployment - Create Schemas now shows simple inline examples first - Pipeline Projects explains when you need the full structure - Both chapters now complement each other without duplication
Mention that the DataJoint Platform (datajoint.com) is designed for hosting and managing full DataJoint projects as an alternative to DIY Docker deployment.
…ect-chapter-TrpJb Move Pipeline Project into Design
- Rename 000-pipeline-project.md to 090-pipeline-project.md - Learners start with simple scripts, learn full project structure last - Update reference in Create Schemas chapter
…ect-chapter-TrpJb Move Pipeline Projects to end of Design section
…ect-chapter-TrpJb Fix figure sizes in Pipeline Projects chapter
- Add YAML frontmatter with title, authors, and date metadata - Add Learning Objectives admonition at the beginning - Add Prerequisites section linking to related chapters - Reorganize content with clearer section hierarchy - Add proper admonition formatting (tip, note, warning, important) - Integrate SQL examples naturally as reference section - Improve quiz formatting with dropdown answer - Add comprehensive Summary section with next steps - Remove broken SQL execution section at the end - Add tables for index mechanisms and query patterns
…hapter-YsjhA Improve Indexes chapter organization and consistency
- Remove excessive bold formatting from bullet points and phrases - Fix typo: "must authentication" → "must authenticate" - Add Links section at end of chapter with relevant database resources
Updated chapter reference list to include hyperlinks to actual chapter files. Also corrected "Relational Practice" to "Data Integrity" to match actual chapter.
…-cleanup-wQQGm Clean up Databases chapter formatting and links
The quiz/assessment chapter has been removed from the book.
…check-z9fAn Remove knowledge check chapter
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.