refactor: added batch commits and Increased task time limits in celery_app.py #464
Conversation
…y_app.py - Increased task time limits in celery_app.py for longer processing times. - Enhanced pagination logic in NotionHistoryConnector to handle large result sets. - Implemented batch commits every 10 documents across various indexers (Airtable, ClickUp, Confluence, Discord, GitHub, Google Calendar, Gmail, JIRA, Linear, Luma, Notion, Slack) to improve performance and reduce database load. - Updated final commit logging for clarity on total documents processed.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 57fd82f..a3a5b13
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (18)
• surfsense_backend/app/celery_app.py
• surfsense_backend/app/connectors/notion_history.py
• surfsense_backend/app/tasks/connector_indexers/airtable_indexer.py
• surfsense_backend/app/tasks/connector_indexers/clickup_indexer.py
• surfsense_backend/app/tasks/connector_indexers/confluence_indexer.py
• surfsense_backend/app/tasks/connector_indexers/discord_indexer.py
• surfsense_backend/app/tasks/connector_indexers/github_indexer.py
• surfsense_backend/app/tasks/connector_indexers/google_calendar_indexer.py
• surfsense_backend/app/tasks/connector_indexers/google_gmail_indexer.py
• surfsense_backend/app/tasks/connector_indexers/jira_indexer.py
• surfsense_backend/app/tasks/connector_indexers/linear_indexer.py
• surfsense_backend/app/tasks/connector_indexers/luma_indexer.py
• surfsense_backend/app/tasks/connector_indexers/notion_indexer.py
• surfsense_backend/app/tasks/connector_indexers/slack_indexer.py
• surfsense_web/app/(home)/login/GoogleLoginButton.tsx
• surfsense_web/app/dashboard/page.tsx
• surfsense_web/components/chat/ChatInputGroup.tsx
• surfsense_web/components/sidebar/app-sidebar.tsx
Description
Screenshots
API Changes
Change Type
Testing Performed
Checklist
High-level PR Summary
This PR implements performance optimizations across the backend indexing system by introducing batch commits every 10 documents for all connector indexers (Airtable, ClickUp, Confluence, Discord, GitHub, Google Calendar, Gmail, JIRA, Linear, Luma, Notion, Slack) to reduce database load. It also increases Celery task time limits from 1 hour to 8 hours to accommodate longer-running indexing operations, and enhances the Notion connector's pagination logic to properly handle large result sets. Additionally, the frontend receives updates to the chat connector selector UI to support both indexed document types and live search connectors.
⏱️ Estimated Review Time: 30-90 minutes
💡 Review Order Suggestion
surfsense_backend/app/celery_app.pysurfsense_backend/app/connectors/notion_history.pysurfsense_backend/app/tasks/connector_indexers/airtable_indexer.pysurfsense_backend/app/tasks/connector_indexers/clickup_indexer.pysurfsense_backend/app/tasks/connector_indexers/confluence_indexer.pysurfsense_backend/app/tasks/connector_indexers/discord_indexer.pysurfsense_backend/app/tasks/connector_indexers/github_indexer.pysurfsense_backend/app/tasks/connector_indexers/google_calendar_indexer.pysurfsense_backend/app/tasks/connector_indexers/google_gmail_indexer.pysurfsense_backend/app/tasks/connector_indexers/jira_indexer.pysurfsense_backend/app/tasks/connector_indexers/linear_indexer.pysurfsense_backend/app/tasks/connector_indexers/luma_indexer.pysurfsense_backend/app/tasks/connector_indexers/notion_indexer.pysurfsense_backend/app/tasks/connector_indexers/slack_indexer.pysurfsense_web/components/chat/ChatInputGroup.tsxsurfsense_web/app/dashboard/page.tsxsurfsense_web/app/(home)/login/GoogleLoginButton.tsxsurfsense_web/components/sidebar/app-sidebar.tsxsurfsense_web/app/(home)/login/GoogleLoginButton.tsxsurfsense_web/app/dashboard/page.tsxsurfsense_web/components/sidebar/app-sidebar.tsxsurfsense_web/components/chat/ChatInputGroup.tsx