[Question]: V0.20.4 DeepDoc batch upload parsing speed is slow (2000 files), consistently below 1%

[hao310810401]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Describe your problem

V0.20.4 DeepDoc batch upload parsing speed is slow (2000 files), consistently below 1%

[dosubot]

There are no existing solutions documented for slow batch upload parsing in DeepDoc v0.20.4, but recent code and PRs suggest several areas to investigate for performance improvements. Batch parsing uses Python's concurrent.futures for parallelism, but default settings may limit the number of worker threads, which can slow down large uploads. You can try increasing the worker pool size and tuning chunk sizes in your configuration to improve throughput. Bottlenecks may also come from synchronous disk I/O or excessive locking in the orchestration logic—reviewing these areas could help speed up processing.

If you are parsing complex documents, consider using the MinerU backend with the vlm-http-client option to offload parsing to a remote vLLM server, which can significantly improve performance and scalability for large batches. This setup only requires network connectivity and no GPU on the RAGFlow server itself. For details, see the MinerU backend configuration.

Recent changes have also introduced more flexible parser selection and chunking methods, which may affect parsing speed depending on your configuration. You can experiment with different parser options and chunking strategies to see if they improve your batch upload performance. See the chunking and parser selection refactor for details.

If you need further tuning, check your hardware resources and ensure your environment is not I/O bound. Optimizing disk access and reviewing concurrency settings are the most actionable steps based on the current codebase.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[hao310810401]

What default parameters can be modified to optimize? Does version 0.21 support multithreading?

[dosubot]

You can optimize batch parsing in RAGFlow by tuning several default parameters. The most impactful are concurrency and batch size settings, which you can set via environment variables in your .env or Docker Compose files. Key parameters include:

• MAX_CONCURRENT_TASKS (default: 5): number of parallel parsing tasks
• MAX_CONCURRENT_CHUNK_BUILDERS (default: 1): parallel chunk builders
• MAX_CONCURRENT_MINIO (default: 10): parallel MinIO operations
• EMBEDDING_BATCH_SIZE and DOC_BULK_SIZE: batch sizes for embedding and document parsing

For example:

MAX_CONCURRENT_TASKS=4
MAX_CONCURRENT_CHUNK_BUILDERS=2
EMBEDDING_BATCH_SIZE=4
DOC_BULK_SIZE=4

You can increase these values if your hardware allows, but monitor for memory or CPU exhaustion and restart your containers after changes. For OCR-heavy workloads, you can also increase rec_batch_num in deepdoc/vision/ocr.py (default: 16), but this must be changed in code and will use more memory. Reference Reference

Version 0.21 does support multithreading and asynchronous parallelism for parsing tasks. The codebase uses Python's ThreadPoolExecutor and trio for concurrent processing in figure parsing, OCR, and other routines. You can control the level of parallelism with the above environment variables and, for some parsers, with the PARALLEL_DEVICES setting (for example, in PDF parsing). Reference Reference

For chunking, parsers like RAGFlowJsonParser expose max_chunk_size (default: 2000, multiplied by 2 internally) and min_chunk_size (default: max(max_chunk_size-200, 50)), which you can adjust in code if you process large JSON/JSONL files. Reference

In summary: yes, v0.21 supports multithreading, and you can optimize batch parsing by tuning concurrency and batch size parameters via environment variables and, for some cases, by editing parser code directly.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[hao310810401]

What are the parameters for parsing files that support multiple concurrent executions in v0.21?

[dosubot]

In RAGFlow v0.21, you can control concurrent file parsing using these environment variables:

• MAX_CONCURRENT_TASKS (default: 5) — sets the number of parallel parsing tasks.
• MAX_CONCURRENT_CHUNK_BUILDERS (default: 1) — limits parallel chunk building.
• MAX_CONCURRENT_MINIO (default: 10) — controls parallel MinIO operations.
• DOC_BULK_SIZE (default: 4) and EMBEDDING_BATCH_SIZE (default: 16) — set how many documents or embeddings are processed in each batch.
• PARALLEL_DEVICES (default: 0, auto-detected GPU count) — for some parsers (like PDF), this sets the number of parallel devices/threads for parsing and OCR.

Set these as environment variables before starting RAGFlow to tune concurrency for your hardware and workload. Some module-specific concurrency (like figure parsing) uses a hardcoded thread pool size (max_workers=10) and would require code changes to adjust further.
Reference 1 | Reference 2 | Reference 3 | Reference 4

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[KevinHuSh]

Start more task executor in container, python rag/svr/task_executor.py.