Optimize DB calls for list endpoints

[zhiltsov-max]

Motivation and context

• Improved performance of GET /api/tasks and GET /api/jobs requests

Current requests to list endpoints consist of 2 DB queries: COUNT and page objects. The COUNT query uses the same subquery that's used for page objects.

Problems:

1. Given that we actually compute COUNT DISTINCT, the DB does this over all the columns (including from select_related()). This results in degraded performance for no actual reason, because we actually need the COUNT of unique ids.
2. The DB has to compute the same joins for the subquery twice, and it also reduces the overall performance.
3. List endpoint filters and standard permission checks can result in queries that have a number of OR clauses in the WHERE clause. This prevents Postgres from using column indices in these requests, despite having them available.
4. The COUNT queries use some other clauses that prevent it from using indices as well (e.g. ORDER BY, COUNT <columns>), while they do not affect the result.

How has this been tested?

Manual testing on a large synthetic DB.

Checklist

• I submit my changes into the develop branch
• I have created a changelog fragment
• I have updated the documentation accordingly
• I have added tests to cover my changes
• I have linked related issues (see GitHub docs)

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.

[sonarqubecloud]

Quality Gate passed

Issues
8 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[codecov-commenter]

Codecov Report

❌ Patch coverage is 21.15385% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.42%. Comparing base (5499aa9) to head (329ff73).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9851      +/-   ##
===========================================
+ Coverage    73.39%   75.42%   +2.02%     
===========================================
  Files          410      411       +1     
  Lines        45663    45721      +58     
  Branches      4086     4086              
===========================================
+ Hits         33516    34485     +969     
+ Misses       12147    11236     -911     

Components	Coverage Δ
cvat-ui	77.16% <ø> (+<0.01%)	⬆️
cvat-server	73.95% <21.15%> (+3.73%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
8 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud