Skip to content

Add journalEntries view - #2697

Open
aldo-expensify wants to merge 2 commits into
mainfrom
aldo_add-journals-view
Open

Add journalEntries view#2697
aldo-expensify wants to merge 2 commits into
mainfrom
aldo_add-journals-view

Conversation

@aldo-expensify

@aldo-expensify aldo-expensify commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Details

Adds a view journalEntries to UNION all journal tables so it is easier to write custom queries targeting all journal tables

The view (click to view it)
CREATE VIEW journalEntries AS
SELECT
    id,
    query,
    hash
FROM
    journal
UNION
ALL
SELECT
    id,
    query,
    hash
FROM
    journal0000
UNION
ALL
SELECT
    id,
    query,
    hash
FROM
    journal0001
UNION
ALL
SELECT
    id,
    query,
    hash
FROM
    journal0002
UNION
ALL
SELECT
    id,
    query,
    hash
FROM
    journal0003
UNION
ALL
SELECT
    id,
    query,
    hash
FROM
    journal0004
UNION
ALL
....

The code here should keep the view up to date with all journal tables if the table number ever changes.

Currently this is done in the readdb.sh here: https://github.com/Expensify/Salt/blob/554c37ac5d10959f8d2ae6890a925c50ec614f13/shared_pkgs/bedrock/files/readdb.sh#L309-L355

I think we can drop that special code and param -j from readdb.sh if we create this view.

Fixed Issues

Fixes https://github.com/Expensify/Expensify/issues/661721

Tests

  1. Compile Bedrock
  2. Restart bedrock
  3. Verify that the journalEntries view is created correctly in the bedrock db
  4. Compile Auth using this version of bedrock
  5. Restart auth
  6. Verify that the journalEntries view is created correctly in the auth db

Internal Testing Reminder: when changing bedrock, please compile auth against your new changes

@aldo-expensify
aldo-expensify marked this pull request as ready for review July 30, 2026 06:33
@aldo-expensify
aldo-expensify requested a review from Copilot July 30, 2026 06:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a journalEntries SQLite view that unions all physical journal shard tables, providing a stable way to query journal data without having to dynamically build UNION statements across journal* tables. It updates tests and documentation to use the new view.

Changes:

  • Add automatic creation/update of a journalEntries view during SQLite journal initialization.
  • Update cluster tests to query journalEntries instead of enumerating journal* tables.
  • Update documentation to describe querying journal data via journalEntries.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
sqlitecluster/SQLite.cpp Creates/refreshes journalEntries view to UNION ALL all existing journal shard tables.
test/clustertest/tests/CompressionTest.cpp Switches journal reads in tests to use journalEntries.
test/clustertest/tests/BoundParametersTest.cpp Switches journal reads in tests to use journalEntries.
docs/blockchain.md Documents the new journalEntries view as the supported way to query across journal shards.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread sqlitecluster/SQLite.cpp
SQResult journalEntriesView;
SASSERT(!SQuery(db, "SELECT type, sql FROM sqlite_master WHERE name = 'journalEntries'", journalEntriesView));
if (journalEntriesView.empty()) {
SASSERT(!SQuery(db, journalEntriesViewQuery));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is instant and thus safe to run when we deploy this code?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is my understanding

@tylerkaraszewski tylerkaraszewski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm approving this because I think this code is fine, though I wonder if anything that actually uses it will have performance issues.

Comment thread sqlitecluster/SQLite.cpp
SASSERT(journalEntriesView.size() == 1);
SASSERT(journalEntriesView[0][0] == "view");
if (journalEntriesView[0][1] != journalEntriesViewQuery) {
SASSERT(!SQuery(db, "BEGIN IMMEDIATE"));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this do?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like it starts the transaction as write transaction from the start. Without it SQlite would wait to see a write statement before making it a write transaction. That is what I understood from the documentation at least: https://sqlite.org/lang_transaction.html

I'm guessing this would still work if you drop IMMEDIATE.

@aldo-expensify

Copy link
Copy Markdown
Contributor Author

though I wonder if anything that actually uses it will have performance issues.

Same issues we would have with readdb.sh -j, right? or do you see any difference?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants