feature/Jump to Last Change

[BeckettFrey]

Problem

Currently, users must manually iterate to find the most recent prior version of a page where any change occurred. This is inefficient, especially for pages with many versions, and makes it hard to track specific updates or regressions.

Proposed Solution

Add a function and/or API endpoint (e.g., getMostRecentChange(pageId, currentVersionId, diffType)):

• Iterate backwards from the current version (selected to version) and find the most recent version whose contents differ (using body_hash, or by calling the diff API with a fast diffType such as html_text_dmp).
• Return the UUID and basic metadata of the previous version where a change actually occurred.
• Expose this function in the frontend and/or backend API so it can be used to power a "Jump to Last Change" feature in the UI.

[BeckettFrey]

Hey @Mr0grog thanks again for all your guidance on the previous PR, this mission really resonates with me both morally and by subject matter. Would it make sense for me to join the org, or is there a usual path for becoming a member?

[Mr0grog]

Currently, users must manually iterate to find the most recent prior version of a page where any change occurred.

Ah, yeah, you are looking at this from the perspective of just this JS front-end. For the most part, people arrive at this UI from tasking/analysis spreadsheets that are output every week by https://github.com/edgi-govdata-archiving/web-monitoring-task-sheets

So analysts who are monitoring these sites generally start with a row a in spreadsheet that contains a bunch of other info about what changed (and how likely that change is to be meaningful), and that links them directly to a comparison here that is already fairly well narrowed-down to a time range that covers a notable change.

There is a much longer story about why things work that way and how they might be different, but major changes to that overall workflow are not really on the table right now for a lot of reasons.

• Iterate backwards from the current version (selected to version) and find the most recent version whose contents differ (using body_hash, or by calling the diff API with a fast diffType such as html_text_dmp).

The main problem here is that none of these diffs are really fast per se, or at least not fast enough for this kind of usage. We used to use body_hash for this (see the different field), but it turns out it just does not work very well. The vast majority of pages have something different in them every time (e.g. a CSP or XSS nonce) that is not meaningful from a content perspective, but does cause them to have a different hash.

There are a lot of neat ways we might optimize this or pre-calculate comparisons (specialized diffs, hashing a simplified version of the page, simhashes, etc.), but there’s a lot of work and nuance in doing that effectively and efficiently since a) a good algorithm will change over time and b) "versions" can get added to the system out of order from various sources, you can’t just compare two versions consecutive versions of a page and be done forever. That work also mainly belongs in https://github.com/edgi-govdata-archiving/web-monitoring-db

Would it make sense for me to join the org, or is there a usual path for becoming a member?

If you are interested in joining EDGI and volunteering more broadly, you can sign up at: https://envirodatagov.org/volunteer/

[BeckettFrey]

Thanks for the thorough context that helps a lot. I hadn’t realized how closely the UI ties into the larger task-sheet workflow. I get why adding automated backward-diff traversal isn’t as straightforward or necessarily beneficial in the broader pipeline.

Definitely more nuance here than is visible from the front-end layer. I appreciate you laying it out. Maybe someone with deeper knowledge of the backend (maybe even future me) can explore new approaches later on.

And thanks for the EDGI volunteer link, I'm definitely interested and will sign up.

[Mr0grog]

Yeah, sorry to be a curmudgeon with a lot of "wrong repo!" or "you're misunderstanding this [thing that is totally unexplained so why would you know]!" answers. 😞 Part of the issue is that there's just a lot of legacy cruft here, and another big thing is just how many small repos and services this project is spread across (not unrelated to the other legacy stuff!). It's not in a place where it's very easy for someone less expert to get into, and that's not your fault.

I think I'd assumed you'd seen the overview repo for the Web Monitoring projects, but if not, you should check out: https://github.com/edgi-govdata-archiving/web-monitoring (warning: the ARCHITECTURE doc is out of date).

If you're still really interested in this (or other projects at EDGI!), you should definitely fill out that volunteer form, but I’m also happy to do a video call or just talk more via e-mail (see my GitHub profile) about how stuff is set up and cover more background.

[BeckettFrey]

That’s a generous offer, I'll shoot you an email. I think talking through things will help me absorb the big picture more quickly, especially if some docs are out of date.

In the meantime, I’ll check out the overview repo (already filled out the form).

Really appreciate it!