[API Request] update/changed timestamp value on all entities #32
Replies: 2 comments 14 replies
-
|
+5000 :) But needs the rest of the API (Sorting and filtering on it) to be really useful and start to offer differential sync to clients. There's also the complexity of knowing the kind of changes that happens. Typically the changedValue can't indicate that a song is removed from the database. Or artists ... There's also indirect changes like an album is changed, but the change is a new artist, you then need to also sync that new artist. And doing differential on every entry to detect those can be highly inefficient. So it to resume it would be nice to extend this to allow proper differential sync for offline first clients. |
Beta Was this translation helpful? Give feedback.
-
|
Well it's in the name, this is not really the place to discuss client side architecture. Anyway as you have probably noticed there's relations between things, songs / albums / artists. The relation is based on IDs. As soon as anything touches IDs like additions / deletions the situation becomes more complex. There's basically 2 ways to address things depending on the server. |
Beta Was this translation helpful? Give feedback.
-
|
I think this is the place to discuss this since we are talking about which approach we should use for improved syncing. I've worked on both types of systems. Those systems offering journalling and RESTful systems that focus on entity state. Systems that cause lots of changes are not necessarily a good fit for journaling since they create lots of entries in the change table and playing those changes back causes a lot of overhead. Especially when the same entity is changed multiple times or there are long delays between sync. Journaling is good for huge datasets where a small portion of the data is changing. So a complete song list for instance might be a good fit for journaling, maybe. Still not sure why you would choose to playback the journal to pickup deletions and a few other things then have to go off and get changes since x. Wouldn't the journaling contain the changes since x as well? After working on both systems, as a general rule, I'm in favor of a more RESTful approach where we simply offer the current state at the time of sync. It's far too easy for the journal to become too noisy and cause sync headaches. It's also much less prone to catastrophic bugs and easier to implement. Maybe I've just been bitten by journaling issues where the entire sync falls flat on its face due to iterating through too many journal entries or a missing entry causing database inconsistency. ;) |
Beta Was this translation helpful? Give feedback.
-
You are mixing things here it seems. Journaling is at the global database level if we want proper state and relationships. This sentence does not make a lot of sense in this concept.
Again this is not what I said, the journaling contains all the changes. Journaling is done at server side, knowing at what X time the client did sync is known by the client it then handle the journal actions after that X. Having the server do journaling by clients would be insane to ask to the servers.
Does not fit at all the requirement for differential syncs and optimized syncs. If you only expose the state at instant X this means that clients needs to get all the data so can only do complete sync..... We are talking about offline first apps that have a copy of the data and all the relationships. Those clients needs to know the changes that they can sync safely with differential sync (So asking for the data changed after the last date of sync) and know if there's changes that prevent that since that date and trigger a full sync. Having an endpoint exposing the last changes to differentiate, metadata/user data change, addition and deletions / relationship changes allows clients to decide what they do. Having the created/changed for each items + filter on them allows clients to only request the changed data and sync that changed data and not all the database. |
Beta Was this translation helpful? Give feedback.
-
That's not how REST sync works. And I never suggested anything like that. It typically has lists of entity types that contain change flags to determine if the client needs to sync a given entity. If there are too many entity types, the api would contain an endpoint that is a list of lists. Frequently changed data would be separated out in the api, or simply included in the list itself, to reduce overhead. For example, the api would have a favorite song list so the client would not have to process all 300,000 songs to get the changes. I also said getting a complete song list might be a good choice for differential sync because of the high number of entities. But do we really need this level of overhead for playlists, artists or favorite albums? Having an api that filters the data changes causes the remote client to process that list then iterate through the differential sync list to detect deletions. In the case of playlists, forcing the client to make at least 2 network calls then iterate through 2 lists when the data would most certainly fit in one mtu is not good either. My issue is does the api move towards a more RESTful approach or this un-RESTful hybrid approach that could actually take longer to sync, is harder to implement and generates more bugs. |
Beta Was this translation helpful? Give feedback.
-
|
You are again mixing a tons of different things and concept here .... The API is already mostly RESTful even if not using the HTTP verbs to handle modification and deletions. You seems to only think about playlists and groups of things like albums. I'm talking about proper offline first app with complete data sync .... There's no concept of favorite list or anything in that case, the app needs to have a copy of the 300 000 songs. To update the non breaking changes filtering by last changed is the only efficient in a rest way without a database log system .... For the rest sorry most also does not make sense about how things currently work. The playlist changed attribute only refer to changes to the actual playlist, so name or image or song list content / order change. If any song inside the playlist is changed like increased playcount or changed name or metadata then changed of the playlist is not changed at all. So in all cases to have the up to date data you already need multiple calls... Because if a playlist changed have not changed it does not mean the songs have not changed their data. |
Beta Was this translation helpful? Give feedback.
-
|
What does a change mean for non-atomic entities? For an album, has it changed if:
Same for, say, genres, considering constituent albums and tracks? |
Beta Was this translation helpful? Give feedback.
-
|
It would mean any property that is exposed for that entity. So there's also the scraped metadata or the cover art. And the number of songs if there's addition / deletion or replacement of some. Changes that don't impact returned data for an album like song metadata / user data change should not change this else clients would resync the album for no reason as it should already have detected the song changes and synced them. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Type of change
API Clarification
Proposal description
A change timestamp should be added to all entities so that remote clients can determine if their local cache is current without further action.
A change timestamp attribute should also be added to the top level list entity to indicate if any entities in the list have changed. The client can then skip iterating through the list if nothing has changed.
getPlaylists already contains the change attribute for this very reason but it was not added to the other entities.
For example, in the getStarred2 api. A remote client needs to iterate through the albums listed and subsequently call getAlbum to determine if the songs in the album have changed.
getStarred2 returns
Backward compatibility impact
adding as attribute to entity would not break compatibility
Backward compatibility
API details
All entities should contain a "changed" attribute containing a timestamp to reduce network traffic and processing time
Example:
<album id="1987" name="Graceland" artist="Paul Simon" artistId="404" coverArt="al-1987" songCount="22" duration="5173" playCount="180" created="2015-11-20T13:54:17.000Z" starred="2022-11-01T05:04:24.300Z" year="1986" genre="Blues" changed="2023-04-11T13:54:17.000Z"/>
Security impacts
none
Potential issues
No response
Alternative solutions
My current solution is to iterate through the entity lists, when the changed value is not available, but this obviously causes more network traffic and overhead.
Beta Was this translation helpful? Give feedback.
All reactions