Skip to content

Sync: SDK loops to get All entries without backpressure #271

@t-botz

Description

@t-botz

Context

Trying to load Contentful every month to Elastic search, incremental updates done via webhooks.
We have a very large space with 40k+ entries.

Issue

When doing:

 contentfulClient.sync().fetch()

We get

You have exceeded the rate limit of this Space by making too many API requests within a short timespan. Please wait a moment before trying the request again.

Expectation

The SDK should let us have some back pressure. We didn't expected the SDK to loop using the sync rest api and trying to load all entries in memory. Given the SDK use RxJava and Flowable it would make sense to yield each result from the sync API.

We couldn't find an easy to extend the Contentful SDK to do that.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions