limit of the new cursor api

1

Hello,

I would like to do some preliminary study about the new Cursor API. I have learned 2 points as below:

  1. there would be an upper limit (10000) for the offset parameter of ‘Get Records API’.
  2. there would be an upper limit (10) for the active cursors within one domain.

Based on these 2 points, I have a question, when the number of active cursors reaches the limit of 10, seems I have to turn
to the “Get Records API” instead, but I can not meet some requirements in this way if the offset parameter reaches the limit
of 10000.
Do you have some suggestions?

 

Thanks,

Aaron

2

The best practice here is to keep using the Cursor API without switching to the Get Records API.
Although cursors will have a lifespan, cursors can be deleted after all records are retrieved from them, or the cursor is deleted with the Delete Cursor API.
Deleting the cursors will open a new slot for a new cursor to be created.

The management of the cursors (creating and deleting them in a timely manner) will be the key here.

3

Thanks for your quick reply.

I understand that cursor can be deleted after usage, but I am concerned about the case when the number of cursors in use reach the limit of 10. In this case, should I re-send continuously until there is a new slot for a new cursor to be created?

4

It depends on the use-case.

Could you explain the scenario in which you reach the 10 cursor limit, as there may be multiple ways you can encounter this.

5

Sure. Since the limit is ‘can create up to 10 cursors at one time per domain’, so I think a typical scenario is that when I send request to create a new cursor, there’re already 10 cursors created and still active within the domain for other apps which I don’t even have a chance to manage. 

6

The method of data retrieval may vary depending on the order you need to state in the query.

If your use-case is acceptable to have records retrieved in order of Record IDs, then we suggest not using the Cursor APIs, but to use the Get Records API to retrieve the data. In this case, please refer to the Get all records from an App (the seek method) article on how to implement this.
The specific code sample can be found here, which does not state any offsets.

If the order needs to be more complex, please share us an example query that would be frequently used.

 

Is my understanding correct, that you are trying to implement a customization on an App that would retrieve all record data from that App, and that this same customization may be applied to several other Apps?

7

Thanks for your suggestion.

Yes, we are considering the seek way for one of our use scenario where the records are only in order of Record IDs.

In this scenario, I have made a quick test with both the cursor way and seek way. Due to my test result, the seek way has a similar performance with the cursor way, when the data is not very large, cursor way is bit of faster than seek way, but when the data gets larger, seek way gets even faster compared with cursor way. In consderation of this test result and also the cursor limit, I think seek way is a better choice in this scenario.

By the way, both these two ways are about 1 time slower than the offset way we are using now since we use some asynchronous method (like Promise.All) to speed up it. Do you have some suggestions about the performance?

 

In other of our use scenario where we allow users to customize their orders when retrieving data with an arbitrary paging, I havn’t thought of a proper way.

8

Thanks for testing the codes out on your end.

As for performance for mass data extraction, the best practice is to keep the query simple.
The more complex the query, the more time it will take for all of the records to be retrieved.
Any efforts made to simplify the query made to the server side, and sorted/filtered on the client side should direct towards better overall performance, though you may need to test for a good balance between server and client side efforts. Hope this helps!