Pulling large quantities of data through API

Hi!

For our use case, there will be a need to potentially pull large quantities of historic data - depends on a business. For new data (since business installed the app) we can use webhooks.

How would you recommend we do this - pulling of large quantities of data?

@dennis Thanks for your previous answer!

Thanks!
Ivan

You can get 100 transactions in one API response, so that would give me 400 transactions/second (at 4 API calls/second). Even for a large merchant you should be able to backfill the data pretty quickly, no?

Hi Dennis,

Sorry for late replay. Looks like I unchecked something in my profile because I got no notification email for your post.

In theory it should be quickly. But. We have a merchant that has 300-400 orders per day. Sometimes 500+. If we would like to pull all their orders and transactions for the last let say 2+ years, it would take quite a while.

We send 4 calls, then we wait for their return to know if there are more data to pull - so not to ‘hammer’ unnecessarily with constant 4 calls/sec while waiting for lets say just 200 entities, and not to lock out our other merchants because of the rate limit. Those calls do take some seconds. So it could take hours (orders + transactions).

We’ll try to see what the best approach is.

Thanks!
Ivan

@IvanQ how do you pull data , do i have to spend sdk development cost and develop app or i can do it from java from my desktop ?