Efficient Data queries

Question
,
#1

Hi,

So I have a general query about best practices. In several pages of my app, I am dealing with several data resources. Earlier, I was using the default add data variable option which used to populate collection of data records but then I realised that it is limited to 100 records only (my backend is airtable)

So now I am using the HTTP request along with Offset to load all data into my data variable. Now some of my tables will eventually have 1000s and possibly 10,000s of records. That means i will be running the HTTP request possibly 100 or more times which seems to be a bit of overkill.

The reason I am doing this is in case I want the user to be able to search the database, I need to have all records available to search from as I don’t know if the relevant record will be retrieved in the 1st GET operation or the 100th.

This seems a bit inefficient. Is there a better way to do this?

#2

Hi,

If the only reason to have all those records loaded is to have them searchable, it might be best to implement the search in the backend instead of in the app. Consider that all of the 10000s of entries will be loaded and stored to the device’s memory, and that airtable has a limit on API calls per second as well - loading them all will take a long time, much more than is offset by the potentially quicker in-memory search compared to another API call.