So I have a general query about best practices. In several pages of my app, I am dealing with several data resources. Earlier, I was using the default add data variable option which used to populate collection of data records but then I realised that it is limited to 100 records only (my backend is airtable)
So now I am using the HTTP request along with Offset to load all data into my data variable. Now some of my tables will eventually have 1000s and possibly 10,000s of records. That means i will be running the HTTP request possibly 100 or more times which seems to be a bit of overkill.
The reason I am doing this is in case I want the user to be able to search the database, I need to have all records available to search from as I don’t know if the relevant record will be retrieved in the 1st GET operation or the 100th.
This seems a bit inefficient. Is there a better way to do this?