Airtable 100 item limit

I’ll start off by stating that I’m a newbie, so forgive me if this is basic. I was able to follow the API setup using Airtable but I need the full dataset and Airtable limits the amount you can pull into Appgyver to 100 records using the data variable. I tried following the instructions in the forum found in Airtable + pagination and a few similar posts but have been unable to replicate the results.

What I’ve done: I downloaded the HTTP Request node and added it to the logic of my page. I put my Airtable URL with key into the HTTP Request URL as a string. I am able to see the data is being pulled in via an alert that shows on my phone, so the data is there. I’ve been trying to use the set data variable and put it into an app variable that I can call on to put in a repeating list, but this is where I get lost. The set data variable name I set to the output of another node (the HTTP Request node), but the only option is outputs[“HTTP request”].error.rawError. I set this to a formula and instead make it outputs[“HTTP request”].resBody.records. Then I created an app variable. That’s as far as I can get. I believe I’m either doing something wrong, missing steps, and/or need to create the app variable schema? I know that I need to essentially recreate from scratch the steps that are automated via the API tool in the data folder, but it’s unclear to me what I need to do to see the data in my list. Also, I don’t understand how to get the offset data that would give me my entire dataset post 100 items. Any helps for this newbie??? I’ve been trying to tackle this for about 3-4 days now with no luck :frowning:

1 Like

You get the offset in resbody.offset

In the next request, you can send back the offset by appending or concatenating to the HTTP request like this “…?offset=”

Put an if condition that repeats the request only if offset is not empty. Once offset is empty, that means you have got the last “packet”

And most important: put a delay/pause of at least 200-300 milliseconds between each query else AirTable will punish you: no more than 5 requests in a second.

Thank you Nihal. After writing my question I was able to get the ?offset= and can see it in an alert. I’ll have to try adding the if condition to the logic and see what happens.

The problem I’m running into now is that I want to be able to put all the HTTP Requests into a single variable (app or page) and I don’t know how to get them into a single variable. The logic seems to overwrite the previous data saved to the variable so I’m only keeping 100 items at a time in a variable. What I’m trying to do is put everything from the Airtable base into a list that is searchable and filterable. Is this possible? Thanks again!

Any ideas how I could incorporate a search into an airtable list?

Use the concatenate function to add the updated list to the variable, like this: “CONCAT(appVars.alllistings, outputs[“HTTP request”].resBody.records)” (this happens in the block after the HTTP request, and before the Set app variable.

Thanks again Nihal. I’m confused about where the CONCAT formula goes. You said after HTTP Request and before Set app variable? Does it go in the last Utility block? I’m not following the logic of how this puts all records into one variable. I’m getting very close I think but still not quite there.

Also, how do I set last varialbe to loadingcomplete to yes?

After the HTTP Request:
(1) Concat the listings with the HTTP response. At the first instance, the listings will be empty, so concat empty with HTTP response results in the first set of listings. In subsequent responses, the listings will get added.
(2) get the Offset from the HTTP response and store it in a local variable. The order of 1 and 2 can be interchanged.
(3) check if offset is empty. If empty, loading is complete (go to step 4) If not empty, go back to HTTP request with the API delay and append the offset to the HTTP request

To set loadingcomplete to yes, I declared it as a text, but you can use any variable type you prefer (binary/number/etc). I just use that to signal that the loading is complete. Instead, you can directly now load your container with the data from the listings variable at this point.