Airtable 100 item limit

I’ll start off by stating that I’m a newbie, so forgive me if this is basic. I was able to follow the API setup using Airtable but I need the full dataset and Airtable limits the amount you can pull into Appgyver to 100 records using the data variable. I tried following the instructions in the forum found in Airtable + pagination and a few similar posts but have been unable to replicate the results.

What I’ve done: I downloaded the HTTP Request node and added it to the logic of my page. I put my Airtable URL with key into the HTTP Request URL as a string. I am able to see the data is being pulled in via an alert that shows on my phone, so the data is there. I’ve been trying to use the set data variable and put it into an app variable that I can call on to put in a repeating list, but this is where I get lost. The set data variable name I set to the output of another node (the HTTP Request node), but the only option is outputs[“HTTP request”].error.rawError. I set this to a formula and instead make it outputs[“HTTP request”].resBody.records. Then I created an app variable. That’s as far as I can get. I believe I’m either doing something wrong, missing steps, and/or need to create the app variable schema? I know that I need to essentially recreate from scratch the steps that are automated via the API tool in the data folder, but it’s unclear to me what I need to do to see the data in my list. Also, I don’t understand how to get the offset data that would give me my entire dataset post 100 items. Any helps for this newbie??? I’ve been trying to tackle this for about 3-4 days now with no luck :frowning:

1 Like

You get the offset in resbody.offset

In the next request, you can send back the offset by appending or concatenating to the HTTP request like this “…?offset=”

Put an if condition that repeats the request only if offset is not empty. Once offset is empty, that means you have got the last “packet”

And most important: put a delay/pause of at least 200-300 milliseconds between each query else AirTable will punish you: no more than 5 requests in a second.

1 Like

Thank you Nihal. After writing my question I was able to get the ?offset= and can see it in an alert. I’ll have to try adding the if condition to the logic and see what happens.

The problem I’m running into now is that I want to be able to put all the HTTP Requests into a single variable (app or page) and I don’t know how to get them into a single variable. The logic seems to overwrite the previous data saved to the variable so I’m only keeping 100 items at a time in a variable. What I’m trying to do is put everything from the Airtable base into a list that is searchable and filterable. Is this possible? Thanks again!

Any ideas how I could incorporate a search into an airtable list?

Use the concatenate function to add the updated list to the variable, like this: “CONCAT(appVars.alllistings, outputs[“HTTP request”].resBody.records)” (this happens in the block after the HTTP request, and before the Set app variable.

2 Likes

Thanks again Nihal. I’m confused about where the CONCAT formula goes. You said after HTTP Request and before Set app variable? Does it go in the last Utility block? I’m not following the logic of how this puts all records into one variable. I’m getting very close I think but still not quite there.

Also, how do I set last varialbe to loadingcomplete to yes?

After the HTTP Request:
(1) Concat the listings with the HTTP response. At the first instance, the listings will be empty, so concat empty with HTTP response results in the first set of listings. In subsequent responses, the listings will get added.
(2) get the Offset from the HTTP response and store it in a local variable. The order of 1 and 2 can be interchanged.
(3) check if offset is empty. If empty, loading is complete (go to step 4) If not empty, go back to HTTP request with the API delay and append the offset to the HTTP request

To set loadingcomplete to yes, I declared it as a text, but you can use any variable type you prefer (binary/number/etc). I just use that to signal that the loading is complete. Instead, you can directly now load your container with the data from the listings variable at this point.

1 Like

can you please share the formula inside the set app variables, if condition and others … as it will be more clear

HTTP Request: JOIN([“https://api.airtable.com/v0/appy/CreatedListings?offset=”,appVars.offset])

step 1: set alllistings to: CONCAT(appVars.alllistings, outputs[“HTTP request”].resBody.records)
step 2: set offset to: outputs[“HTTP request”].resBody.offset
step 3: if condition: IF(IS_EMPTY(appVars.offset),true,false)
step 4: set loadingcomplete to: yes

So what this does is send an HTTP Request (fill in your value with the right url for your AirTable) with appends an ?offset= at the end of that URL. In your first call, the offset is empty, so AirTable returns the first hundred items. At the same time, airtable also returns an offset.

Then you concat those 100 items in allistings in step 1 (with an initial empty list), and store the offset provided by AirTable in a local variable (called offset or anything else you prefer).

In step 3, you check if the offset returend by AirTable is empty. If empty, it means there are no more rows. If that’s the case, you fork to step 4, and set loading complete to yes. If the returned offset is not empty, you repeat the HTTP request (after a delay of about 500 ms to avoid your AirTable API getting blocked).

The next the HTTP request is called, it will append the offset= at the end of the URL with the offset AirTable returned in the previous call. This prompts AirTable to send the next 100 records, and a fresh offset if there are more records. The new batch of records get concat to the previous hundred records in step 1, and the cycle continues until AirTable returns an empty offset, which means all rows have been loaded.

1 Like

Thanks Nihal for the detailed answer.

All done from the Logic side. Can you also share that how to populate the listing (airtable base) data in a list item using the HTTP Request method?
As we can Repeat with the list item with the Data variable of collection of data records via REST API method.
chrome-capture (1)
But what to Repeat with with the HTTP Request method. And what to bind with Primary Label & Secondary Label to populate the data in the list item?

Hello, so now think in this way. Phase 1 was to load the data from AirTable to an AppGyver object. In the example I used, all my data was pulled into an appvariable called all listings.

Now, in Phase 2, to show that data in a repeating list, I created a ScrollView. Inside the ScrollView, I created a Container. In that container, I set “Repeat with” to that appvariable. In other words, now the container will repeat with “rows” of the AirTable.

But AppGyver will not know automatically what to show inside the container. All it knows is that each container corresponds to a row of data you got.

So now, you have to tell AppGyver what to show (which “columns” for the current row) inside that container, depending on the schema of your data. My schema has “fields” as shown below

So, to show some of these text fields, inside the container, create a Title, Paragraph, Card, or some suitable text box inside. Set the Title to current.fields.[selectedfield1]. Add other text boxes for [selectedfield2] and so on. Similar for image. If your AirTable has an image, you can create an image inside the repeating container, and point that to current.fields.[imagefieldname]

Thus, the whole container repeats with each item of alllistings (rows), and within the container, you can pick and choose the fields (columns) you want to display.

1 Like

Also, to add to my reply above, once you load the data into AppGyver, the subsequent steps are independent of the method used to load the data. In other words, whether you used API, HTTP request, or whatever other technique, to load the data in phase 1, that method does not matter in phase 2 (displaying the data in the repeating container). The only way it matters is in the way you read the schema.

1 Like

I’ve problems with concat. If I don’t use concat (Just to try) data variable is assigned to response record. If i use concat (also without iterations) it gives me null values. It seems there is an error, but can’t understand wich.
Any idea?

Hi @Andrea_Costigliolo do you still have the problems with CONCAT? If so, could you give a little more details how you have used it? i.e. the Formula you are using and logic configuration outline would be a good start

@Nihal_Parkar I wanted to say thank you very much for the above explanation.

I was previously importing my entire datasets without using App variables (using API directly), and it seems this method is much more efficient.

1 Like

I finally got this to work for the first time today. Yes, I’ve been working on this on and off since June of 2020. One of the biggest problems I had was recognizing that this method of getting data from Airtable is a COMPLETELY SEPARATE METHOD from the data variable method. What you are doing here is you are totally side stepping the the data configurator API method and simply using an HTTP request to do the same thing. Then you are putting the data into an app variable (instead of the data variable).

As a newb, this was not obvious to me at first and I spent many hours struggling to figure out how to get my page logic to jive with the data variable. I’m mentioning this in the hopes that it may save someone else hours and hours of frustration. If someone needs help with this issue I’m happy to respond with more and walk through it but I don’t have time at moment. Ping me and if I see it I’ll respond.

1 Like

Hello Brandon,
I keep reading this post but my knowledge is limited.
I have set up Airtable API succesfully but this method confuses me.
If you can, when you can, can you please post some info with photos how you did it?
(The idea is: app loads, home screen with 20 news posts, bottom of page there a next button, loads another 20 posts…and so on. What happens when user reaches page 6?)

Thanks in advance

Hi Edvin_Rushitaj!

At the end of this issue i have posted images with all stepts i followed and got it Airtable http request (100 item limit / pagination).

Bye
Mei Li

Oh I’m sorry for not responding sooner! I’ve been distracted with other projects and haven’t used Appgyver since my last post. Just using again now for a new project. Were you able to get it working? Not sure I remember all the steps at the moment but as I re-familiarize myself hopefully it will come back to me if you need help.