Airtable + pagination

I implemented a data source with Airtable which was pretty straightforward. I’m running into a problem as Airtable will return a max of 100 records at a time. If the table has more than 100 records, Airtable will return an offset value that you have to pass back to the API to get the next 100.

There’s not really a way to do this with a data source that I can see. The GET in Airtable returns an object like:

{ 
  records: array of the table schema,
  offset: the value to pass back to the API in the next call to get the next 100 records

}

I’ve set the Response Key Path to “records” to get the data. How can I access the “offset” value?

1 Like

The REST API direct integration doesn’t support that at the moment, unfortunately.

The easiest way to get around this is to use the HTTP request node instead, so you get your hands on the whole response – you can still use Set data variable, you would just set it to the formula outputs["HTTP request"].resBody.records, and access the offset via outputs["HTTP request"].resBody.offset.

1 Like

Perfect - thank you!

1 Like

If I want to fetch entries of an airtable base that exceeds 100 in a list item,
as you said that I should use the http request node to request data and I did and verified by an alert that data is pulled.
but can you please guide me that how to populate the list item with this data?

seeking for your help

If you know how many records you need, a naive way is to just string together multiple HTTP request nodes with different offset values, and finally merge their response outputs into one array with a formula like

CONCAT(outputs["Get 1-100"].resBody.records, outputs["Get 101-200"].resBody.records)

Alternatively you can build a loop with e.g. a page variable for fetchMore that is set to true if the API returns info about the next page and then serves as an IF condition clause to determine if more data should be fetched or the loop finished. Each set of fetched data would then be added to a page/data variable array with the CONCAT function before checking if more data should be fetched still.

1 Like

Thanks @Harri_Sarsa it worked !! Now I am able to fetch more than 100 items,
but there is another issue that by adding Http request logic, the app got very slow. I can not tap any component.
any help?

The first thing to look is the looping logic — could you be triggering the loop twice in every iteration so you’d have an exponentially growing load? Can you post a screenshot of your logic?

1 - HTTP request JOIN([“https://api.airtable.com/v0/appy/CreatedListings?api_key&offset=”,appVars.offset])

2 - Set app variable CONCAT(appVars.alllistings, outputs["HTTP request"].resBody.records)

3 - Set app variable outputs["HTTP request"].resBody.offset

4 - if condition IF(IS_EMPTY(appVars.offset),true,false)

5 - Set app variable appVars.loading to yes[text] ( just to signal that the loading is complete)

If you remove the wire going straight to the Delay node from the HTTP request, you should be good – now if appVars.offset is not empty, every execution of HTTP request causes exponentially growing number of loops.

Airtable___pagination_-_AppGyver

1 Like

Removed the mentioned wire, but taking long time same as before :pensive:

You’re still using Delay with 500ms so if you have to loop that a few times, it’ll take some time.

I think you should cut the line from Flow function #4 (If condition) and instead connect directly back to HTTP Request

The other part is that, if you repeat the records (the app variable) on page, after each Set app variable all will be rendered from scratch. This might make your app slow if you have a lot of components to repeat. I’d consider adding some pagination at this point as huge amount of items take some time to render.

(Also, asking just to be safe: Have you checked that the offset you’re using actually increases after each iteration?)

I’ve problems with concat. If I don’t use concat (Just to try) data variable is assigned to response record. If i use concat (also without iterations) it gives me null values. It seems there is an error, but can’t understand wich.
Any idea?

I have done what you said as follows:
1 - I have cut the line from Flow function #4 (If condition) and instead connect directly back to HTTP Request
2 - And also I have convert offset variable from app to page

But the same problem exists :pensive:

And also there is new problem that the text data which I want to fetch are showing in from app preview portal in laptop but in android appgyver app the data is invisible :confused:

And I have checked that the offset increases and the data is iterating

Can some appgyver member show me how to paginate my items?

Hi Viraj, this may not be the most elegant way, but it works for me.

I start populating my client list (I’m retrieving a list of clients from Airtable) by calling a ‘HTTP request’ flow function when my client page is mounted. If there are more than 100 records (Airtable’s limit) or if you’ve specified a pageSize less than the records available, the response will contain an offset value.

I created a page variable called client_offset, which I assign the offset value to (if it exists).

I then have a button at the bottom of my client list, which becomes visible when my client_offset variable is not empty. Clicking on the button calls a new HTTP request, which includes the offset value. The URL for my HTTP request (button) is as follows:

https://api.airtable.com/v0/[YOUR_BASE_ID]/Client?fields[]=Name&fields[]=Thumbnail&offset=”+ pageVars.client_offset +"&view=Grid%20view"

You obviously need to store the new offset value with each GET and use that in your next HTTP request.

I then add the new list to my existing list, as follows: UNION(pageVars.client_list, pageVars.new_list)

Hope this helps.
Francois