Airtable http request (100 item limit / pagination)

This is similar to a few other topics on the forum, but I am going to go into a fair bit of detail here…
I am trying to import a data table (into an app variable). This data is (only) used to drive some mathematical/statistical formulas. I had previously succeeded in importing via an API but the 100 record ‘limit’ for Airtable ensured that my formulas fail due to the incomplete table.
I have read over and over again a few forum topics about creating a HTTP GET request that replicates that API but pulls the entire data set, but I simply cannot get it to work…so below is everything I have done. I am new to this and talking about no coding ability, that’s me…even things like api-keys I need to Wikipedia…it is a struggle at times but I usually get there but this has me stumped….
Note: I have this attached to a ‘button’ at the moment, longer term it would be ‘page mount’
Note II: One of the fundamental problems I have is not knowing how to test whether the http GET is successful. I tried populating an appVar with the first 100 records and couldn’t see it working in the debugger, so I am flying blind. Admiteddly, I am new to the debugger! But moving on….

My canvas….

image

  1. http request
    image

My http is https://api.airtable.com/v0/app6Fwdfozm8zT0N2/ShuttleTimes and I have the api key in the header looking like this;
image

I have also tried it with the api keep appended to the URL and no header.
My join formula is;
image

  1. App Variable = Test
    image
    image

The ‘Test’ app variable is created with the following schema;
image

  1. App Variable = offset
    image
    image

The ‘offset’ app Variable is created as type = text;

  1. Utility IF condition
    image

  2. ‘loading complete’ app variable is created as type = text

Repeating myself…help generally would be appreciated but how I ‘view’ these http requests in the debugger.

Thanks
Paul

In your JOIN, you will need to enclose everything in the parenthesis in brackets, and after the brackets, show what character is joining the members of the list. Here, it will be an empty character.

Unless you do that, AppGyver does not know what to join and how, which is why you see the validation warning above :slight_smile:

As an example, use this format:
JOIN([“https://api.airtable.com/v0/appy/CreatedListings?offset=”,appVars.offset],"")

JOIN([comma-separated members of list],"")

Also, did you confirm you see a response in the Data Configurator? You can test the API call that way.

You can also use Postman (or run the API query directly in the browser to see the raw results) to confirm if you are getting something in return.

Hi Nihal

Thanks for [ ] help. I had tested the http in an online tool (reqbin.com) previously but dropping it straight into the web browser had not occurred to me. I quickly grabbed the offset value and appended that and got the next 100 values too…I can see why those brackets are crucial now.

In terms of testing the API, yes, I see a response from that. Some of the formulas that use the output work and I can see the below in the debugger;

With the http request, I am expecting to see ~250 records here, but nothing;

Getting closer…but not quite!
I think I have found the problem. I created a different button linked to a http request and an app variable that pulls back the offset value. It doesn’t work when I have the api_key in the header but does pull back a value with the api_key in the url.
My problem now is I cannot figure out how to get the api_key=123456 into the URL…confusing the issue is that my offset value has a forward/slash but perhaps that is normal.

Thanks
Paul

Hi! did you find an answer or solution? I have the same trouble, i was googling every technical word to understand stept by stept, I made a mistake in m app variables (“test” in your case) and with your image an explanation i could find my error. am still stuck. Please if you got the solution post it. All no-coders like me will appreciate that.

Thnks.
Mei Li

Hi Mei Li

No, I didn’t find the solution. I moved away from Airtable, there are other backends such as Google Firestore and Backendless where it is relatively easy to get around the pagination issue. You can still use an API for both…

Thanks
Paul

Hi again! I just wanted to tell you that after hours and hours reading word by word all explanations related, i figure it out. In a next comment i will post stept by stept with images explaining what i did, maybe it would save a lot of time to future no-coders or you can try again if you want. I will try to quote everybody because, i found ell the answers in the forum from different community members in different topics, non of these were mine.

See you next comment! :blush: :blush:

OMG please do :sweat_smile:
I’m struggling 2 days no with no success.
Thnx in advance

I’d like to second that. I’ve been struggling with this for the last couple of weeks. I can see the data, get the offset, get a second request, and then things fall apart. I’m most of the way there (I think) but just can’t get the pieces together. Thank you in advance for the guidance!

Hi! again i am struggling to post all the images i did, stept by stept. There are 65 images. Ill be posting 15 images per post

Image 1

Image 2

Image 3

Image 4

Image 5

Image 6

Image 7

Image 8

Image 9

Image 10

Image 11

Image 12

Image 13

Image 14

Image 15

Sorry if did all the images with obvious stepts, I was thinkin that i would really like to have this request method stept by stept.

See you next post.

Mei Li

Hi again no-coders! Here there are next 15 images (16-30).

Image 16

Image 17

Image 18

Image 19

Image 20

Image 21

Image 22

Image 23

Image 24

Image 25

Image 26

Image 27

Image 28

Image 29

Image 30

This tutorial was made specially for no-coders amateurs like me. Hoping you understand if i mention obvious spteps

Mei Li

Hi again! Here you have the next 15 images (31-45). Hoping you are not getting boried with this long explanation :sweat_smile:

Image 31

Image 32

Image 33

Image 34

Image 35

Image 36

Image 37

Image 38

Image 39

Image 40

Image 41

Image 42

Image 43

Image 44

Image 45

See you in the next comment!!!

Mei Li

Hi again :sweat_smile: Now I am figuring out that… it was too much images. Anyway its done. Let me repeat that i am a no-coder, so maybe the language i used it is simply and not properly, aditionally Iam no native english speakers, so you can find some gramatical errors. But if you have any question please post it and quote me, i will try to answer you as soon as i can…:slight_smile:

Image 46

Image 47

Image 48

Image 49

Image 50

Image 51

Image 52

Image 53

Image 54

Image 55

Image 56

Image 57

Image 58

Image 59

Image 60

Image 61

Image 62

Image 63

Image 64

Image 65

Hoping this images help you!!

Finally, all stepts was thanks to:

Paul_Mc posts, Nihal_Parkar posts in this issue Airtable http request (100 item limit / pagination).

Brandon_Winkler posts, Nihal_Parkar posts, and Taj_Bellucci in “Airtable 100 item limit” issue .

vereggen posts, and William_Glass posts, in Setup AppGyver and Firebase REST API.

Wayne_Culbreth posts, Harri_Sarsa AppGyver Team Member posts, Taj_Bellucci posts, and Tomi_LaaksoAppGyver Team Member posts, in Airtable + pagination issue.

I considered people who post “got it” or similar, because this phrases let me know that it would work it.

Thanks every body for posting images, formulas or comments, I would not discover it by myself without this posts. I will continue using appgyver, and if i have new troubles in other things i will post it to get a solution … :blush:

Bye
Mei Li

1 Like

Not all heroes wear capes :smiley:
This is a tremendous help and work from you. Thanks. Will follow the steps and get back with results

1 Like

Hey Mei,
so I followed your tutorial step by step, my data is being pulled (confirmed via dev tools in chrome and debugging) but i have a few problems:

  1. http request is pulling too much data (I have only 100+ just for testing)

  2. named all fields in app vars exactly as in airtable but i cant see nothing when repeated

  3. only HTTP and IF flows are giving green light, the other 2 flows are giving yellow warnings

  4. dev tools shows that the request is working but nothing i showing on screen

  5. page logic is exactly as yours (or maybe im blind and theres something im missing :sweat_smile: )

2 Likes

THANK YOU!!!

I will try this out ASAP…I can already see that I’ve made a bunch of small mistakes in my approach that added up to nothing working. I am really looking forward to tonight when I’ll have time to try this out.

Thanks again…I will report back progress as soon as I have results…

1 Like

And thanks for letting us all know about your results. I thought I was the only one struggling with those json and list warnings…

Another “thanks”…thanks for providing the links to the other pages. I’ve read a couple of these posts but not all of them.

1 Like

A quick update: This is working beautifully! I can finally pull in my entire table. I only ran into two minor snags, which are easily fixed:

  • be sure to add the authorization header to the http request

  • be sure to reset ‘alllistings’ when you navigate away from the page. if you don’t, you’ll pull in duplicate (and triplicate, etc) values ‘alllistings’

I’m going to keep playing with this and will report back other findings. Right now I am trying to figure out how best to pull in the item details once you click on list item. Using HTTP requests like this is new to me.

1 Like

Hi! Edvin_Rushitaj . Dont worry about yellow warnings. It also works ok. As i can see your data is pulling. Following your fields, your trying to pull images, i havent tried yet, because my data doesnt have images, but try with “Image URL” or “URL (any protocol)” value types or maybe “any text(matches all formats)”. I am posting some images to help you in your case.

Please comment back, if you have success or not, the more cases solved we have in the forum the more learning we get for future. Probably not now but in a few days i am going to pull data with images, so i am very interested.

See you
Mei Li

Image 1

Image 2

Image 3