• You MUST read the Babiato Rules before making your first post otherwise you may get permanent warning points or a permanent Ban.

    Our resources on Babiato Forum are CLEAN and SAFE. So you can use them for development and testing purposes. If your are on Windows and have an antivirus that alerts you about a possible infection: Know it's a false positive because all scripts are double checked by our experts. We advise you to add Babiato to trusted sites/sources or disable your antivirus momentarily while downloading a resource. "Enjoy your presence on Babiato"

How to scrape multiple pages using the https://webscraper.io/ extension

MinMary

Active member
Trusted Uploader
Sep 22, 2019
153
69
28
I am having some challenges using the web scraper to get some data from upwork.com. The scraping seems to work but I can't get it to scrape more pages automatically. It seems to just scrape the first page and stops. How can I solve this? Here's my sitemap.


{"_id":"jobs","startUrl":["https://www.upwork.com/search/jobs/"],"selectors":[{"id":"Project links","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.job-title-link","multiple":true,"delay":0},{"id":"project title","type":"SelectorText","parentSelectors":["_root"],"selector":"a.job-title-link","multiple":true,"regex":"","delay":0},{"id":"sing - project title","type":"SelectorText","parentSelectors":["Project links"],"selector":"header h2","multiple":false,"regex":"","delay":0},{"id":"sing-project desc","type":"SelectorText","parentSelectors":["Project links"],"selector":".job-description div","multiple":false,"regex":"","delay":0},{"id":"budget","type":"SelectorText","parentSelectors":["Project links"],"selector":".job-features li:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"skills","type":"SelectorText","parentSelectors":["Project links"],"selector":"section:nth-of-type(5)","multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":".pagination li.ng-scope:nth-of-type(n+3) a","multiple":false,"delay":0}]}
 
AdBlock Detected

We get it, advertisements are annoying!

However in order to keep our huge array of resources free of charge we need to generate income from ads so to use the site you will need to turn off your adblocker.

If you'd like to have an ad free experience you can become a Babiato Lover by donating as little as $5 per month. Click on the Donate menu tab for more info.

I've Disabled AdBlock