IRobotSoft Visual Web Scraping & Web Automation Forum
http://irobotsoft.org/bb/YaBB.pl
General Category >> Newcomers >> keep google maps from loading
http://irobotsoft.org/bb/YaBB.pl?num=1468962582

Message started by nabeshin on 07/19/16 at 17:09:42

Title: keep google maps from loading
Post by nabeshin on 07/19/16 at 17:09:42

I'm trying to scrape a company's location pages.  Each one has a google map that tries to load.  It works fine in a regular browser but irobot gets hung up on it and can't even load the next link in the list.  Anyway to have the program 'block' anything from google as if it were an advertisement?

I had attempted a search but came up empty.

Title: Re: keep google maps from loading
Post by IRobotSoft Administrator on 07/19/16 at 22:35:07

Have you tried to change the IE version from menu: Advanced -> IE Version -> IE 11?


Title: Re: keep google maps from loading
Post by nabeshin on 07/20/16 at 17:00:20

I had figured out it got hung up on google based on what was loading in the bottom left corner.  Now with IE 11 version it gets hung up on some kind of Facebook App.

This is the site im trying to scrape.

http://cornerstore.com/StoreHome/1

I have given the bot a list of URLs similar to this but going up to number 2050, but only in chunks of 500.

Title: Re: keep google maps from loading
Post by nabeshin on 07/20/16 at 17:09:05

Just figured out it is saving all of the data on the first page over and over, once for each URL in the list, but it isn't actually going to the next URL.

Title: Re: keep google maps from loading
Post by nabeshin on 07/20/16 at 17:11:41

not to post three times in a row, but i have figured it out, and will document it here for posterity.

I had the action as "get list of links" when I gave it the list, but it should have been "go to URL" and then been given the list.  It seems to be working now.

IRobotSoft Visual Web Scraping & Web Automation Forum » Powered by YaBB 2.1!
YaBB © 2000-2005. All Rights Reserved.