In this tutorial, we'll demonstrate how to extract data from Google Maps. In the first part of the tutorial, we'll use a free tool, built for smaller scale scraping. In the second part, we'll show how to use Oxylabs API for more effective, bigger scale scraping (you can get a free trial here.
To run this tool, you need to have Python 3.11 installed in your system.
Open up a terminal window, navigate to this repository and run this command:
make install
This tool is used to scrape Google Maps results from a given search results page.
First of all, open up Google Maps and enter something to search for in the Search field.
We'll be searching for restaurants in Manhattan, New York.
After pressing enter, copy and save the URL of the page you landed on. We'll be using it in the scraping tool.
In this example, we'll be using this URL: https://www.google.com/maps/search/restaurants/@40.7660158,-73.9833944,14z/data=!4m5!2m4!5m2!1e0!4e9!6e5?entry=ttu
To scrape Google Maps results from your provided URL, run this command in your terminal:
make scrape URL='<url>'
With the URL we retrieved before, the command should look like this:
make scrape URL='https://www.google.com/maps/search/restaurants/@40.7660158,-73.9833944,14z/data=!4m5!2m4!5m2!1e0!4e9!6e5?entry=ttu'
By default, the tool will scrape a small number of results that are visible when you load the page.
To scrape the full results, add FULL=True to your command.
The full command should look like this:
make scrape URL='https://www.google.com/maps/search/restaurants/@40.7660158,-73.9833944,14z/data=!4m5!2m4!5m2!1e0!4e9!6e5?entry=ttu' FULL=True
Be aware that this will take a bit longer, since the tool will automatically scroll to the bottom of the results list.
After running the command, you should see this in your terminal:
When the tool has finished running, you should see a file named locations.csv in the directory you were running the tool.
If you open the generated CSV file, the data should look something like this:
In case the code doesn't work or your project is of bigger scale, please refer to the second part of the tutorial. There, we showcase how to scrape public data with Oxylabs Scraper API.
Google Maps Scraper enables effortless public data extraction with geographic references from Google Maps and Google Places. This short guide will show you the process of scraping Google Maps using Oxylabs' Scraper API.
You can retrieve Google Maps data by providing the URL to our service. Our API will return the results in JSON format.
The below code examples demonstrate how you can get Google Maps results. First, you need to send the instructions to our service using the Push-Pull method:
import requests
from pprint import pprint
# Structure payload.
payload = {
'source': 'google',
'url': 'https://www.google.com/maps/search/restaurants/@40.7660158,-73.9833944,14z/data=!4m5!2m4!5m2!1e0!4e9!6e5?entry=ttu',
'geo_location': 'New York,New York,United States',
'render': 'html'
}
# Get response.
response = requests.request(
'POST',
'https://data.oxylabs.io/v1/queries',
auth=('USERNAME', 'PASSWORD'),
json=payload
)
# This will return a response with job status and results url.
pprint(response.json())Once the job is done, you can retrieve the results by making another request and including the job ID from the previous response, for instance:
import requests
# Get response.
response = requests.request(
'GET',
'http://data.oxylabs.io/v1/queries/{job_id}/results',
auth=('USERNAME', 'PASSWORD')
)
# This will return the JSON response with results.
print(response.json())The above code snippet will retrieve the results in JSON format:
{
"results": [
{
"content": "<!doctype html>\n<html lang=\"en\">\n<head>...</script></body>\n</html>\n",
"created_at": "2023-07-25 10:01:01",
"job_id": "7089545068712824833",
"page": 1,
"status_code": 200,
"updated_at": "2023-07-25 10:01:20",
"url": "https://www.google.com/maps/search/restaurants/@40.7660158,-73.9833944,14z/data=!4m5!2m4!5m2!1e0!4e9!6e5?entry=ttu"
}
]
}From local landmarks to various businesses, with Oxylabs’ Google Maps Scraper you’ll easily access the public data you need. If you have any questions or need assistance, don’t hesitate to contact our 24/7 support team via live chat or email.
Read More Google Scraping Related Repositories: Google Sheets for Basic Web Scraping, How to Scrape Google Shopping Results, Google Play Scraper, How To Scrape Google Jobs, Google News Scrpaer, How to Scrape Google Scholar, How to Scrape Google Flights with Python, How To Scrape Google Images, Scrape Google Search Results, Scrape Google Trends
Also, check this tutorial on pypi.