Scrape data from AdWords with Ruby Mechanize

Why would you want to scrape data from AdWords when Google provide an API and email reports? Well, managing a number of different AdWords accounts for various clients can be challenging. It can be very time consuming logging into individual accounts and checking stats on a daily basis. The AdWords API is good, but not always suitable for certain client accounts. Email reports are nice, but you still have to collate the emails from your inbox. Often the best way to download specific stats into one file is to build a web page scraper. Using a scrape script provides you with a lot more flexibility, and allows you to focus on exactly the sort of information you think is important.

Recently, we discovered a neat little Ruby library called Mechanize. Mechanize does an excellent job of navigating forms, pages and links and extracting data.

Here is an example AdWords spend scrape script to get you started…

require 'rubygems'
require 'mechanize'

adwords_accounts = [
	["adwords1@skyrocketonlinemarketing.com", "password"]
]

total_accounts = adwords_accounts.length

(0..total_accounts-1).each do |i|
	puts "#{adwords_accounts[i][0]}"
	agent = Mechanize.new
	agent.get("https://adwords.google.com")
	puts "#{adwords_accounts[i][0]}: #{agent.page.title}"
	sign_in_form = agent.page.forms[1]
	sign_in_form.Email = adwords_accounts[i][0]
	sign_in_form.Passwd = adwords_accounts[i][1]
	page = agent.submit(sign_in_form, sign_in_form.buttons.first)
	puts "#{adwords_accounts[i][0]}: #{agent.page.title.gsub(/\n/,' ')}"
	agent.get("https://adwords.google.com/select/snapshot")
	puts "#{adwords_accounts[i][0]}: #{agent.page.title.gsub(/\n/,' ')}"
	if agent.page.title == "Redirecting"
		agent.get("https://adwords.google.com/select/snapshot")
		puts "#{adwords_accounts[i][0]}: #{agent.page.title.gsub(/\n/,' ')}"
	end
	stats_title = agent.page.at(".stats-title").text
	stats_clicks = agent.page.at("tr:nth-child(1) .stats-data").text
	stats_imps = agent.page.at("tr:nth-child(2) .stats-data").text
	stats_cost = agent.page.at("tr:nth-child(6) .stats-data").text
	puts "#{adwords_accounts[i][0]}: #{stats_title} #{stats_imps} #{stats_clicks} #{stats_cost}"
	puts "\n"
end

To get this to work, all you need to do is enter your account information on line 5. If you then want to use this for more accounts, simply add them to the adwords_accounts array.

Related posts:

1. Prepare AdWords campaigns in Excel for upload into AdWords Editor
2. AdWords Tuneup!
3. AdWords campaigns on the iPhone
4. AdWords seminars
5. AdWords Quality Score

Tagged as: adwords, mechanize, nokogiri, ruby

Sponsored links: