The Adwords spider bot is used to analyse the content of advertising landing pages, which helps determine the Quality Score that Google assigns to your PPC ads.
Google takes this Quality Score into account, as well as the amount of money you are willing to bid, when it determines the position of your ads. Therefore, ads with a high Quality Score can rank higher than competitors’ – even if they are paying more than you.
It clearly makes sense to ensure the Adwords spider bot gives your landing pages as high a score as possible. To help you, I’ve put together some frequently asked questions about how the bot works.
What is the user agent name of the PPC bot that checks relevancy on a landing page?
Adsbot.
Does this bot obey robots.txt?
The AdsBot is not a “search crawler” in the strict sense, so robots.txt is irrelevant (i.e. it doesn’t visit all the links in the landing page). It only crawls the URLs of keywords with an ‘Active’ status.
Does this bot support wildcards e.g /filename*
The wildcards would be in robots.txt and therefore not relevant.
Does this Adbot run JavaScript and drop a cookie?
No.
Is keyword (KW) relevancy on landing page used as a factor that determines Quality Score?
The keywords have to be relevant to the landing page. Relevance refers to the usefulness of information to a user (such as an ad, keyword, or landing page). The quality of this relevance is reflected by a keyword’s Quality Score.
Is KW relevancy on landing page used as a factor in determining KW editorial approval?
The keyword policy states: “Target relevant and specific keywords. Use specific keywords that accurately reflect your products, services, or the site you’re promoting. If you offer a location-specific product or service, you might consider using keywords that reflect your location.”
An example: A New York apartment rental agency would not be allowed to run on only the overly general keyword ‘rentals.’ The agency should instead use keywords such as ‘New York rental agency’ or ‘find NY apartments.’
When matching KW relevancy, which is used: the advert destination URL or the KW destination URLS?
Keyword Relevancy is solely based on Click-through rate (CTR). High CTR = High Relevancy.
When matching KW relevancy, are mis-spellings such as ‘credit carRd’ accounted for?
Since the KW relevancy is based solely on CTR then the mis-spelling will have a high relevancy if it has a high CTR.
When matching KW relevancy does the bidword have to be an [exact] or [standard] match with the keyword on the landing page?
It will have a high relevancy if it has a high CTR. See the point above.
When matching KW relevancy, where does the bot check on the landing page (e.g does it look in title, meta KW, meta description, alt tags, anchor text, or only the text content on the page)?
Again the KW Relevancy is solely based on CTR. High CTR = High Relevancy.
I hope these questions and answers have given you an insight into the ways in which the Adwords spider operates. I’m very happy to take any questions in the comments.