Troubleshooting the OpenAI Ads crawler's access to a Shopify store

Other
, , , , ,
1

The client's problem

OpenAI Ads x Shopify Setup
We have set up our first OpenAI ad whic connects to our Shopify store.
After serving for a couple days we got this error:

Not approved
Ad has not passed policy
Landing page issue: the site blocks crawler access in robots.txt.
What you can do:

  • Make sure the landing page is publicly accessible
  • Edit and resubmit the ad
2

My guidance

1

Ensure that OAI-AdsBot is not blocked in robots.txt (generated from robots.txt.liquid).

2

An important special case of §1 occurs when robots.txt.liquid is correct, but the URLs you specified in OpenAI Ads Manager Beta are incorrect.
In particular, Shopify's default robots.txt.liquid outputs to the generated robots.txt the contents of the robots.default_groups variable, which contains reasonable default blocking rules:

Disallow: /collections/*+*
Disallow: /search

In this special case, it is necessary to correct the URLs specified in OpenAI Ads Manager Beta, rather than correct robots.txt.liquid.

3

3.1

If §§1-2 do not resolve the issue, then it is necessary to analyze the settings of your store's Web Application Firewall (WAF, most commonly provided by Cloudflare).
§2.3.1.4 of the RFC 9309 standard («Robots Exclusion Protocol») states:

If the robots.txt file is unreachable due to server or network errors, this means the robots.txt file is undefined and the crawler MUST assume complete disallow.
For example, in the context of HTTP, server errors are identified by status codes in the 500-599 range.

Therefore, if the WAF unexpectedly returns an HTTP status code in the 500-599 range when OAI-AdsBot requests robots.txt, then OAI-AdsBot will interpret this response as a complete disallow, resulting in the diagnostic message you received: «the site blocks crawler access in robots.txt».

3.2

If, however, the WAF blocks OAI-AdsBot with the HTTP code 403 («Forbidden»), which is correct for such blocking, or with the less correct but acceptable 401 («Unauthorized»), then, according to §2.3.1.3 of RFC 9309, «the crawler MAY access any resources on the server», and therefore a crawler compliant with the standard should not interpret the code 401 or 403 as «the site blocks crawler access in robots.txt».
However, it is unknown to what extent OAI-AdsBot complies with RFC 9309; therefore, §3.2 is also worth considering.