How we scraped 2.2 million Shopify customer reviews

We found that every store built with Shopify that uses a service called Yotpo has a public JSON file available with product-level customer reviews. For example, a product from the store Chella has their file available here.

The problem was that the URL requires both a Store ID and Product ID. The URL appears with the following structure:

https://api-cdn.yotpo.com/v3/storefront/store/{Store ID}/product/{Product ID}/reviews

‍

The Product ID is the same Product ID that Shopify uses to identify the product. This one was easy to find. Every Shopify store has several public JSON files that are available on the same URL route that exposes store, product, variant, and stock information. The product information for the store Chella appears here.

The Store ID gave us more trouble. This is a unique identifier created by Yotpo. After nearly giving up, we went back to our roots to come up with a creative hack. It turns out that every Shopify store that uses Yotpo installs a script in their HTML header that includes the Store ID.

That's when the real fun started.

  1. We bought a list on a site called BuiltWith to find Shopify stores based in the US that also use Yotpo.
  2. We wrote a script to go to the home page of each store, download the HTML, look for a <script> tag with src="//staticw2.yotpo.com/, and grab the Store ID.
  3. We built a crawler in simple Javascript to run through the list of stores. The crawler dynamically places in the Store and Product IDs into the Yotpo URL, finds the customer reviews, and indexes them into our standard data model.

The crawler has been running for 3 days now and we've already added 2.2 million real customer reviews on Agora. From a business perspective, this has already improved our conversion rates as more products have social proof. The next step is to make these reviews at the product-level easily accessible via API.

We're surprised by how much data is publicly available, if you know where to look.