- Python
- Web scraping
- Price data
- Apify
How to Scrape Canadian Grocery Prices with Python
No Playwright, no proxy pool, no selector maintenance. The scraping runs on Apify and Python gets JSON, which means the interesting part is what you do with the rows. Five Actors already agree on their field names, so they concatenate.
Savvi7 min read
Start here
pip install apify-client pandas, grab a token from the Apify console, and run this:
import pandas as pd
from apify_client import ApifyClient
client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("sunny_eternity/canada-grocery-price-comparison").call(run_input={
"queries": ["eggs", "milk", "chicken breast"],
"location": "Vancouver, BC",
"retailers": ["loblaws", "saveonfoods", "tnt"],
})
rows = [r for r in client.dataset(run["defaultDatasetId"]).iterate_items()
if r["_type"] == "comparison"]
df = pd.DataFrame(rows)
print(df[["query", "retailer", "matched_product", "price", "comparable_unit_price"]])call() blocks until the run finishes and returns the run object. iterate_items() pages the dataset for you. The _type filter matters even on a default run, since run_meta rides along with the products.
One chain, in depth
The comparison Actor matches a shopping list across chains. When you want one chain’s whole catalogue instead, a retailer Actor is the tool, and the shape of the call is the same.
run = client.actor("sunny_eternity/loblaws-grocery-scraper").call(run_input={
"banner": "superstore",
"categories": ["dairy-eggs"],
"postal_code": "V5K 0A1",
})
products = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(len(products), "products from", products[0]["location_name"])categories: ["all"] walks the entire store, which is where call() stops being the right choice. Start the run, keep the id, come back for the dataset:
run = client.actor("sunny_eternity/loblaws-grocery-scraper").start(run_input={
"banner": "superstore", "categories": ["all"], "postal_code": "V5K 0A1",
})
# Later, in another process. Rows stream in while the run is still going.
info = client.run(run["id"]).get()
print(info["status"])
for item in client.dataset(info["defaultDatasetId"]).iterate_items():
...Five Actors, one frame
Here’s the part that saves a weekend. The retailer Actors were built to share a field set, so rows from four different chains stack without renaming anything.
Common to every retailer Actor
| Field | What it holds |
|---|---|
| store, name, price | Chain, product title, current price in dollars with no symbol |
| unit_price | The retailer’s own unit price string, in whatever basis it publishes |
| comparable_unit_price | Normalized $/100 g or $/100 ml, null when the size will not parse |
| package_size, selling_type | Size string, plus by_weight or by_unit |
| was_price, is_on_sale, multi_buy_deal | Pre-sale price, sale flag, promotion text |
| product_url, product_id, image_url | Links and identifiers |
| location, location_name, location_postal_code | The store the prices came from |
KEEP = ["store", "name", "price", "comparable_unit_price",
"package_size", "was_price", "is_on_sale", "product_url",
"product_id", "location_name"]
JOBS = [
("sunny_eternity/loblaws-grocery-scraper",
{"banner": "superstore", "categories": ["dairy-eggs"], "postal_code": "V5K 0A1"}),
("sunny_eternity/loblaws-grocery-scraper",
{"banner": "nofrills", "categories": ["dairy-eggs"], "postal_code": "V5K 0A1"}),
("sunny_eternity/save-on-foods-pricesmart-scraper",
{"banner": "saveonfoods", "categories": ["dairy-eggs"], "postal_code": "V5K 0A1"}),
("sunny_eternity/tnt-grocery-scraper",
{"categories": ["dairy-eggs"], "postal_code": "V5K 0A1"}),
("sunny_eternity/costco-scraper",
{"search_terms": ["eggs", "milk"], "postal_code": "V5K 0A1", "max_items": 200}),
]
frames = []
for actor_id, run_input in JOBS:
run = client.actor(actor_id).call(run_input=run_input)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
frame = pd.DataFrame(items).reindex(columns=KEEP) # missing field -> NaN
frame["run_date"] = pd.to_datetime(run["startedAt"]).date()
frames.append(frame)
df = pd.concat(frames, ignore_index=True)
print(df.groupby("store")["comparable_unit_price"].describe())reindex(columns=KEEP) is doing the defensive work. An Actor that doesn’t emit one of those fields gives you a NaN column instead of a KeyError, which is what you want when five sources are involved and one of them changes.
For which slugs and banners each one takes, the Loblaws guide and the Save-On-Foods guide have the tables.
Many runs into a series
Every run writes its own dataset and nothing persists between them, so a price series is a matter of listing past runs and reading each one.
ACTOR = "sunny_eternity/canada-grocery-price-comparison"
rows = []
for run in client.actor(ACTOR).runs().list(limit=52, desc=True).items:
if run["status"] != "SUCCEEDED":
continue # keep half-finished scrapes out
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
if item.get("_type") != "comparison":
continue
rows.append({**{k: item.get(k) for k in
("query", "retailer", "price", "comparable_unit_price", "product_id")},
"date": run["startedAt"].date()})
hist = pd.DataFrame(rows)
print(hist.pivot_table(index="date", columns="retailer", values="price", aggfunc="sum"))Filtering on SUCCEEDED is the line people skip and regret. A run that died two retailers in returns a smaller basket, and a smaller basket reads as a price drop.
What bites
- The synchronous endpoint has a timeout.
run-sync-get-dataset-itemsandcall()suit small runs. Whole-store scrapes needstart()plus a later read. _typebefore anything else. Onlycomparisonandcategory_browse_productare product rows. Skip the filter and your means include summary records.- Roll your own paging and you’ll get it wrong.
iterate_items()handles offsets;list_items()with manual arithmetic silently truncates a large dataset. comparable_unit_priceis nullable. Products with no parseable size come backNone. Filling those with zero drags every average you compute afterwards.- Date the rows from the run.
scraped_atis per row, so a long run straddling midnight lands on two dates.startedAtkeeps one run on one date.
For the matching and confidence fields the comparison Actor adds on top of these, see the comparison guide. For the summary and category records a monitoring setup wants, the price monitoring guide covers intel mode.
Frequently asked questions
Do I need Playwright or Selenium for this?
No. The Actors run the browser work on Apify and hand you JSON, so the Python side is an API client and a DataFrame. Nothing renders locally.
What does pip install give me?
apify-client is the only requirement. Add pandas if you want the joins and the pivots, which is what most of this guide does with the rows once they arrive.
Should I use run-sync-get-dataset-items or start a run and poll?
call() blocks until the run finishes and suits anything you would wait on at a terminal. For whole-store scrapes, start it and read the dataset later, since a long run outlives the synchronous endpoint’s timeout.
How do I page through a large dataset?
iterate_items() on the dataset client handles paging for you and yields dicts. Reaching for .list_items() with your own offset arithmetic is the version that goes wrong at 50,000 rows.
Can I put results from different retailers in one DataFrame?
Yes. The retailer Actors already agree on store, name, price, unit_price, comparable_unit_price, package_size, was_price, is_on_sale, product_url and the location fields, so a concat on that subset needs no renaming.
Why is comparable_unit_price None on some rows?
The package size would not parse, usually because the title carries no size at all. Treat those rows as missing rather than filling them with zero, which quietly drags any average you compute.
How do I keep prices from different days apart?
Stamp every row with the run’s startedAt before it goes in the frame. Rows carry scraped_at, but pulling the date off the run keeps a single run on a single date even when it straddles midnight.
What is the _type field for?
It tells you which record you are holding. Only comparison and category_browse_product are product rows; the rest are roll-ups and run metadata. Filter on it first or your DataFrame ends up with summary rows mixed into the products.
Try it on your own data
The Canadian Grocery Price Comparison API runs on Apify. Paste an input, hit Start, and see what comes back before you write any code against it.
Get the Canadian Grocery Price Comparison APIOther retailers
- Loblaws, No Frills & Superstore API12 Loblaw banners with PC Optimum offers and multi-buy deals.
- Save-On-Foods & PriceSmart APIFour Pattison Food Group chains across BC and the prairies.
- T&T Supermarket APICanada’s largest Asian grocery chain, in BC, AB, ON and QC.
- Costco Scraper APIWarehouse prices, deals and item numbers from costco.ca and costco.com.
They all share the same field names. See them side by side on the grocery data API page, with a sample record for each.
Keep reading
- How to Compare Grocery Prices Across 16 Canadian Chains in One CallThis one isn’t a scraper, it’s a matcher. How the confidence scoring works, how to filter on it, and how to get basket totals per store.
- How to Monitor Competitor Grocery Prices in CanadaThe output mode the other guides skip. One run gives you price position per banner, category stats and the twenty deepest discounts, ready to chart.
- How to Scrape Loblaws, Superstore and No Frills PricesTwelve banners share one input and one output shape. Which banner covers your region, the promo fields only this chain exposes, and how to join across banners.