Open the Shopify App Store listing for Klaviyo: Email Marketing & SMS and you see a rating of 4.7 across 3,268 reviews. Open the star breakdown on the same page and count it yourself: 2,857 five-star, 84 four-star, 31 three-star, 37 two-star, 259 one-star. Work out the mean and you get 4.60, not 4.7.

Nothing is wrong with either number. They measure different things, and Shopify says so in its own documentation. But almost every "best Shopify apps" list treats the headline rating as a straight average of merchant opinion, ranks apps by it, and stops there — at exactly the moment Shopify is actively adding and removing reviews from those totals with, in its own words, no end date.

What changed on July 6, 2026

Shopify published two announcements the same day.

The first introduced a dedicated App Store requirement, policy 1.3, "Always use honest and transparent review practices." The rules against incentivized reviews already existed inside the Partner Program Agreement, but as Shopify put it, that "made them easy to overlook." The new requirement spells out the consequence: "When an app is found to be incentivizing reviews, we'll remove a significant portion of its reviews." Policy 1.3.1 bans offering incentives or withholding features in exchange for a review, "regardless of whether this occurs through the app or external channels."

The second announcement covered untrusted review detection. Shopify expanded the signals it uses to judge authenticity and — critically — applied them retroactively: "Because we're applying these signals to existing reviews as well as new ones, a meaningful number of reviews that don't meet our authenticity bar will be unpublished over the coming weeks." Shopify warned that apps doing nothing wrong would also lose reviews, because untrusted reviewers spread activity across many listings.

Then it partly reversed, and told everyone this never stops

On September 3, 2026, Shopify posted a follow-up in the developer forum: it had "adjusted the signals we use to assess authenticity," and "some reviews may be restored, while others may be unpublished."

The following day, asked directly how long the sweep would run and whether an app that got reviews back was finished being assessed, the Shopify staff member answered:

There is no end date, reviews are being assessed continuously from when they are submitted and to whenever the information behind them changes. Reviews that are visible today may be hidden over time, or vice versa.

For a merchant building a shortlist, that single sentence changes how the page should be read. An app's review count is not a cumulative record of everyone who ever reviewed it. It is a live figure that can move in both directions without anyone editing a review — and Shopify explicitly says it "may not notify" developers when reviews are removed.

The rating was never an average

This is documented, and it is the part most comparison articles get wrong. From Shopify's developer documentation:

The overall rating isn't simply an average of star ratings; it's weighted to prioritize recent, useful, and trustworthy reviews.

Unpublished reviews "aren't visible on the Shopify App Store, and they don't count toward your app's review total or its overall rating." The same weighting logic explains why the displayed rating can sit above or below the arithmetic of the published distribution.

Four listings, checked September 5, 2026:

App	Displayed rating	Reviews	5★	4★	3★	2★	1★	Mean of the published distribution
Judge.me Product Reviews App	5.0	46,389	45,355	775	68	36	155	4.96
Loox ‑ Product Reviews App	4.9	9,516	8,953	328	44	28	163	4.88
PageFly ✦ AI Page Builder	4.9	5,892	5,573	202	27	21	69	4.90
Klaviyo: Email Marketing & SMS	4.7	3,268	2,857	84	31	37	259	4.60

Three of the four round cleanly. Klaviyo's does not: the weighting lifts a 4.60 distribution to a displayed 4.7. Judge.me runs the other way in a subtler sense — a listing showing a flat 5.0 also carries 155 one-star reviews, which the headline number cannot express.

Source: Shopify App Store (Judge.me Product Reviews App), retrieved September 5, 2026

Source: Shopify App Store (Klaviyo: Email Marketing & SMS), retrieved September 5, 2026

Klaviyo's shape is worth naming, because it recurs across large integration apps: more one-star reviews than four-star and three-star combined. That is not a mediocre app. It is a heavily used app where a specific subset of merchants hits a specific wall — and the distribution tells you that far better than 4.7 does.

Who is allowed to review, and for how long

Two rules from Shopify's documentation set the boundaries of the whole dataset:

• Only merchants who currently have the app installed can leave a review, plus a window of 45 days after uninstalling.
• Reviews are not published on submission. There is a delay while Shopify checks them, and "some reviews aren't published right away, but may go live later as trust in the review increases."

The 45-day window matters more than it looks. A merchant who installed an app, fought with it for six months, ripped it out, and then wrote up the experience two months later cannot post that review at all. The population you are reading skews toward people still using the product.

This is not the first sweep, either. On May 15, 2025, Shopify archived "a significant number of outdated, unhelpful, and untrusted reviews," noting that archived reviews "are not factored into app ratings or total review counts" and that archived review decisions cannot be appealed. The visible history on a listing reflects that. Loox shows 9,516 reviews across 952 pages, but page 952 — the oldest visible — stops at October 2023, even though the same listing carries a review from a merchant with "Over 5 years using the app."

The line the star rating hides

Every review on an App Store listing carries three pieces of context that no aggregate captures: the store name, the country, and how long that merchant had been using the app.

That last one does the most work. Filter Loox to five-star reviews sorted newest, and on September 4, 2026 a review from a UK store with "28 minutes using the app" sits four entries above one from a US store with "About 1 year using the app." Both count the same toward 4.9. On the oldest visible page of the same listing are two entries from "Ma boutique" — the Shopify default store name, never changed — with "4 minutes using the app" and "8 minutes using the app," and no review text at all.

Shopify has been asked whether install-to-review time factors into its trust scoring. On September 4, 2026, its answer was: "We can't share the specific signals that triggered an action on individual reviews." So treat the usage line as your own filter, not one Shopify has applied for you.

It cuts both ways. Judge.me's most recent one-star, dated September 4, 2026, comes from a US store with "Over 3 years using the app." That is a considered complaint from a long-term paying user, and it is worth more of your attention than a hundred five-stars written in the first hour.

Two URL parameters do most of the work

The reviews page has a Refine panel with per-star checkboxes and a Sort control offering only two options: Most relevant and Date: Newest to oldest. There is no "lowest rating first." But the filters are in the query string, so you can go straight there:

https://apps.shopify.com/<app-slug>/reviews?locale=en&ratings%5B%5D=1&sort_by=newest

That combination — one star, newest first — is the single most useful view of any app listing, and it takes ten seconds. Run it on Loox, rated 4.9, and the top of the list is a merchant with "Over 5 years using the app" disputing a $399.99 charge on a $49.99/month plan, because they could not tell what the app counted as an "order" when most of their volume arrives from marketplace integrations.

Source: Shopify App Store (Loox ‑ Product Reviews App), retrieved September 5, 2026

Nothing about that is a scandal — the developer replied within a day, publicly. But it is exactly the kind of billing-denominator surprise that decides whether an app works for your store, and it is invisible at 4.9. There is also a language filter with 23 options, and it is not decorative: two of Judge.me's four most recent one-star reviews on September 5, 2026 were written in Portuguese and Spanish. An English-only skim misses them.

Why Shopify's own systems still count these numbers

Reviews are not just a display element. They feed placement.

Under Built for Shopify's proven usefulness standard, Shopify says it "automatically gauge[s] usefulness based on how many merchants installed your app, the number of reviews, and your app's average rating," on a rolling window. The published thresholds are concrete: a minimum of 50 net installs from active shops on paid plans, a minimum of five reviews, and a minimum recent app rating — recent, not lifetime. The same three criteria gate eligibility for App Store story pages and for the increased-visibility achievement that grants a search ranking boost and placement in the admin's Picked for you modal.

So a sweep that unpublishes reviews can move an app's badge, its ranking, and its position in Shopify's own recommendations. Which means the apps surfaced to you in the App Store in September 2026 are partly a product of a detection system that Shopify says is still being tuned.

A reading order that survives the next sweep

1. Read the distribution, not the rating. Count the one-star bucket against the four- and three-star buckets.
2. Open ratings[]=1&sort_by=newest. Read the ten newest one-stars before anything else.
3. Check the usage line on every review you weight. Under a week is a setup opinion. Over a year is a verdict.
4. Read the developer replies, and their dates. Shopify's guidance to developers is to reply to negative reviews rather than bury them; silence on a months-old billing complaint is a signal.
5. Screenshot the count and rating on the day you decide. Both are mutable, and you will want to know whether a later change reflects the app or the sweep.
6. Treat Built for Shopify as an engineering check, not a satisfaction score. It covers clean uninstall, admin performance, embedding, and App Store compliance. Klaviyo does not carry the badge; Judge.me and Loox do. That is not a ranking of which serves your store better.

What none of this fixes

Developers in Shopify's forum have made the point repeatedly since July: removing fraudulent reviews months later does not undo the ranking, installs, and genuine reviews an app accumulated while they were live. Shopify's response on September 4, 2026 was that apps violating review policies "can face delisting or account termination, not just review removal," evaluated case by case — but it declined to detail the appeal process, the safeguards against false positives, or the signals in use.

For a merchant, the practical conclusion is narrower and more useful than the argument. The star rating on a Shopify App Store listing is a weighted, recency-biased, continuously recomputed number derived from a population that excludes anyone who left more than 45 days ago. The distribution, the usage duration on each review, and the newest one-star reviews are all published on the same page, cost nothing to read, and are far harder to game.

Sources

• Strengthening trust in App Store reviews — Shopify developer changelog, July 6, 2026
• Updated App Store Requirements: 1.3 Always Use Honest and Transparent Review Practices — Shopify developer changelog, July 6, 2026
• An update on App Store reviews — Shopify Developer Community Forums, September 3–4, 2026
• Manage app reviews in the Shopify App Store — shopify.dev
• Shopify review policies — Shopify Help Center
• Shopify App Store requirements — shopify.dev
• About Built for Shopify — shopify.dev
• Built for Shopify achievement criteria — shopify.dev
• Archiving outdated, unhelpful, and untrusted reviews on the Shopify App Store — Shopify developer changelog, May 15, 2025