Batch Sync

Batch Sync keeps the Solr search index aligned with what the PIM considers published. Two independent families of endpoints live under /api/b2b/pim/products/batch-sync:

1. Catalog batch sync — a scored cleanup + resync job over the catalog (POST /, history on GET /).
2. Index consolidation — per-channel reconciliation between the published set in MongoDB and the Solr index: scan (preview), reindex (push what is missing), remove-stale (delete what should no longer be there).

Endpoints

What "stale" means

A product is stale when it is present in the Solr index for a channel but is not in that channel's currently published set in MongoDB.

stale = solrCodes(channel) − { entity_code : isCurrent && status === "published" && channel matches }

Anything that leaves the published set produces stale documents: a product unpublished or drafted by a publishing policy, deleted upstream, retagged to a different channel, or superseded by a new current version. Until they are removed those documents keep appearing in search, on category pages, and in facet counts.

Its mirror image is missing: published in MongoDB but absent from the index. missing means the index is behind; stale means the index is ahead. The distinction matters — see the guard protocol below.

Scan — the preview

Computes the gap per channel. This is the preview of remove-stale: it counts exactly the documents that call would delete, using the same query. Always scan first.

curl https://cs.vendereincloud.it/api/b2b/pim/products/batch-sync/scan \
-H "x-auth-method: api-key" \
-H "x-api-key-id: ak_acme_live_1234" \
-H "x-api-secret: sk_live_abcdef..."

{
"success": true,
"solr_available": true,
"scanned_at": "2026-09-08T06:20:11.004Z",
"channels": [
  { "channel": "b2b",        "published": 16800, "indexed": 16812, "missing": 0, "stale": 12, "in_sync": false },
  { "channel": "(untagged)", "published": 0,     "indexed": 69,    "missing": 0, "stale": 69, "in_sync": false }
],
"totals": { "channel": "TOTAL", "published": 16800, "indexed": 16881, "missing": 0, "stale": 81, "in_sync": false }
}

Reindex — push what is missing

Indexes the channel's needs-indexing set, or an explicit list of entity codes. Returns 202 with a job_id; the work runs in the background.

Reindex is additive and safe: it writes documents, never deletes them.

Remove stale — delete what should no longer be indexed

Deletes from Solr every document of the channel that is not in the channel's published set. Returns 202 with a job_id; the deletion runs in the background.

{ "success": true, "job_id": "consol-remove-stale-1757312400000-4k2j1", "status": "running" }

The outcome lands on the BatchSyncLog entry for that job_id — readable through GET /api/b2b/pim/products/batch-sync (activity history) — under cleanup_result:

{ "mode": "remove_stale", "removed_count": 81 }

The guard protocol

remove-stale must be driven by a wrapper that scans first and refuses to fire on an unsafe preview. This is how the operational scripts and the scheduled Windmill jobs drive it, and any new caller should follow the same sequence.

1. GET /scan. If solr_available is false, stop — without an index there is nothing to reconcile against.
2. Require missing === 0. Products published but not indexed mean the index is behind; reconciling in that state is exactly the failure above. Call reindex first, then re-scan.
3. Survival guard. The survivors — indexed − stale — must still cover at least min_survive_ratio of the channel's published count (0.5 in the scripts). The signature of the incident was not "many removals" but an index gutted relative to the database: 306 survivors against ~17,000 published, 1.8%.
4. Stale-share cap. Secondary ceiling on stale / indexed (max_stale_ratio, 0.30). It was raised from 0.15 after a legitimate cleanup — a publishing policy had drafted every product with no image or no price — tripped the guard and left 1,892 dead entries in the index for three days.
5. Apply the ratios only above a floor. Percentages are meaningless on a small index: a channel reporting stale: 69 against indexed: 61 is routine cleanup, not a catastrophe. The scripts skip both ratio guards below stale_floor (250) entries.
6. Re-scan afterwards. A post-run missing > 0 is the signature of an over-broad removal and should be surfaced loudly.

const scan = await GET("/batch-sync/scan");
if (!scan.solr_available) return "abort: no index";

const ch = scan.channels.find(c => c.channel === channel);
if (!ch || ch.stale === 0) return "noop";

if (ch.missing > 0) {
await POST("/batch-sync/reindex", { channel });   // catch up first
return "reindex: re-scan on the next run";
}

const survivors = ch.indexed - ch.stale;
const applyRatios = ch.stale >= STALE_FLOOR;       // 250
if (applyRatios && survivors < MIN_SURVIVE * ch.published) return "abort: index would be gutted";
if (applyRatios && ch.stale / ch.indexed > MAX_STALE) return "abort: stale share too high";

await POST("/batch-sync/remove-stale", { channel });
const after = await GET("/batch-sync/scan");       // post-run check
if (after.channels.find(c => c.channel === channel)?.missing > 0) alert("over-broad removal");

Check a single product

Fast lookup of one product's sync state across MongoDB and Solr — the quickest way to answer "why is this product still showing in search?" or "why is it missing?" before reaching for a channel-wide operation.

Catalog batch sync

The scored cleanup + resync job, independent from the consolidation endpoints above.

Activity history: the BatchSyncLog entries for both the catalog job and the consolidation operations, including cleanup_result.removed_count for every remove-stale run.