1. Why a script works on day one and goes quiet by week three

Drop a tutorial script into crontab and the first three days of reports look fine. Trouble surfaces somewhere between week three and week five, and none of it arrives as an error.

1.1 Four failures that raise no alarm

All four share one property: monitoring stays silent. Status codes, exception counts, and exit codes are healthy while the numbers in the report drift. Expecting alerts to catch that is asking monitoring to guess business meaning.

1.2 Tutorials stop at getting data. You need data that lines up

A standard tutorial reads: install dependencies, send a request, parse four fields, write a CSV, add time.sleep(random.uniform(1, 3)), done. That path answers whether data is obtainable.

Production asks a different set. Does this batch cover the same products as yesterday? Do fields for one product in two marketplaces mean the same thing? If this run fails, will a rerun produce two different prices? When the invoice lands, can you say how many usable records each dollar bought? None of those answers live in the request. They live in the structure around it.

There is a second cost, and it is larger: rework. A broken script on a laptop costs thirty seconds to fix and rerun. Duplicate rows sitting in pandas for three weeks cost every downstream report built on them, and you may not know which ones those are. Hours saved on structure come back with interest at the first data incident.

The split below follows that boundary. The middle four sections cover the anti-bot gates, which decide whether your traffic counts as traffic. The eight after that cover the engineering, which decide whether the data lines up day after day.

2. Five Python routes, and how much work each leaves you

Choosing a route is not choosing an API. It is choosing how much code your team still owns. For the same job, the spread across five routes runs past an order of magnitude.

One test settles it: which layer of failure do you intend to own? The hidden cost of self-hosting is not the server bill. It is the two or three days of triage after Amazon adjusts its defenses. Write down who gets paged at 3 a.m. before you compare feature tables. Authorization boundaries, cost structure, and fit for each of the five routes are set out in the Amazon data API overview.

3. Gate one: TLS and HTTP/2 fingerprints

Teams tend to assume they are negotiating at the HTTP layer. The handshake finishes before your first header goes out, and it decides whether the page you receive is a page.

3.1 The handshake happens before your request

On an HTTPS connection the client opens with a ClientHello carrying TLS version, cipher suite list, extensions and their order, elliptic curves, point formats, and signature algorithms. In a real browser that set is fixed by browser version and operating system. In Python it is fixed by the ssl module and how OpenSSL was compiled. The server needs no URL to conclude that this is a Python client.

Hence the counterintuitive outcome: a current Chrome User-Agent, a residential proxy, and three requests per minute per IP, and the response is still a challenge page. The User-Agent is a string you wrote at the HTTP layer. The handshake fingerprint you never touched.

3.2 JA3 and JA4: from order-as-identity to sorted hashing

JA3 concatenates five ClientHello fields, TLS version, cipher suites, extensions, elliptic curves, and point formats, in the order they appear, then hashes them with MD5. One software stack yields one stable result, so the hash becomes a client identity. Vendors keep a library of known browser fingerprints and treat anything outside it as suspect.

JA4, from FoxIO, exists because Chrome 110 began permuting extension order. The same Chrome build now produces many JA3 values, so detection by exact JA3 match broke. JA4 sorts extensions before hashing, adds a readable prefix, and folds in QUIC, ALPN, and signature algorithms. Mainstream defenses moved to order-stable fingerprints years ago, which is why pinning a JA3 value no longer buys much.

3.3 Why the default Python client has no browser counterpart

Published comparisons put a default requests client at single to low double digits on targets that fingerprint, httpx with HTTP/2 a little higher, and a client whose handshake matches a browser in the 60–90% band, same IPs and same rate, with the TLS stack as the only variable. Such numbers vary by target, page type, and IP quality, so trust the order of magnitude and not the decimals.

The takeaway: a fingerprint is admission, not advantage. It sits first in the evaluation because it is the one variable that, when wrong, makes everything after it moot.

3.4 Swapping the handshake: curl_cffi and three cautions

curl_cffi is a Python binding over curl-impersonate. It ships browser TLS and HTTP/2 stacks inside the wheel and switches profiles with one argument.

from curl_cffi import requests

resp = requests.get(
    "https://www.amazon.com/dp/B08N5WRWNW",
    impersonate="chrome",                      # latest stable profile
    headers={"Accept-Language": "en-US,en;q=0.9"},
    proxies={"https": "http://user:[email protected]:8000"},
    timeout=25,
)

Three traps:

• Do not override the User-Agent.impersonate="chrome" brings a matching UA, the Sec-Fetch-* family, and header order. Hand-writing a newer Chrome UA undoes the consistency the library just restored.
• Pin the profile version or let it float, but pick one per service.impersonate="chrome" tracks library upgrades; chrome124 is reproducible and goes stale. Do not mix profiles inside one Session, since cookies and fingerprints should share a lifetime.
• Match Accept-Language to the storefront.de-DE,de;q=0.9 for amazon.de, ja-JP,ja;q=0.9 for amazon.co.jp. A language that contradicts the site is the cheapest signal you can hand over.

3.5 The second fingerprint: HTTP/2

Passing the handshake reveals another layer. After the connection preface, the client sends a SETTINGS frame carrying HEADER_TABLE_SIZE, ENABLE_PUSH, INITIAL_WINDOW_SIZE, and MAX_CONCURRENT_STREAMS, plus a WINDOW_UPDATE increment, priority frames, and pseudo-header order. Chrome, Firefox, and Safari each differ, and Go’s standard library or Python’s h2 implementation differ from all three. A client whose TLS says Chrome and whose HTTP/2 says Go reads as a contradiction, and contradictions are what these systems hunt. That is why assembling httpx[http2] on your own tends to fall short.

If the target also runs JavaScript challenges that collect navigator, screen, and WebGL telemetry, a plain HTTP client has no answer at that layer. Either run a real browser engine or hand the layer to a service that already handles it.

3.6 How to verify the change took hold

Do not treat a successful response as proof. Call a public fingerprint endpoint once and compare the returned ja3n and ja4 against your target browser version, then fetch a real page and confirm the body contains productTitle. Both passing is the bar.

from curl_cffi import requests

def fingerprint_report(profile: str) -> dict:
    r = requests.get("https://tls.browserleaks.com/json", impersonate=profile, timeout=20)
    d = r.json()
    return {"ja3n": d.get("ja3n_hash"), "ja4": d.get("ja4"), "ua": d.get("user_agent")}

print(fingerprint_report("chrome"))
print(fingerprint_report("safari"))   # the two profiles should differ

One caveat: curl_cffi support on Windows is incomplete because HTTP/3 dependencies fail to build there. Linux and macOS are fine. If you run in containers, verify wheel compatibility against your base image before rollout.

3.7 A clean fingerprint does not mean clean data

Fingerprints decide whether you look automated. They say nothing about whether the payload is what you asked for. Defenses have a quieter option than blocking: serve degraded data. Prices lag a day, sponsored slots vanish from results, availability sticks to in stock. Status 200, every field present, parsing clean, and it only shows up against a human spot check. That is why section 9 exists after all this work.

4. Gate two: browser fingerprints beyond TLS

Once Playwright or Puppeteer enters the design, the attack surface grows from the protocol to the whole runtime. Headless browsers leak in systematic ways, and most leaks sit open under default settings.

4.1 Seven tells of a headless browser

4.2 Consistency beats sophistication

The common mistake is optimizing field by field: newest User-Agent, an enthusiast GPU renderer string, a 4K screen. Each looks reasonable; together they describe a MacBook with an RTX 4090, a combination whose prior probability is near zero. Detection hunts contradiction, not outdated hardware.

In practice, keep three to five device profiles and switch all five fields as a unit: User-Agent and platform, WebGL renderer, screen and pixel ratio, timezone and language, Accept-Language. Volume does not help. Internal agreement does.

4.3 Which Amazon data needs JavaScript

This decides whether you carry a browser cluster, and the cost difference is an order of magnitude. A working split:

• Server-rendered and available in plain HTTP: title, brand, main and sub BSR category, rating and review count, bullet points, and most variant attributes.
• Needs rendering or interaction: live Buy Box price and seller rotation, some promotion badges, sponsored markers and ad slots on search pages, lazy-loaded review blocks and the Customer says summary, and delivery estimates after a postal code is set.

If your needs sit in the first group, curl_cffi covers them and a browser fleet is dead weight. If they sit in the second, search-page ad slots and postal-code pricing above all, the browser cost is unavoidable, and that is the point where many teams move to a data API instead.

5. Gate three: when residential IPs stop being optional

Once the fingerprint is right, the IP is next. No trial and error needed here; the ASN tells you the outcome in advance.

5.1 Three IP types, three trust sources

Price bands come from published rate cards across proxy providers in 2026 and vary by plan and region, so read them as magnitude. The substance: a datacenter IP is not a weaker version of the same option, it is classified before evaluation begins, so a perfect fingerprint never reaches the same scoring path.

5.2 Difficulty by page type

Treating Amazon scraping as one workload is the biggest misread in this field. Protection varies by surface, and one IP and fingerprint setup can differ by tens of percentage points between a product page and a seller page.

Public benchmarks put datacenter IPs on protected surfaces somewhere in the 10–40% range, residential between 50% and 95% depending on IP hygiene and sharing, and mobile above 85% in most cases. The bands are wide because the variables are many, but the order is stable: failed requests cost money, and in most billing models they cost the same as successful ones.

5.3 Five signals your datacenter IPs are done

Do not wait for the success rate to collapse. Two of these five justify a move:

• Challenge pages above 5%, with no improvement after changing fingerprint profiles;
• Success rate decaying by hour, 90% at dawn and 60% by afternoon, back to 90% the next day, which is pool exhaustion;
• The same request works from office network in a browser and fails from a server;
• Sticky sessions dropping before page three;
• Geo drift in the payload, such as a currency that does not match the storefront you asked for.

5.4 The hidden costs of residential IPs

Traffic is not the whole bill. Four items go missing from most estimates:

• Failed requests still bill.Under per-GB pricing, a challenge page is traffic. A 40% success rate means 2.5× the bandwidth.
• IP hygiene.Cheap residential pools carry addresses already worn out by other jobs. At the same rate card, pool quality can double or halve your success rate.
• Sticky session premiums.Holding one IP for a window, needed for paging or logged-in flows, costs extra credits or a higher tier.
• Operations.Pools decay. Someone has to watch the success distribution, retire bad exits, and retune per-IP rates. This is headcount, and it is the line most often left out. For a method that compares vendors on quoted price against real invoices, see the Amazon data API comparison.

5.5 Residential or mobile: when the premium pays off

Mobile exits carry more trust because carrier CGNAT puts hundreds of real subscribers behind one address, so blocking it is expensive for the defender. Mobile is also pricier and adds latency, which rules it out as a default.

Layer it instead: residential for product and category pages, mobile reserved for two jobs, the heaviest surfaces such as seller and storefront pages, and an emergency lane when residential success drops below threshold. The deciding number is not unit price but the cost of a failed run: if a failure cannot be repaired until the next day, multiply that delay by the failure rate and it exceeds the mobile premium in most cases.

One more: do not judge on the blended success rate. Break it down by page type, and one endpoint is, in most cases, dragging the average, needing only a small share of traffic on a higher-grade exit.

5.6 When datacenter IPs are enough

Three cases do not need residential exits: the target is an unprotected API you own or are authorized to call; the site carries light defenses, such as a small Shopify store; or the exit layer belongs to a data service you call. The third case is section 16.

6. Gate four: obfuscation, cadence, sessions, and provenance

Fingerprint and IP decide who you are. Cadence and behavior decide whether you look like a shopper. This layer needs no cleverness, only the discipline to stop being tidy.

6.1 Jitter is not random.uniform

A fixed two-second interval and a uniform draw between one and three seconds look just as scripted in a time series: both have visible bounds. Human intervals are long-tailed, short for the most part with occasional multi-minute gaps, and they arrive in clusters, five pages then a pause.

import random, asyncio

async def human_gap(base: float = 1.2) -> float:
    """Long-tailed gap: most waits 0.6-1.8s, occasional 5-20s pause."""
    if random.random() < 0.08:              # 8% chance of a long pause
        return random.uniform(5.0, 20.0)
    return random.expovariate(1.0 / base)   # exponential, no upper bound

async def paced_worker(work, limit_per_min: int = 20):
    """Hard per-IP cap per minute; wait when exceeded."""
    gap = 60.0 / limit_per_min
    for item in work:
        await asyncio.sleep(await human_gap(gap))
        yield item

Three parameters matter more than the interval itself: requests per IP per minute, single digits on search pages and looser on product pages; consecutive requests per session, switch exits after twenty to thirty; and the schedule itself, since starting every job on the hour is a pattern.

6.2 Rotate per request or hold a sticky session

Let session state decide. Product pages are stateless, so per-request rotation saves the pool and lowers risk. Search paging, repeated queries under one postal code, and logged-in flows need stickiness; rotating mid-flow costs the session and re-triggers challenges. A middle path works well: rotate per session rather than per request, hold one exit inside a session, replace it when the session ends.

6.3 Referer chains and provenance

Landing on a product page by deep link, with no referer, no search history, and no cookies, is a minority path. Real visits arrive from search, category pages, or off-site ads. The cheap fix is two steps: request the search or category page first, then enter the detail page with that URL as referer. One change covers referer and cookie initialization together.

6.4 Cookie lifetime

Reuse a cookie jar through a Session instead of opening a fresh connection per call, and stop short of keeping one jar forever. Amazon rotates session tokens, so an unchanging cookie becomes a signal of its own. Give sessions a lifetime, fifteen to thirty minutes or twenty to fifty requests, whichever comes first, then discard and rebuild.

6.5 Geographic consistency: a German IP asking for en-US

Exit country, storefront domain, Accept-Language, timezone, currency, and postal code must come from one place. For the German storefront that means a Frankfurt exit, amazon.de, de-DE, Europe/Berlin, and EUR; for postal-code pricing, the code must fall inside the exit country. Encode this as an assertion rather than a convention people are expected to remember.

6.6 Boundaries: compliance and rate

Amazon’s terms restrict automated access, and case law in several jurisdictions supports collecting public data. The two do not cancel out. Three rules hold in practice: take only what is public and reachable without login; stay away from personal data and copyrighted content in full; keep rates low enough to impose no burden. A legal review before a commercial launch costs less than remediation after one.

6.7 Pre-launch checklist

Ordered by failure probability times triage cost, each verifiable in ten minutes. Run through it before production and most night pages disappear.

1. Fingerprint self-test: call a public fingerprint endpoint and confirm ja3n and ja4 match your target browser version, rather than treating a successful response as proof.
2. Challenge detection: put the marker strings into a function so a 200 response can still be judged a failure.
3. Six fields, one source: exit country, domain, Accept-Language, timezone, currency, postal code, as assertions.
4. Timeouts as a tuple: connect and read timeouts set apart, so half-open connections do not fill your workers.
5. Retry budget: retry transient failures and rate limits only; contract failures stop the batch; exhausted budget stops the job.
6. Backoff with jitter: add randomness, or a batch of failures retries in the same second.
7. Rate limits in triplicate: concurrency, requests per second, and requests per IP per minute.
8. Date in the primary key: asin + marketplace + captured_at + contract_version, and prove a rerun adds no rows.
9. Cost in the logs: persist attempts, status, marketplace, and contract_version.
10. A repair path: decide how a failed day gets fixed, same-day rerun or tomorrow’s window. Without an answer, do not scale.

7. Request layer: one class for timeouts, credentials, retries, fingerprint

With all four gates behind you, structure comes next. The first move is to pull scattered request parameters into one object so that one call has one definition.

Three rules: set timeouts, with connect and read as separate values; take credentials from the environment rather than source; retry inside the class and return the attempt count. The class below fits both a self-built scraper and a data API, with the difference sitting in transport.

# pip install curl_cffi
import os, random, time, logging
from dataclasses import dataclass
from curl_cffi import requests as creq

log = logging.getLogger("amz")

@dataclass
class FetchResult:
    ok: bool
    status: int
    payload: dict | None
    attempts: int
    error: str | None = None

class AmazonClient:
    """One call = one request with a fingerprint, timeouts, and a budget."""

    ENDPOINTS = {                       # paths per the official integration docs
        "product":    "/api/v1/amazon/product",
        "search":     "/api/v1/amazon/search",
        "review":     "/api/v1/amazon/review",
        "bestseller": "/api/v1/amazon/bestsellers",
    }

    def __init__(self, base_url: str, token: str | None = None,
                 profile: str = "chrome", proxy: str | None = None,
                 max_attempts: int = 3):
        self.base = base_url.rstrip("/")
        self.token = token or os.environ["PANGOLINFO_TOKEN"]
        self.profile = profile
        self.proxies = {"https": proxy} if proxy else None
        self.max_attempts = max_attempts
        self.session = creq.Session(impersonate=profile)

    def fetch(self, kind: str, params: dict, marketplace: str) -> FetchResult:
        url = self.base + self.ENDPOINTS[kind]
        headers = {
            "Authorization": f"Bearer {self.token}",
            "Accept-Language": self._lang(marketplace),
        }
        delay = 1.0
        for attempt in range(1, self.max_attempts + 1):
            try:
                r = self.session.get(url, params={**params, "marketplace": marketplace},
                                     headers=headers, proxies=self.proxies,
                                     timeout=(5, 30))          # (connect, read)
                if r.status_code == 200:
                    return FetchResult(True, 200, r.json(), attempt)
                if r.status_code in (429, 503):                # backoff-able
                    self._sleep_backoff(attempt, delay, r.headers.get("Retry-After"))
                    continue
                return FetchResult(False, r.status_code, None, attempt, f"http {r.status_code}")
            except Exception as e:                             # timeout, reset, parse
                log.warning("attempt %s failed: %s", attempt, e)
                self._sleep_backoff(attempt, delay)
        return FetchResult(False, 0, None, self.max_attempts, "exhausted")

    @staticmethod
    def _sleep_backoff(attempt: int, delay: float, retry_after: str | None = None):
        wait = float(retry_after) if retry_after else delay * (2 ** (attempt - 1))
        time.sleep(min(wait + random.uniform(0, 0.4), 30))     # jitter, avoid retry sync

    @staticmethod
    def _lang(marketplace: str) -> str:
        return {"US": "en-US,en;q=0.9", "DE": "de-DE,de;q=0.9",
                "JP": "ja-JP,ja;q=0.9", "UK": "en-GB,en;q=0.8"}.get(marketplace, "en-US,en;q=0.9")

Three details get skipped: timeout as a tuple with five seconds to connect and thirty to read, so half-open connections do not stall the worker; backoff with jitter, or a batch of failures retries in the same second; and attempts returned, because it is a variable in the cost formula in section 14.

If you are still scraping, point session.get at the product URL, keep impersonate, wire in the proxy pool, and leave the rest alone. We worked through the cost line between scraping and calling a data API in an earlier piece.

8. Field contract: agree on what the thing is called

The most common data incident is not missing data, it is data whose meaning shifted. price was the Buy Box price yesterday and the list price today. reviews was a count last week and a rating this week. Upstream does not announce it and downstream does not ask.

from dataclasses import dataclass
from typing import Optional

@dataclass(frozen=True)
class ProductSnapshot:
    asin: str
    marketplace: str
    captured_at: str          # ISO date, part of the key
    title: str
    brand: Optional[str]
    price: Optional[float]    # Buy Box price; currency alongside it
    currency: str
    rating: Optional[float]   # 0-5
    review_count: Optional[int]
    bsr_main: Optional[int]
    bsr_category: Optional[str]
    contract_version: str = "2026-09-01"

P0_FIELDS = ("asin", "title", "price", "currency")   # missing any means unusable

Three disciplines: never change a field’s meaning in place, bump contract_version instead; mark every nullable field Optional so no None arrives by surprise; and use the capture date rather than the write time, so a rerun lands on the same date.

9. Quality gate: coverage and fill rate are two numbers

Collapse these into one success rate and you lose the ability to detect degradation.

• Coverage: of the 1,000 ASINs on my list, how many came back. Denominator is the job list.
• Fill rate: of the records returned, what share has P0 fields populated. Denominator is returned records.

def is_robot_check(html_or_json) -> bool:
    """Interstitials often return 200, so judge by content."""
    if isinstance(html_or_json, str):
        markers = ("[email protected]", "Enter the characters you see",
                   "Robot Check", "automated access")
        return any(m in html_or_json for m in markers)
    return False

def quality_gate(records: list[dict], wanted: set[str]) -> dict:
    got = {r["asin"] for r in records if r.get("asin")}
    usable = [r for r in records
              if not is_robot_check(r) and all(r.get(f) not in (None, "") for f in P0_FIELDS)]
    return {
        "coverage": len(got & wanted) / max(len(wanted), 1),
        "fill_rate": len(usable) / max(len(records), 1),
        "usable": len(usable),
        "blocked": len(records) - len(usable),
    }

Set thresholds by business: for price monitoring, coverage at or above 98% and P0 fill rate at or above 95%, and block the whole batch with an alert below that rather than letting half a batch reach downstream. Half a batch costs more than none, because it looks like success.

This layer is a section of its own in any Amazon data pipeline, since it is the one module you rewrite when the upstream vendor changes. Domains and exits change; the definition of degraded data does not.

10. Paging and dedupe: fix the key first

Get the key wrong and the better your paging, the more duplicates you store. Four elements: asin + marketplace + captured_at + contract_version. Two locate the object, one the moment, one the definition.

def dedupe(rows: list[dict]) -> list[dict]:
    seen, out = set(), []
    for r in rows:
        key = (r["asin"], r["marketplace"], r["captured_at"], r.get("contract_version", ""))
        if key in seen:
            continue
        seen.add(key)
        out.append(r)
    return out

Two paging traps. First, depth caps: Amazon exposes a limited number of result pages, and requests past that return the first page or an empty one, which reads as growth that is slower than expected. Second, the result set moves under you, so page two on a second pass can hold different products. Drive jobs from a fixed ASIN list instead of discovering the set by paging.

11. Four failure classes: not every failure deserves a retry

Sort failures into four classes with different responses. Blending them produces the worst of both: retries that should have happened do not, and retries that should not have happened multiply the invoice.

The fourth is where teams go wrong, because it is not a failed request, it is a changed upstream. A hundred retries return the same empty field at a hundred times the cost. Count each class on its own and spend the retry budget on the first two only.

12. Concurrency: a semaphore, not random.sleep

More workers is not more throughput. Three numbers set the ceiling: the rate grant upstream, your exit IP count, and what the target tolerates per IP. Hold in-flight requests with asyncio.Semaphore and requests per second with a token bucket.

import asyncio, time

class TokenBucket:
    def __init__(self, rate: float, burst: int):
        self.rate, self.tokens, self.burst, self.ts = rate, burst, burst, time.monotonic()

    async def take(self):
        while True:
            now = time.monotonic()
            self.tokens = min(self.burst, self.tokens + (now - self.ts) * self.rate)
            self.ts = now
            if self.tokens >= 1:
                self.tokens -= 1
                return
            await asyncio.sleep(1 / self.rate)

async def run_all(client, jobs, concurrency: int = 4, rps: float = 2.0):
    sem, bucket = asyncio.Semaphore(concurrency), TokenBucket(rps, burst=int(rps * 2) + 1)
    async def one(job):
        async with sem:
            await bucket.take()
            return await asyncio.to_thread(client.fetch, *job)
    return await asyncio.gather(*(one(j) for j in jobs))

Tune in one direction: start at four workers, watch the 429 ratio against end-to-end latency, and raise both. At the point where throughput flattens and queue time grows, drop back one notch. The ceiling is, in most cases, the upstream grant, not your hardware.

13. Storage: write snapshots, not overwrites

Price monitoring, rank tracking, and review growth all assume history. With INSERT OR REPLACE and the key from section 10, reruns are idempotent and history accumulates on its own.

import sqlite3

DDL = """
CREATE TABLE IF NOT EXISTS product_snapshot (
  asin TEXT NOT NULL, marketplace TEXT NOT NULL,
  captured_at TEXT NOT NULL, contract_version TEXT NOT NULL,
  title TEXT, brand TEXT, price REAL, currency TEXT,
  rating REAL, review_count INTEGER, bsr_main INTEGER, bsr_category TEXT,
  PRIMARY KEY (asin, marketplace, captured_at, contract_version)
);"""

def save(con: sqlite3.Connection, rows: list[ProductSnapshot]) -> int:
    con.executemany(
        "INSERT OR REPLACE INTO product_snapshot "
        "(asin, marketplace, captured_at, contract_version, title, brand, price,"
        " currency, rating, review_count, bsr_main, bsr_category) "
        "VALUES (?,?,?,?,?,?,?,?,?,?,?,?)",
        [(r.asin, r.marketplace, r.captured_at, r.contract_version, r.title, r.brand,
          r.price, r.currency, r.rating, r.review_count, r.bsr_main, r.bsr_category)
         for r in rows])
    con.commit()
    return len(rows)

Change detection collapses into one window function: order an ASIN’s rows by date and compare LAG(price) with the current value. No separate comparison job needed.

14. Cost instrumentation: put money inside the call

For a self-built route:

cost per 1,000 usable records = 1000 x unit price x mean attempts x (1 + render multiple)
                                --------------------------------------------------------
                                     success rate x fill rate x (1 - block rate)
                                + proxy traffic per 1,000 + engineering amortization per 1,000

All three numbers in the denominator come from your own logs, not from a vendor average. Mean attempts comes from attempts in section 7. Proxy traffic bills by the gigabyte, challenge pages included. Engineering amortization is the line left out most often: one defense upgrade costs two or three person-days to diagnose, and that spreads across every record that month.

Then run a comparison: 1,000 ASINs, same cadence, one week self-built and one week on a data API, and compare three numbers, cost per 1,000 usable records, median data latency, and pages at 3 a.m. The third number settles the argument in most cases.

15. Telemetry: four metrics, two alerts

Four metrics cover it: coverage, P0 fill rate, p95 end-to-end latency, and cost per 1,000 usable records. The first two come from section 9, the third from per-request timing, the fourth from the section above.

Two alerts are enough:

• Silent degradation: coverage or fill rate below threshold for two consecutive periods. Degradation or a redesign is the likely cause, and you want to know before your users do.
• Cost drift: cost per 1,000 usable records up more than 30% week over week. Block rates or runaway retries are the usual cause, and the success rate can still look healthy at that point.

Put attempts, status, marketplace, and contract_version on every log line. The first question in any incident is when it started and which marketplaces it touches, and without those four fields you get to rerun history to find out.

16. Where Pangolinfo sits in this code

Lay the structure out and three of the four gates plus eight steps are yours by necessity: define the field contract, define the quality thresholds, define what the data is for. The rest exists because clean data does not arrive without infrastructure.

The Pangolinfo Amazon Scraper API takes the rest. Inside the single request you send:

The division in one sentence: you send one request and receive structured JSON, and the seven layers in between belong to us. Billing uses a single record-based unit, with no residential IP surcharge, no render multiple, and no per-endpoint difficulty multiplier. Rates and plans sit on the pricing page, and integration details in the API documentation.

Three things we do not do, and say so up front: we do not define your field contract, since whether price means Buy Box or list price is your business decision; we do not decide retention windows or permitted uses, which sit with you and your counsel; and we do not collect data behind a login, so seller central, orders, and ad reports stay out of scope.

The value is less about the proxy bill than about deleting one cycle from your roadmap: defense upgrade, triage, code change, backfill.

17. Questions teams ask

Eleven questions in three groups: the first three come from “it does not run”, the next three from “is this going to get us in trouble”, and the last five from “is it worth the money”. That is also the order they reach our integration inbox.