datawebotURLs and extraction scope
Learn how product, collection and catalogue URLs determine datawebot. extraction scope, page traversal, markets, variants and returned records.
Product, collection or catalogue
| Operation | Input | Result |
|---|---|---|
| product | Product URL | One product and its reported variants |
| collection | Collection URL | Products on the visited collection pages |
| catalogue | Store homepage | Products on the visited catalogue pages |
URL shape examples
These reserved placeholder URLs show the product, collection and store path shapes. They are not real storefronts and must not be submitted. Replace them only with a public URL you are authorized to use and the service currently supports.
- https://merchant.example/products/sample-item
- https://merchant.example/collections/sample-group
- https://merchant.example/
Filters and variant selection
Tracking parameters and fragments are discarded. Current retrieval does not apply URL query filters, sort orders or selected variants. Product requests include all reported variants.
Review the normalized source before submitting. Use public HTTPS URLs without credentials, private sessions or custom ports.
Market and currency
Current reviewed storefronts use their default US market and USD. A country's reference page does not automatically enable a different storefront.
Prices are source observations. They do not include inferred exchange rates, shipping charges, personal discounts or tax calculations.
Limits and another source
The current maximum is 25 page visits per request, including one currency-check page. Each listing requests 50 products per feed page and extracts every usable record returned. Follow limits and the hard budget determine traversal, not record counts.
Another public ecommerce URL enters the same Processing flow. Additional confirmation may be required before execution.
Following pages
Pagination chooses the total result pages per listing, including the first. Linked products chooses the first N detail pages across all visited listings. Collections chooses the first N other collections from one collection-directory page; that directory also consumes credit. Collections have no recursive levels in this adapter.
The adapter visits the currency check, initial listing and pagination, collection directory and collections with their pagination, then linked product details. Other page types are not followed. A homepage input selects the catalogue feed; the HTML homepage is not fetched.
The default is 2 pages and 2 credits, with following off. Increase both budgets if you want more visits. The estimate is a bounded maximum, not a promise that all selected pages exist.
Need help? Contact support.