datawebotBuild a product price history
Keep observations without overwriting yesterday's evidence.
Keep the right fields.
Workflow field map| Field / step | Input | Rule |
|---|---|---|
| Observation key | storefront_id + variant.id + retrieved_at | Use strings for identifiers. |
| Price | amount + currency | Store decimals; preserve the original currency. |
| Completeness | run completeness + warnings | Retain partial and missing-value flags. |
| Variant | SKU | Example price | Example availability |
|---|---|---|---|
| Small / Navy | DEMO-TEE-S-NV | USD 32.00 | Example: available |
| Medium / Navy | DEMO-TEE-M-NV | USD 32.00 | Example: available |
| Medium / Sand | DEMO-TEE-M-SD | USD 34.00 | Example: unavailable |
| Large / Navy | DEMO-TEE-L-NV | USD 32.00 | Example: unknown |
All names, identifiers, prices and availability values are fictional. They do not describe a merchant’s catalog or inventory.
Store observations, not just the latest price.
Keep a source namespace, stable variant identity and retrieval timestamp in every row. Re-importing the same observation must not create a second measurement.
Scroll to compare →
| Keep | Record | Decision |
|---|---|---|
| Primary key | Storefront + product ID + variant ID + retrieval time | A retry of the same saved result becomes a no-op. |
| Price representation | Source decimal text + ISO currency | Avoid binary floating-point storage; use decimal arithmetic for calculations. |
| Comparison basis | Market + pack + condition + seller where applicable | Add these fields when adapting the recipe to marketplace offers. |
| Evidence | Source URL + completeness + warning JSON | Retain the original export alongside the normalized observation table. |
| History | Append a newly retrieved observation | Do not overwrite yesterday's record when today's value is unchanged. |
Run the synthetic price-history exercise.
Download the Python recipe and the two fictional JSON records below into one folder. The product, merchant URL, timestamps and prices are invented to demonstrate the import and comparison rules. The recipe uses Python’s standard SQLite and decimal libraries; no API key or live request is needed.
python price-history.py history.sqlite price-history-example-first.json price-history-example-later.jsonOne price changes; three stay the same.
In this fictional exercise, Small / Navy changes from 32.00 to 34.00 USD between two invented timestamps. The other three prices stay the same. An unchanged price still produces a new observation, while re-importing either file must not duplicate it.
Scroll to compare →
| Variant | First observation | Later observation |
|---|---|---|
| Small / Navy | 32.00 USD | 34.00 USD · change +2.00 |
| Medium / Navy | 32.00 USD | 32.00 USD · unchanged |
| Medium / Sand | 34.00 USD | 34.00 USD · unchanged |
| Large / Navy | 32.00 USD | 32.00 USD · unchanged |
Extend the recipe deliberately.
The illustration uses one fictional merchant and a stable option tuple. Marketplace offers need additional identity and delivery fields before comparisons are meaningful.
Scroll to compare →
| Keep | Record | Decision |
|---|---|---|
| Collection schedule | Your scheduler starts a bounded extraction | Save the run ID before yielding and keep a spending ceiling. |
| Result retention | Retrieve all result pages, then persist your own copy | The extraction service is not a permanent historical-price archive. |
| Corrections | Keep the original observation and append a correction record | Do not silently edit raw evidence after downstream decisions use it. |
| Currency conversion | Separate rate, source and effective date | Keep the original price unchanged; do not mix conversions into raw history. |
Content reviewed 2026-10-07.
Before you use the output
Schedule collection in your application. A saved extraction does not create recurring monitoring. Compare the same variant, market and currency before calculating a change.
Check one record first.
Start with the smallest useful scope. Compare the returned record with its source, confirm variant identity and inspect missing fields before applying the mapping to a larger dataset.
Keep extraction warnings and the retrieval timestamp with downstream output. When a required field is missing, leave it unresolved or return to the source; do not fill it from an assumption.