- Add last_scrape_ok and last_scrape_at columns (migration 004)
- Update scrape status after every search, test, and streaming search
- Search page: broken stores show red X checkbox, strikethrough name,
"failing" label, and are auto-deselected on page load
- Untested stores show "untested" label
- Users can still manually select broken stores if they want to try
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parallel scraping with Puppeteer blocks the Node.js event loop,
preventing SSE events from flushing. Sequential processing means
each store completes, sends its event, and the client sees it
before the next store starts.
Also sorts stores so cheerio-based (fast) stores run first,
giving the user results sooner while Puppeteer stores load.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without yields, the async operations block the event loop and
SSE events pile up unsent until the entire search completes.
Adding setTimeout(0) yields after start and store_complete events
lets Node.js flush the write buffer to the client.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backend: new /api/search/stream SSE endpoint that emits events
as each store completes: start (store list), store_complete
(results + duration), store_error, and done (final meta).
Frontend: results page now shows live progress per store with
spinning indicators while searching, checkmarks when done, and
X marks on errors. Each store chip shows product count and
response time. Results stream into the table as stores complete
instead of waiting for all stores to finish.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add browser-scraper.ts using Puppeteer for JS-heavy stores
- Add render_js flag to store model, migration, YAML sync, and UI
- Scraper engine auto-selects cheerio vs Puppeteer based on flag
- Store forms include JS rendering toggle in Advanced section
- Create first store config: HG Spot (Croatian electronics retailer)
- Update Dockerfile with Chromium for production Puppeteer support
Tested: HG Spot returns 15 products per page with correct names,
prices (EUR), links, and images using headless browser rendering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
