Logs each phase (launch, navigate, wait selector, extract, close)
so we can diagnose where Puppeteer gets stuck. Also skips the
expensive page.content() call since full HTML is only needed
for the test endpoint, not search.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parallel scraping with Puppeteer blocks the Node.js event loop,
preventing SSE events from flushing. Sequential processing means
each store completes, sends its event, and the client sees it
before the next store starts.
Also sorts stores so cheerio-based (fast) stores run first,
giving the user results sooner while Puppeteer stores load.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without yields, the async operations block the event loop and
SSE events pile up unsent until the entire search completes.
Adding setTimeout(0) yields after start and store_complete events
lets Node.js flush the write buffer to the client.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
networkidle2 waits for all network activity to settle, which hangs
on sites with analytics, trackers, and websockets. domcontentloaded
fires much earlier, then waitForSelector handles the dynamic content.
HG Spot now completes in ~2.5s instead of timing out.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fastify was buffering the SSE response, causing store progress
to appear stuck. Using reply.hijack() hands off the raw response
so events flush immediately to the client.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backend: new /api/search/stream SSE endpoint that emits events
as each store completes: start (store list), store_complete
(results + duration), store_error, and done (final meta).
Frontend: results page now shows live progress per store with
spinning indicators while searching, checkmarks when done, and
X marks on errors. Each store chip shows product count and
response time. Results stream into the table as stores complete
instead of waiting for all stores to finish.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Puppeteer import at top level was blocking tsx watch mode,
preventing the server from starting. Now imported dynamically
only when a JS-rendered store is actually scraped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Croatian electronics retailer, 24 products per page.
Uses .card.mobile-card containers with h3 for names,
.product-price for prices, a.card-link for links.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bigger padding, larger icon, rounded-lg, ring on focus,
text-sm instead of text-xs. Matches the search bar feel.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Header now shows "Search results for: <keyword>" with meta stats,
and a "New Search" button that navigates back to the homepage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Filter bar now has two rows: text filter + count on top,
store toggle chips on the bottom. Chips wrap on narrow screens.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each store appears as a clickable chip with a checkmark. Click to
exclude a store from results (greys out with strikethrough). Click
again to re-include. Multiple stores can be toggled independently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Compact table view with thumbnail, product name, price, store, link
- Click column headers to sort (toggles asc/desc)
- Filter bar to search within results by product name
- Store dropdown filter when multiple stores present
- Sticky header, hover-reveal "Open" link, sort direction arrows
- Shows "X of Y shown" count when filtering
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add try/catch to all onMount API calls so the UI recovers
gracefully when the server isn't ready yet instead of hanging
on the loading skeleton forever.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add browser-scraper.ts using Puppeteer for JS-heavy stores
- Add render_js flag to store model, migration, YAML sync, and UI
- Scraper engine auto-selects cheerio vs Puppeteer based on flag
- Store forms include JS rendering toggle in Advanced section
- Create first store config: HG Spot (Croatian electronics retailer)
- Update Dockerfile with Chromium for production Puppeteer support
Tested: HG Spot returns 15 products per page with correct names,
prices (EUR), links, and images using headless browser rendering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sidebar is now hidden by default, opened via a small hamburger
button in the top-left corner. Clicking a nav link or the overlay
closes it. Keeps focus on the search bar.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Dark-first design with #0a0a0b backgrounds and subtle borders
- Purple/violet accent colors for primary actions
- Inter font with custom design tokens
- Sidebar navigation replacing top nav bar
- Compact, information-dense tables and cards
- Consistent component classes (btn-primary, input-field, card, label)
- Custom scrollbar and selection styling
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Stores can now be defined as YAML files in the stores/ directory.
On startup, YAML files are synced into the database. Changes made
via the admin UI are written back to YAML files automatically.
- Add store-sync service (load from files, export to files, write-back)
- Add /api/stores/sync and /api/stores/export endpoints
- Add Sync/Export buttons to admin UI
- Mount stores/ volume in Docker
- Include example store config template
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
