Croatian electronics retailer behind Cloudflare. Works via Docker
Chromium with stealth plugin. 35 products per page, article.cp
containers with .cp-title for names, .current-price for prices.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Switch puppeteer to puppeteer-extra with stealth plugin to
bypass Cloudflare bot detection
- Add Ronis.hr store config (JS-rendered, 48 products per page)
- Stealth mode patches navigator.webdriver, chrome runtime, and
other fingerprints that Cloudflare checks
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Croatian electronics retailer. Uses .product-grid.product-item
containers, h2.title for names, .price for prices. Static HTML,
no JS rendering needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security:
- Fix SQL injection in updateStore — whitelist allowed field names
- Restrict CORS to same-origin in production
- Cap results at 200 per store to prevent memory issues
Code quality:
- Extract shared queryAll/queryOne to src/server/db/query.ts
- Remove duplicated DB helpers from 5 files
- Handle render_js boolean-to-integer conversion in updateStore
UX:
- Validate headers_json as valid JSON before saving (both forms)
- Show error message if JSON is invalid
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add last_scrape_ok and last_scrape_at columns (migration 004)
- Update scrape status after every search, test, and streaming search
- Search page: broken stores show red X checkbox, strikethrough name,
"failing" label, and are auto-deselected on page load
- Untested stores show "untested" label
- Users can still manually select broken stores if they want to try
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each store can now have its own test_query (e.g., "logitech" for
electronics stores). The "Test All" button uses each store's
configured query instead of prompting — just click and watch.
- Add test_query column (migration 003)
- Add field to YAML sync, store forms, and route schema
- Set test_query in HG Spot and Links.hr configs
- Test All runs immediately using per-store queries
- Hover test result to see which query was used
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Click 'Test All' to enter a search query, then each enabled store
is tested sequentially. A 'Test Result' column appears showing:
- Spinner while testing
- Green checkmark with product count and duration on success
- Red X with error on failure
- Disabled stores show dash
Tests run one at a time so results stream in visibly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each category is now a column side by side, so adding more
categories grows horizontally instead of vertically.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clean vertical layout with category headers and indented store
checkboxes. Filled purple checkbox when selected, empty border
when not. Hover highlight on each row.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces dropdown filters with visual store picker organized by
category. Each category shows as a header with checkbox — clicking
it selects/deselects all stores in that category. Individual stores
are toggleable chips with checkmarks. Partial selection shows a
dash indicator on the category checkbox. Only passes specific
store IDs to search when not all stores are selected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Preview now uses position:fixed with z-index 9999, positioned
dynamically via JS on mouseenter. No longer clipped by parent
overflow containers or table cells.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hovering a product thumbnail shows a 192x192 enlarged preview
in a dark container with border and shadow. Appears to the right
of the thumbnail with a smooth scale+fade animation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
tsx watch hangs when puppeteer/chromium imports are involved.
Plain tsx starts reliably. Hot reload isn't critical since
server changes are infrequent compared to client changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add browserless/chromium container to docker-compose
- Add docker-compose.dev.yml for local dev (Chromium on port 3001)
- Browser scraper connects via WebSocket (CHROMIUM_WS env var)
- Falls back to local launch if CHROMIUM_WS not set
- Remove Chromium install from Dockerfile (smaller image)
- Auto-reconnect on browser disconnect
Tested: remote Chromium connects in ~500ms, HG Spot scrapes in
~2.2s total. No longer blocks the Node.js event loop.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Chromium cold launch takes several seconds and blocks the event
loop, preventing SSE events from flushing. Now the browser is
warmed up during server startup if any store uses render_js,
so the first search doesn't pay the launch penalty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs each phase (launch, navigate, wait selector, extract, close)
so we can diagnose where Puppeteer gets stuck. Also skips the
expensive page.content() call since full HTML is only needed
for the test endpoint, not search.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parallel scraping with Puppeteer blocks the Node.js event loop,
preventing SSE events from flushing. Sequential processing means
each store completes, sends its event, and the client sees it
before the next store starts.
Also sorts stores so cheerio-based (fast) stores run first,
giving the user results sooner while Puppeteer stores load.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without yields, the async operations block the event loop and
SSE events pile up unsent until the entire search completes.
Adding setTimeout(0) yields after start and store_complete events
lets Node.js flush the write buffer to the client.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
networkidle2 waits for all network activity to settle, which hangs
on sites with analytics, trackers, and websockets. domcontentloaded
fires much earlier, then waitForSelector handles the dynamic content.
HG Spot now completes in ~2.5s instead of timing out.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fastify was buffering the SSE response, causing store progress
to appear stuck. Using reply.hijack() hands off the raw response
so events flush immediately to the client.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backend: new /api/search/stream SSE endpoint that emits events
as each store completes: start (store list), store_complete
(results + duration), store_error, and done (final meta).
Frontend: results page now shows live progress per store with
spinning indicators while searching, checkmarks when done, and
X marks on errors. Each store chip shows product count and
response time. Results stream into the table as stores complete
instead of waiting for all stores to finish.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Puppeteer import at top level was blocking tsx watch mode,
preventing the server from starting. Now imported dynamically
only when a JS-rendered store is actually scraped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Croatian electronics retailer, 24 products per page.
Uses .card.mobile-card containers with h3 for names,
.product-price for prices, a.card-link for links.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bigger padding, larger icon, rounded-lg, ring on focus,
text-sm instead of text-xs. Matches the search bar feel.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Header now shows "Search results for: <keyword>" with meta stats,
and a "New Search" button that navigates back to the homepage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Filter bar now has two rows: text filter + count on top,
store toggle chips on the bottom. Chips wrap on narrow screens.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each store appears as a clickable chip with a checkmark. Click to
exclude a store from results (greys out with strikethrough). Click
again to re-include. Multiple stores can be toggled independently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Compact table view with thumbnail, product name, price, store, link
- Click column headers to sort (toggles asc/desc)
- Filter bar to search within results by product name
- Store dropdown filter when multiple stores present
- Sticky header, hover-reveal "Open" link, sort direction arrows
- Shows "X of Y shown" count when filtering
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add try/catch to all onMount API calls so the UI recovers
gracefully when the server isn't ready yet instead of hanging
on the loading skeleton forever.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add browser-scraper.ts using Puppeteer for JS-heavy stores
- Add render_js flag to store model, migration, YAML sync, and UI
- Scraper engine auto-selects cheerio vs Puppeteer based on flag
- Store forms include JS rendering toggle in Advanced section
- Create first store config: HG Spot (Croatian electronics retailer)
- Update Dockerfile with Chromium for production Puppeteer support
Tested: HG Spot returns 15 products per page with correct names,
prices (EUR), links, and images using headless browser rendering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sidebar is now hidden by default, opened via a small hamburger
button in the top-left corner. Clicking a nav link or the overlay
closes it. Keeps focus on the search bar.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Dark-first design with #0a0a0b backgrounds and subtle borders
- Purple/violet accent colors for primary actions
- Inter font with custom design tokens
- Sidebar navigation replacing top nav bar
- Compact, information-dense tables and cards
- Consistent component classes (btn-primary, input-field, card, label)
- Custom scrollbar and selection styling
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Stores can now be defined as YAML files in the stores/ directory.
On startup, YAML files are synced into the database. Changes made
via the admin UI are written back to YAML files automatically.
- Add store-sync service (load from files, export to files, write-back)
- Add /api/stores/sync and /api/stores/export endpoints
- Add Sync/Export buttons to admin UI
- Mount stores/ volume in Docker
- Include example store config template
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
