Two number inputs next to the text filter let users set a price
range. Results outside the range are hidden instantly. Products
with null prices are excluded when a filter is active.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Search page: each store now shows its average response time
(e.g., "~850ms") pulled from scrape_logs. Shows "untested" or
"failing" for stores without successful scrapes.
2. Results page: store progress badges are now clickable buttons.
Click a completed or failed store to re-search just that store.
Old results from that store are removed, new ones stream in.
Useful for retrying stores that had connection errors.
Hover tooltip shows "Click to retry" or "Click to refresh".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Share pages (/share/*) now show no hamburger button and no
sidebar — just the results table with header. Clean, distraction-free
view for shared link recipients.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When sharing a search, the actual results are now saved alongside
the search config. Opening a shared link shows the cached results
instantly instead of re-running the search against all stores.
- Add results_json column to shared_searches (migration 006)
- Frontend sends current results array when creating a share
- Share page loads cached results directly, shows date saved
- "Search Live Prices" button available to re-run with fresh data
- Falls back to live search if no cached results exist
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove proxy_url from store create/edit forms (field exists in DB
but neither scraper implements proxy support — misleading UX)
- Rename updateGroupApi → updateGroup, setGroupMembersApi → setGroupMembers
for consistent naming with all other API functions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Save a search and get a shareable URL at /share/<id> that preserves:
- Search query
- Selected stores
- Filter text
- Excluded stores
- Sort column and direction
Backend:
- New shared_searches table (migration 005)
- POST /api/share — saves search state, returns share_id
- GET /api/share/:id — loads saved search config
Frontend:
- Share button appears on results page after search completes
- Copies link to clipboard with confirmation message
- /share/[id] page loads saved config, re-runs search with same
store selection, and restores all filters
- Shows "Shared" badge in header with "Search Again" link
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The .product-price class doesn't exist on Ronis.hr. Prices are
in .prices > .price containers. This was causing 0 results since
empty priceText filtered out all products.
BigBang.hr thumbnails: images are blocked by Cloudflare hotlink
protection when loaded cross-origin — this is expected and not
fixable server-side. Product data (name, price, link) works fine.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove puppeteer dependency, use puppeteer-core instead (no bundled Chromium)
- Remove local launch fallback from browser-scraper
- Fail fast with clear error if CHROMIUM_WS is not set
- Simplify closeBrowser to disconnect only
Docker Chromium container is now required for JS-rendered stores.
Dev: docker compose -f docker-compose.dev.yml up -d
Prod: docker compose up (starts both app + chromium)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Croatian electronics retailer behind Cloudflare. Works via Docker
Chromium with stealth plugin. 35 products per page, article.cp
containers with .cp-title for names, .current-price for prices.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Switch puppeteer to puppeteer-extra with stealth plugin to
bypass Cloudflare bot detection
- Add Ronis.hr store config (JS-rendered, 48 products per page)
- Stealth mode patches navigator.webdriver, chrome runtime, and
other fingerprints that Cloudflare checks
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Croatian electronics retailer. Uses .product-grid.product-item
containers, h2.title for names, .price for prices. Static HTML,
no JS rendering needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security:
- Fix SQL injection in updateStore — whitelist allowed field names
- Restrict CORS to same-origin in production
- Cap results at 200 per store to prevent memory issues
Code quality:
- Extract shared queryAll/queryOne to src/server/db/query.ts
- Remove duplicated DB helpers from 5 files
- Handle render_js boolean-to-integer conversion in updateStore
UX:
- Validate headers_json as valid JSON before saving (both forms)
- Show error message if JSON is invalid
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add last_scrape_ok and last_scrape_at columns (migration 004)
- Update scrape status after every search, test, and streaming search
- Search page: broken stores show red X checkbox, strikethrough name,
"failing" label, and are auto-deselected on page load
- Untested stores show "untested" label
- Users can still manually select broken stores if they want to try
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each store can now have its own test_query (e.g., "logitech" for
electronics stores). The "Test All" button uses each store's
configured query instead of prompting — just click and watch.
- Add test_query column (migration 003)
- Add field to YAML sync, store forms, and route schema
- Set test_query in HG Spot and Links.hr configs
- Test All runs immediately using per-store queries
- Hover test result to see which query was used
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Click 'Test All' to enter a search query, then each enabled store
is tested sequentially. A 'Test Result' column appears showing:
- Spinner while testing
- Green checkmark with product count and duration on success
- Red X with error on failure
- Disabled stores show dash
Tests run one at a time so results stream in visibly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each category is now a column side by side, so adding more
categories grows horizontally instead of vertically.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clean vertical layout with category headers and indented store
checkboxes. Filled purple checkbox when selected, empty border
when not. Hover highlight on each row.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces dropdown filters with visual store picker organized by
category. Each category shows as a header with checkbox — clicking
it selects/deselects all stores in that category. Individual stores
are toggleable chips with checkmarks. Partial selection shows a
dash indicator on the category checkbox. Only passes specific
store IDs to search when not all stores are selected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Preview now uses position:fixed with z-index 9999, positioned
dynamically via JS on mouseenter. No longer clipped by parent
overflow containers or table cells.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hovering a product thumbnail shows a 192x192 enlarged preview
in a dark container with border and shadow. Appears to the right
of the thumbnail with a smooth scale+fade animation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
tsx watch hangs when puppeteer/chromium imports are involved.
Plain tsx starts reliably. Hot reload isn't critical since
server changes are infrequent compared to client changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add browserless/chromium container to docker-compose
- Add docker-compose.dev.yml for local dev (Chromium on port 3001)
- Browser scraper connects via WebSocket (CHROMIUM_WS env var)
- Falls back to local launch if CHROMIUM_WS not set
- Remove Chromium install from Dockerfile (smaller image)
- Auto-reconnect on browser disconnect
Tested: remote Chromium connects in ~500ms, HG Spot scrapes in
~2.2s total. No longer blocks the Node.js event loop.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Chromium cold launch takes several seconds and blocks the event
loop, preventing SSE events from flushing. Now the browser is
warmed up during server startup if any store uses render_js,
so the first search doesn't pay the launch penalty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs each phase (launch, navigate, wait selector, extract, close)
so we can diagnose where Puppeteer gets stuck. Also skips the
expensive page.content() call since full HTML is only needed
for the test endpoint, not search.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parallel scraping with Puppeteer blocks the Node.js event loop,
preventing SSE events from flushing. Sequential processing means
each store completes, sends its event, and the client sees it
before the next store starts.
Also sorts stores so cheerio-based (fast) stores run first,
giving the user results sooner while Puppeteer stores load.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without yields, the async operations block the event loop and
SSE events pile up unsent until the entire search completes.
Adding setTimeout(0) yields after start and store_complete events
lets Node.js flush the write buffer to the client.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
networkidle2 waits for all network activity to settle, which hangs
on sites with analytics, trackers, and websockets. domcontentloaded
fires much earlier, then waitForSelector handles the dynamic content.
HG Spot now completes in ~2.5s instead of timing out.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fastify was buffering the SSE response, causing store progress
to appear stuck. Using reply.hijack() hands off the raw response
so events flush immediately to the client.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backend: new /api/search/stream SSE endpoint that emits events
as each store completes: start (store list), store_complete
(results + duration), store_error, and done (final meta).
Frontend: results page now shows live progress per store with
spinning indicators while searching, checkmarks when done, and
X marks on errors. Each store chip shows product count and
response time. Results stream into the table as stores complete
instead of waiting for all stores to finish.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Puppeteer import at top level was blocking tsx watch mode,
preventing the server from starting. Now imported dynamically
only when a JS-rendered store is actually scraped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Croatian electronics retailer, 24 products per page.
Uses .card.mobile-card containers with h3 for names,
.product-price for prices, a.card-link for links.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bigger padding, larger icon, rounded-lg, ring on focus,
text-sm instead of text-xs. Matches the search bar feel.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Header now shows "Search results for: <keyword>" with meta stats,
and a "New Search" button that navigates back to the homepage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Filter bar now has two rows: text filter + count on top,
store toggle chips on the bottom. Chips wrap on narrow screens.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each store appears as a clickable chip with a checkmark. Click to
exclude a store from results (greys out with strikethrough). Click
again to re-include. Multiple stores can be toggled independently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Compact table view with thumbnail, product name, price, store, link
- Click column headers to sort (toggles asc/desc)
- Filter bar to search within results by product name
- Store dropdown filter when multiple stores present
- Sticky header, hover-reveal "Open" link, sort direction arrows
- Shows "X of Y shown" count when filtering
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add try/catch to all onMount API calls so the UI recovers
gracefully when the server isn't ready yet instead of hanging
on the loading skeleton forever.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
