A rule only filtered articles that arrived after it was saved, so the
stories it was meant to remove stayed in the feed for up to two days.
- Saving a negative rule now also removes its matches from the feed
(checkbox, on by default). If the form's dry run of the same rule has
finished, its results are used; otherwise a check runs and removes the
matches as soon as it ends.
- A saved rule's dry run has "Remove these N from the feed".
- Removed articles are filtered with the rule and reason (Filtered,
Rescue), stories left without articles disappear, and "Put back all"
restores them without recording a correction example.
- Dry runs check what's in the feed: articles published within the
feed's time window (at least 24 h, up to 400), and honour enabled
"always keep" rules (keyword exceptions skip the model entirely).
API: POST /api/rules/:id/apply { jobId? }, POST /api/rules/:id/restore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
53 KiB
RSSFeed — Product Requirements Document
Version: 0.5 (Ready for build)
Date: 2026-11-15
Status: Ready for build kickoff — no open questions remain (§14)
Repository: https://git.mariosem.es/msemes/RSSFeed.git
Deliverable: one Docker image, git.mariosem.es/msemes/rssfeed, built and published by Gitea Actions
Reference project for stack and CI conventions: EPGMaster (git.mariosem.es/msemes/EPGMaster)
v0.5 changes: all open questions resolved. Outside access goes through Pangolin, which handles authentication and TLS, so the built-in admin token is removed. All feeds move under one
/feeds/prefix so a single Pangolin rule can cover them, and a public base URL setting keeps feed links correct behind the proxy (§5.11, §8, §9). Stories from Croatian sources get RSSFeed's own title and description like all others. Sources are added in the UI, and the name RSSFeed is confirmed.v0.4 changes: (1) Every story gets a fixed format: a one-sentence title and a description of at most two sentences (§5.6). Key points and the summary-length setting are removed, and the digest reuses each story's title and description as published (§5.8). (2) The original article is never edited, rewritten, or republished. RSSFeed adds only its own title and description and links to the article as published. (3) The UI language is English. Filter reasons are now in English too, since they are part of the UI. Story content stays Croatian.
v0.3 changes: feed controls (§5.10); importance judged neutrally; the LLM treated as given.
v0.2 changes: story clustering, importance, and the daily digest moved into MVP; feed items became stories.
1. Summary
RSSFeed is a self-hosted, single-user news pipeline. You add RSS/Atom sources; it fetches them and has a locally hosted LLM read every article. It throws away what you have said you do not want, groups articles about the same event into one story, and writes a short Croatian title and description for each story. The result is published two ways: as one merged RSS feed you can subscribe to from any reader, and as a clean dark-mode web reader built into the app.
Three things make it different from a plain feed merger:
- Own title and description. The LLM reads the full article, not just the feed excerpt, and writes a one-sentence title that says what happened plus a description of at most two sentences. The article itself is never changed: every item links to it exactly as published.
- Everything in Croatian. Sources can be in any language; the output is always Croatian.
- Negatives. You describe in plain language what you never want to see ("celebrity gossip", "horoscopes", "transfer rumours"). The LLM judges each article against those rules, and only qualified articles reach the feed.
On top of those:
- One story, not five copies. When several outlets report the same event, the feed shows it once, with links to every source.
- Importance. Every story is rated from 1 to 5 on a neutral scale.
- You set the volume. A daily limit, a minimum importance, caps per source and per category, or a few editions a day instead of a constant stream.
- Daily digest. Each morning, one item lists the previous day's most important stories.
Everything runs on your own hardware. No article text, rule, or reading habit leaves your network.
2. Goals and non-goals
Goals
| # | Goal | Phase |
|---|---|---|
| G1 | Add, edit, pause, and remove RSS/Atom sources from the UI | MVP |
| G2 | Merge all sources into a single valid RSS 2.0 feed at a stable URL | MVP |
| G3 | Give each story a one-sentence title and a description of at most two sentences, written by a local OpenAI-compatible LLM from the full article text | MVP |
| G4 | Write every title and description in Croatian regardless of source language | MVP |
| G5 | Filter articles against user-defined negatives before they reach the feed | MVP |
| G6 | Show what was filtered and why, and let the user overrule it | MVP |
| G7 | Simple, clean, dark-mode web UI in English that is comfortable to read | MVP |
| G8 | Ship as a single Docker image built by Gitea Actions | MVP |
| G9 | Group articles about the same event from different sources into one story | MVP |
| G10 | Rate every story's importance on a neutral scale | MVP |
| G11 | Publish a daily digest of the most important stories | MVP |
| G12 | Let the owner control how many stories the feed gets, and of what kind | MVP |
| G13 | Per-topic feeds and reprocessing after a model change | v1 |
Non-goals
- Not an article editor. The original article is never edited, rewritten, shortened, or republished. RSSFeed adds its own title and description, and always links to the article as published.
- Not a public republishing service. The feed carries RSSFeed's titles and descriptions plus links to the originals. It never carries article text, and it is meant for personal use.
- Not multi-user. One owner, one set of sources and rules. No accounts, roles, or sharing.
- Not a cloud-AI product. No calls to hosted LLM APIs. The LLM endpoint is whatever OpenAI-compatible server you run locally.
- Not a paywall or anti-bot bypass. If a site blocks extraction, the title and description are written from the feed excerpt and the story is marked as partial. No headless browser in MVP.
- Not a fact-checker. When sources in a story disagree, the description says so. It does not decide which source is right.
- Not a full RSS reader replacement. The built-in reader covers reading, searching, and bookmarking. Third-party reader apps get the content through the output feed; there is no reader-sync API.
3. User
One persona: the owner. Runs a homelab with Docker and Gitea, already hosts a local LLM, reads news from a mix of Croatian and foreign outlets, and is tired of wading through filler to find what matters. Wants to set it up once, tune the negatives and feed controls over a week or two, and then just read.
User stories
- As the owner, I want to paste a site or feed URL and have the feed found for me, so adding a source takes seconds.
- As the owner, I want one feed URL that contains everything, so my phone reader needs a single subscription.
- As the owner, I want each item to be a one-sentence Croatian title and at most two sentences of description, so I can decide in a few seconds whether to open the original.
- As the owner, I want the original article left exactly as published, so what I open is the source's own work.
- As the owner, I want to write negatives in plain language, so I do not have to maintain keyword lists.
- As the owner, I want to see what got filtered and which rule caused it, so I can trust the filter and fix it when it is wrong.
- As the owner, I want to test a new negative against recent articles before saving it, so I know what it will remove.
- As the owner, I want one item per event, listing every outlet that covered it, so the feed does not repeat itself.
- As the owner, I want to set how many stories my feed gets per day and what kinds, so the feed fits the time I have.
- As the owner, I want to see what my feed settings would have let through last week before I save them, so I can tune them without guessing.
- As the owner, I want a morning overview of yesterday's top stories, so I have one thing to read when I have time for only one.
- As the owner, I want nothing lost when the LLM server restarts, so articles wait and get processed when it comes back.
4. How it works
Each article moves through a fixed pipeline. Every stage is persisted, so a restart or an LLM outage resumes where it stopped.
Sources ─► Fetch ─► Dedupe ─► Extract ─► Keyword ─► Triage ─► Cluster ─► Summarize ─► Web reader
rules (LLM) (embed) (LLM) │
│ │ │ ▼
└────┬─────┘ │ Feed controls
▼ ▼ │
Filtered view Same event as an ▼
rule, reason, existing story: RSS feed
Rescue button added as a source
Daily: stories from the last 24 h ─► top 10 by importance ─► Digest (LLM) ─► Web reader and RSS
"Summarize" produces the story's title and description (§5.6). Every summarized story appears in the web reader. Feed controls (§5.10) then decide which of them enter the RSS feed, and when.
Why the work is split into stages. Each stage keeps expensive calls away from articles that do not need them, which keeps the feed fresh:
- Keyword rules cost nothing and run first.
- Triage reads only the title, excerpt, and opening of the text, so its prompt is roughly a tenth the size of a summarize prompt. Rejected articles never reach the summarize call.
- Clustering uses an embedding, which is far cheaper than a chat call. An article that joins an existing story is not summarized on its own; it is listed as an additional source.
- Importance comes out of the summarize call, so it costs nothing extra. The digest is one small call a day.
Article states
| State | Meaning |
|---|---|
new |
Fetched from a feed, not yet processed |
extracted |
Full text obtained, or fallen back to the feed excerpt |
rejected |
Matched a negative. Visible in the Filtered view, never in the feed |
kept |
Passed triage, waiting to be clustered |
grouped |
Assigned to a story. The story carries the title and description and is what appears in the feed |
failed |
A stage failed after all retries. Visible in Activity with the error |
Story states
| State | Meaning |
|---|---|
pending |
Created from its first article, waiting for its title and description |
published |
Title and description written; visible in the web reader |
updating |
Gained sources and has a refreshed title and description queued. The current version stays visible meanwhile |
failed |
Summarize failed after all retries. Visible in Activity with the error |
Separately, every published story has a feed status: in the feed, waiting for the next edition, or left out with a reason (§5.10).
Two rules hold at every stage: no article is dropped silently, and no story is published without a Croatian title and description. If the LLM is unreachable, articles and stories wait in their current state.
5. Functional requirements
5.1 Sources
- Add a source by feed URL or site URL. For a site URL, discover the feed from
<link rel="alternate">tags and common paths, and let the user pick if there are several. - Supported formats: RSS 2.0, RSS 1.0, Atom 1.0, JSON Feed.
- Per-source settings: display name, enabled, poll interval (default 15 minutes), "fetch full article" on or off (default on), language hint (default auto-detect).
- Per-source health: last successful fetch, last error, consecutive failures, articles in the last 24 hours, and the share of them that were filtered.
- A source that fails repeatedly backs off exponentially up to 6 hours and shows a warning. It is never disabled automatically.
- OPML import and export.
5.2 Fetching and deduplication
- Polite fetching: conditional GET with
ETagandLast-Modified, honorRetry-After, 20-second timeout, an identifiable User-Agent, and at most one concurrent request per host. - Deduplicate on canonical URL (tracking parameters stripped), then on feed GUID, then on a normalized title hash within a 48-hour window.
- Deduplication catches the same article seen twice. Recognizing different articles about the same event is clustering (§5.5).
- On first add, a source imports only items from the last 24 hours, so a new source does not flood the queue with its whole backlog. A source added on a quiet day still gets its 10 newest items from within the maximum age, so it never looks empty.
- Removing a source also forgets its article URLs, so adding the same feed again imports its current items.
- Quiet hours (optional, e.g. 23:00–05:00 local time): no scheduled fetching and no background processing, so no AI calls. New items wait in the feeds and are fetched and processed when the window ends. Editions, digest creation and cleanup still run on time (they cost no AI), and a digest due in the window gets its opening when it ends. Things started by hand (Fetch now, Playground, connection test) still run. A banner shows when quiet hours are on.
- Items older than a configurable maximum age (default 72 hours) at fetch time are stored but skipped by the pipeline.
5.3 Full-text extraction
- Fetch the article page and extract the main content with a Readability-style extractor.
- If extraction fails, returns too little text, or the page is blocked, fall back to the feed's own content and mark the article
partial. A partial article is still processed, and the UI labels its story as based on the excerpt only. - Extracted text is input to the LLM and nothing else. It is stored locally for triage, clustering, and regeneration, but never shown in the reader, served in a feed, or modified. Reading the article always means opening the original.
- Also capture the lead image URL, author, and publication date where available.
5.4 Negatives (the filter)
Two kinds of rule, both managed on one screen:
| Kind | How it works | Cost |
|---|---|---|
| AI rule | A plain-language description, such as "Celebrity gossip, showbiz, and influencer news with no public-interest angle". The LLM judges each article against all active AI rules. | One short LLM call per article |
| Keyword rule | Words, phrases, or a regex matched against title and feed categories. Also covers "skip this category from this source". | Free, runs before the LLM |
Requirements:
- Each rule has: text, enabled toggle, optional scope (all sources or selected ones), and a hit counter with the date of the last hit.
- Rules can be written in any language. The model applies them to articles in any language.
- Keyword rules run first. An article they reject never costs an LLM call.
- The triage call returns structured JSON: decision, the matching rule, a one-sentence reason in English for the Filtered view, and a confidence from 0 to 1.
- An article is rejected only when confidence is at or above the threshold (default 0.7, adjustable as a "strictness" setting). When the model is unsure, the article is kept. A wrongly hidden important story costs more than one piece of filler getting through.
- Exceptions ("always keep"). Optional positive rules that override negatives, for example "Anything about Croatia's national football team" alongside a negative about sports.
- Dry run. Before saving a new or edited rule, run it against the articles published within the feed's time window (at least 24 hours, up to 400) and show what it would remove. Enabled "always keep" rules are honoured, so the result is what the filter would really do.
- Apply to the current feed. Saving a negative rule also removes its matches from the feed right away (on by default; one checkbox). A finished dry run of the same rule is reused, so nothing is judged twice. A saved rule's dry run can be applied with one click. Removed articles appear under Filtered with the rule and reason, a story left without articles disappears, and Put back all restores them without recording a correction.
- Filtered view. Every rejected article is listed with its rule and reason. Rescue sends it on to clustering and summarize, like any kept article.
- Feedback as examples. A rescued article, or a published story the user marks "should have been filtered", is stored as a labeled example. The most recent examples (default 10) are included in the triage prompt, so the filter adapts to corrections without retraining anything.
- Rejected articles never join a story.
- Rejected articles are kept for 14 days for review, then pruned.
Negatives decide what is worth processing at all. Feed controls (§5.10) decide how much of what survives goes into the RSS feed.
5.5 Story clustering
A story is one news event as reported by one or more sources. Every kept article belongs to exactly one story. Most stories have a single source. Clustering is what lets the feed show an event once when several outlets cover it.
Matching an article to a story
- Embed the article's original title and the first ~1,000 characters of its text with a multilingual embedding model, so a Croatian and an English article about the same event can match.
- Compare the embedding against the articles of every story that started within the clustering window (default 48 hours). At the expected volume that is about 1,000 vectors, compared in memory.
- Decide by the best similarity score:
| Similarity | Decision |
|---|---|
| At or above the merge threshold | Join that story |
| Between the review and merge thresholds | Ask the triage model a short yes-or-no question: is this the same event? Join if yes |
| Below the review threshold | Start a new story |
- "Same event" means one specific occurrence: the same incident, decision, match, or announcement. Two articles on the same topic are different stories, for example two separate developments in the same war on the same day. This definition is written into the confirmation prompt.
The thresholds depend on the embedding model. Defaults are calibrated on real data during milestone M3, and the playground (§5.9) shows the similarity between any two articles so they can be tuned.
What a story shows
- One Croatian title and description (§5.6); every source with its original headline, unchanged, and a link to the article as published; and when the story was first and last reported.
- The story's main link points to its lead article. By default the lead is the first non-partial article, preferring Croatian-language sources so the original is readable without translation. A setting turns off the Croatian preference.
Keeping the title and description current
- An article that joins an existing story does not get its own title and description.
- The story's title and description are regenerated from up to 4 member articles, truncated to fit the context window, when:
- the story reaches 3 sources, or
- a new member arrives more than 6 hours after the current version was written, which keeps developing stories up to date.
- At most 3 regenerations per story. The source count, the 6-hour gap, and the cap are all settings.
- Regenerated text keeps the same format limits. Where member articles disagree on a key fact, the description says so (for example "Izvori se razlikuju oko broja žrtava.") instead of picking one.
- The story keeps its feed GUID through every regeneration and every new source, so readers update the item instead of showing a new one.
Corrections
- Split removes an article from a story and makes it a story of its own.
- Merge joins two stories that clustering kept apart.
- Both are stored as examples for the confirmation prompt, the same way filter corrections are (§5.4).
5.6 Title and description
One LLM call per new story produces structured JSON:
| Field | Description |
|---|---|
title_hr |
Exactly one sentence in Croatian that says what happened. Written as a headline, with no full stop at the end. This is RSSFeed's own title; the article's headline is left as published and shown in the source list |
description_hr |
At most two sentences in Croatian with the core facts: who, what, where, when, and the key number if there is one |
category |
One value from a fixed list (§5.12) |
importance |
Integer from 1 to 5, scored against the rubric in §5.7 |
source_lang |
Detected language of the lead article |
Requirements:
- The article itself is never changed. The title and description are RSSFeed's own text, shown next to a link to the article exactly as published. The article is never rewritten, shortened, or re-hosted.
- The format is fixed and checked. After generation, the app verifies that the title is one sentence and the description is no more than two. The sentence counter recognizes Croatian ordinal dates ("15. studenoga"), numbers, and common abbreviations ("npr.", "tj.", "dr."), so it does not count them as sentence ends. On a violation, the call is retried once with a stricter prompt; if it still fails, the app keeps the first sentence of the title and the first two of the description.
- The prompt requires standard Croatian (hrvatski standardni jezik, ijekavica) and forbids Serbian or Bosnian forms. Models commonly blur these, so this is a stated rule and a checked one, not an assumption.
- Language guard. After generation, verify the output is Croatian with a cheap heuristic: diacritics, stop-word profile, and the absence of Cyrillic and of typical ekavian forms. On failure, retry once with a stricter prompt, then mark the story
failedso it never reaches the feed in the wrong language. The guard applies to every Croatian text the app produces, including regenerations and the digest opening. - Grounding. The prompt allows only facts present in the provided articles. Names and numbers are carried over exactly. No commentary, no speculation, and no clickbait: the title states what happened rather than teasing it.
- Stories from Croatian-language sources skip translation but get their own title and description like every other story. A source's own headline and excerpt are never used as the feed text.
- A user glossary of fixed translations and of terms never to translate (names of institutions, products, clubs).
- Every version is stored with the model, prompt version, token counts, and duration. This makes it possible to reprocess the last N days after changing model or prompt (v1).
- A Regenerate action on any story re-runs its title and description.
5.7 Importance
Every story gets an importance score from 1 to 5, judged neutrally: by the scale of the consequences and the number of people affected, regardless of country or region. A Croatian story and a foreign story of the same scale get the same score.
| Score | Rubric |
|---|---|
| 5 | Major event with wide, lasting consequences: a war starts or ends, a government falls, a disaster with many casualties, a decision that affects millions |
| 4 | Significant news with broad consequences that many people will follow |
| 3 | Solid news of general interest |
| 2 | Minor or niche: local events, routine announcements, small updates |
| 1 | Barely news |
Requirements:
- The model's score comes from the summarize call (§5.6). The rubric lives in the editable prompt.
- Coverage boost. A story covered by 3 or more sources gets +1, capped at 5. The app computes this; it is not asked of the model. It corrects for a model that underrates big stories.
- The score is recalculated whenever the story is regenerated or its source count changes.
- Override. The owner can set a story's score by hand in the story view. The app never changes an overridden score afterwards.
- Importance alone never hides a story. It narrows the RSS feed only through the feed controls you set (§5.10), and the web reader can always show everything.
- Used by: feed controls (§5.10), the
min_importancefeed parameter (§5.11), the importance filter in the reader (§5.13), and digest selection (§5.8). - The Activity screen shows the distribution of scores over the last 7 days, which makes a model that rates nearly everything 3 easy to spot.
5.8 Daily digest
Each morning, RSSFeed publishes one item that lists the previous day's most important stories.
- Schedule. Default 06:30 local time (
TZ, default Europe/Zagreb). Covers stories first reported since the previous digest. - Selection. The top 10 stories by importance, with ties broken by number of sources and then by recency. Only stories at importance 3 or above, or at the feed's minimum importance if that is higher, are eligible. Categories excluded in feed controls are left out. Daily and per-source limits do not apply, so an important story that a limit kept out of the feed can still lead the digest. If fewer than 3 stories qualify, no digest is published that day. Time, size, and the minimum importance are settings.
- Content. The title "Jutarnji pregled — 15. studenoga 2026.", a two-sentence opening written by the LLM, then each selected story's title and description exactly as already published, in importance order, each linking to the story's lead article. The digest adds no new text about individual stories.
- Grounding. The opening may use only facts from the selected titles and descriptions. The language guard (§5.6) applies.
- Fallback. If the LLM is unreachable, retry every 15 minutes for up to 6 hours, then publish without the opening. Everything else in the digest already exists, so it still arrives.
- Publishing. The digest is pinned at the top of the Feed screen until the next one replaces it, and is always available at
/feeds/digest.xml. Whether it also appears in the main feed, with the category Pregled, is a feed control (§5.10).
5.9 LLM connection
The app talks to any server that exposes the OpenAI-compatible API (LM Studio, Ollama, llama.cpp server, vLLM, LocalAI, and similar). Model choice, hardware, capacity, and availability are the owner's concern; the app only needs the endpoint and model names.
| Setting | Notes |
|---|---|
| Base URL | For example http://192.168.1.50:1234/v1 |
| API key | Optional. Most local servers ignore it |
| Triage model | Judges negatives and answers the same-event question in clustering. A small, fast model is enough |
| Summary model | Writes titles, descriptions, regenerations, and the digest opening. May be the same as the triage model |
| Embedding model | A multilingual embedding model, used for clustering (§5.5) |
| Concurrency | Default 1. Raise it if your server handles parallel requests |
| Request timeout | Default 120 seconds |
| Max article characters | Default 12,000 per article. Longer text is truncated to fit the context window |
Requirements:
- Uses
POST /v1/chat/completions,GET /v1/modelsto populate the model pickers, andPOST /v1/embeddingsfor clustering. - Test connection button: checks reachability, lists models, and runs one sample triage, one sample title and description, and one embedding. Reports latency and whether the server honors JSON-schema structured output.
- Structured output uses
response_formatwith a JSON schema where the server supports it, falls back to JSON mode, then to prompt-only with validation and one repair retry. Reasoning blocks (<think>…</think>) are stripped before parsing. - Circuit breaker. After repeated connection failures, for example while the server restarts, the queue pauses, the UI shows a banner, and the app probes periodically and resumes on its own. Nothing is lost and nothing is published unprocessed.
- Prompts are editable in Settings, with a reset-to-default, and are versioned. That covers triage, same-event confirmation, title and description, regeneration, and the digest opening. The format check (§5.6) applies whatever the prompt says.
- Playground. Paste an article URL and see the extracted text, the triage verdict, and the title, description, and importance score, under the current settings. Compare two models side by side, and see the similarity score between any two articles. This is how Croatian quality and the clustering thresholds get checked before the feed depends on them.
5.10 Feed controls
The owner decides how much goes into the RSS feed and what kind. All controls are in Settings → Feed.
Feed controls apply only to the RSS feed. Every published story stays in the web reader, where the "All stories" view also shows the ones left out and why.
Delivery mode
| Mode | How it works | Best for |
|---|---|---|
| Live (default) | A story enters the feed as soon as it is summarized, if it passes the controls below. Limits fill first come, first served | Following the news through the day |
| Editions | Stories are collected and released at set times, for example 07:00, 12:00, and 18:00. Each edition takes the most important stories since the previous one, up to its size | Reading at fixed times, and getting the best of each batch rather than the first to arrive |
The trade-off: Live mode cannot know whether a more important story will arrive later in the day, so once the daily limit is reached it lets in only breaking stories. Editions see the whole batch before choosing, at the cost of waiting until the next release.
Controls
| Control | What it does | Default |
|---|---|---|
| Stories per day | Live mode: the most stories that enter the feed in one day | 50 |
| Editions | Editions mode: release times and stories per edition. The screen shows the resulting daily total | 07:00, 12:00, 18:00, 15 stories each |
| Minimum importance | Stories scored below this stay out of the feed | 1 (everything) |
| Breaking level | Stories at or above this score enter immediately in either mode, ignoring limits and caps | 5 |
| Per-source limit | The most stories per day led by one source, so a prolific outlet cannot dominate | Off |
| Source overrides | For each source: its own minimum importance (higher or lower than the feed's) and its own daily maximum. Importance is one scale for all news, so specialist sources score lower than world news; this balances a busy general source against quiet specialist ones. The preview shows stories a day per source | None |
| Categories | For each category: include or exclude, and an optional daily maximum | All included, no maximum |
| Order | Always newest first by publication time (a story's earliest report), in the feed file and the reader. A story processed late (a backfill, a retry, after quiet hours) takes its place in time instead of jumping to the top | Fixed |
| Updated stories | Keep their original date, or move to the top when they gain sources or are regenerated | Keep original date |
| Include the digest | Whether the daily digest also appears in the main feed | On |
| Item content | Whether items include the source list and the lead image | Both on |
| Feed length | The most items in the feed file at any time | 100 |
| Time window | The oldest story the feed file includes | 48 hours |
How much you receive is set by the limits, not by feed length. Reader apps keep every item they have already fetched. Feed length and time window only decide what a reader sees when it first subscribes or after being offline. The amount of news you actually get is set by stories per day or editions, minimum importance, and the caps. Feed length should stay comfortably above a day's worth of stories, otherwise a reader that polls rarely can miss items.
Preview. While you change any control, the screen shows how many stories per day the new settings would have let into the feed over the last 7 days, and lists the ones they would have left out. It works like the dry run for negatives (§5.4).
How decisions are made and kept
- Each story's feed decision is made once: when it is summarized (Live) or at its edition (Editions). It is recorded with a reason: daily limit, edition full, below minimum importance, category excluded, category limit, or source limit.
- A story left out of the feed is re-evaluated if its importance rises, for example when it gains sources and becomes breaking.
- A story already in the feed is never removed from it. Changed settings apply to new decisions.
- Add to feed in the web reader puts any left-out story into the feed by hand.
5.11 Output feed
- All feeds live under one prefix,
/feeds/, so a single Pangolin rule can cover them (§9):GET /feeds/all.xmlreturns RSS 2.0GET /feeds/all.jsonreturns JSON Feed 1.1GET /feeds/digest.xmlreturns only the daily digests
- Each feed's self link is absolute, built from the public base URL (
RSSFEED_PUBLIC_URL) or, when that is unset, from the proxy's forwarded headers. - Each item is a story that feed controls (§5.10) admitted:
| RSS element | Content |
|---|---|
<title> |
The story's one-sentence Croatian title |
<description> |
The story's description, at most two sentences |
<content:encoded> |
The description, followed by the list of sources with each original headline linking to the article as published, when the source list is enabled |
<link> |
The lead article, as published |
<pubDate> |
When the story was first reported, or when it was last updated if the updated-stories control says so |
<category> |
The story's category, plus the importance as a second category with its own domain attribute |
<enclosure> |
The lead image, when enabled and available |
<guid> |
Stable per story |
- New sources and regenerations update the item under the same GUID, without creating a duplicate in the reader.
- Query parameters narrow the feed further for one subscription:
limit(default is the feed length control, max 500),category,source(stories with at least one article from that source), andmin_importance. They never add stories that feed controls left out. The digest item is always included regardless ofmin_importance. - Supports
ETagandIf-None-Matchso readers can poll cheaply. - Optional feed token: when set, the feeds are served only under
/feeds/<token>/, for example/feeds/<token>/all.xml. It adds a secret path for when the feeds are opened in Pangolin without a login, which RSS reader apps usually need.
5.12 Categories
A fixed, editable list, assigned by the summarize call: Politika, Svijet, Gospodarstvo, Tehnologija, Znanost, Sport, Kultura, Zdravlje, Crna kronika, Ostalo. The digest uses its own category, Pregled. Category names are story content, so they stay in Croatian in both the feed and the English UI. Categories drive the per-category feed controls (§5.10), filtering in the reader, and the per-category feeds planned for v1.
5.13 Web UI
The interface is in English. Story content (titles, descriptions, categories, and the digest) is in Croatian.
Dark mode by default. Seven screens:
| Screen | Purpose |
|---|---|
| Feed (home) | Today's digest pinned at the top, then stories: title, description, a small 1-to-5 importance marker, source count, relative time, category. Toggle between "In feed" and "All stories"; left-out stories show why, with Add to feed. In Editions mode the list is grouped by edition. Newest first by publication time; filter by importance, source, and category, full-text search, mark as read, bookmark |
| Story | Title and description, every source with its original headline and a link to the article as published, importance with a manual override, feed status, Regenerate, Split for each article, and Merge with another story |
| Filtered | Rejected articles with the rule and reason. Rescue button. Filter by rule |
| Sources | Add, edit, pause, remove. Health per source. OPML import and export |
| Rules | Negatives and exceptions. Toggle, scope, hit counts, dry run |
| Settings | Feed controls with preview; LLM connection and test; prompts; glossary; clustering window, thresholds, and regeneration triggers; coverage boost; digest time, size, and minimum importance; schedule; retention; feed URLs ready to copy; feed token; playground |
| Activity | Queue depth per stage; recent failures with errors; LLM latency and throughput; importance distribution; share of articles grouped into multi-source stories; stories per day in the feed and left out, by reason; format-check retries; last run times, including the last digest and edition |
Design requirements:
- Reading comes first. Body text at 17 to 18 px, line length capped at about 70 characters, generous line height, and contrast meeting WCAG AA on the dark theme.
- One accent color, no decorative chrome, no deep nested menus. Every screen reachable in one click from a persistent navigation.
- Works on a phone. Installable as a PWA.
- Keyboard shortcuts in the reader:
jandkto move,Enterto open the story,oto open the original,bto bookmark,ato toggle "All stories",/to search. - Clear states for empty, loading, error, and "LLM offline, 37 articles waiting".
- A light theme is optional and follows the system preference only if the user turns that on.
6. Additions beyond the original brief
The six additions recommended in v0.1 are all part of the MVP:
| # | Addition | Specified in |
|---|---|---|
| 1 | Filter transparency: Filtered view, Rescue, dry run, per-rule hit counts | §5.4, §5.13 |
| 2 | Corrections teach the filter: feedback examples and "always keep" exceptions | §5.4 |
| 3 | Two-stage LLM pipeline with a free keyword pre-filter | §4, §5.4 |
| 4 | Playground, model comparison, and the Croatian language guard | §5.6, §5.9 |
| 5 | Story clustering across sources and languages | §5.5 |
| 6 | Importance scoring and the daily digest | §5.7, §5.8 |
Added in v0.3 at the owner's request: feed controls (§5.10).
Still deferred:
| Addition | Why it matters | Phase |
|---|---|---|
Per-category feeds, such as /feeds/tehnologija.xml |
Lets a reader app organize by topic, each feed with its own controls | v1 |
| Reprocessing the last N days | Regenerate titles and descriptions in the background after switching to a better model | v1 |
| Push alerts for breaking stories | Notification through ntfy, Telegram, or a Home Assistant webhook | v2 |
| Feed profiles | Several output feeds, each with its own sources, negatives, and controls, for example work and personal | v2 |
| Archive search by meaning | "What happened with X last month". Cheap because MVP already stores embeddings | v2 |
| Audio digest | The daily digest read aloud by a local text-to-speech model, published as a podcast feed | Later |
7. Architecture and stack
Same conventions as EPGMaster, so the Dockerfile, release workflow, and project layout can be adapted instead of invented.
| Layer | Choice |
|---|---|
| Runtime | Node 24, TypeScript run through Node's native type stripping |
| Layout | npm workspaces: server/ and web/ |
| Backend | Express 5, single process, single port |
| Database | SQLite through the built-in node:sqlite, WAL mode, stored on the /data volume |
| Frontend | React 19 and Vite 7, built to static files and served by Express |
| Feed parsing and writing | fast-xml-parser |
| Extraction | @mozilla/readability with linkedom |
| LLM client | A thin fetch-based client for the three endpoints in §5.9. No SDK dependency, which also makes it easier to tolerate quirks of local servers |
| Similarity search | Embeddings stored as BLOBs in SQLite and compared in memory by cosine similarity. About 1,000 vectors in a 48-hour window, so no vector database |
| Scheduling and queue | In-process scheduler for polling, editions, and the digest. The job queue is a SQLite table, so there is no Redis or second container |
One container, one volume. Everything mutable lives in /data: the database, logs, and exports.
Data model (main tables)
| Table | Key fields |
|---|---|
source |
feed URL, site URL, name, enabled, poll interval, full-text flag, ETag, last-modified, last fetch, last error, failure count |
article |
source, GUID, canonical URL, URL hash, original headline and excerpt, extracted text, partial flag, author, image URL, detected language, published and fetched times, state, story, embedding |
verdict |
article, decision, rule, reason (English), confidence, model, prompt version, overridden-by-user flag |
story |
GUID, lead article, current version, model importance, coverage boost, importance override, source count, first and last reported, state, regeneration count, feed status, feed reason, entered-feed time, edition |
story_version |
story, version, Croatian title, Croatian description, category, importance, model, prompt version, format retries, tokens in and out, duration |
edition |
scheduled time, released time, size, stories |
digest |
date, selected stories, opening text, model, kind (with opening or fallback), state |
rule |
kind (AI, keyword, exception), text, enabled, source scope, hit count, last hit |
example |
kind (filter or clustering), article or story pair, expected decision, note |
job |
subject (article, story, edition, or digest), stage, attempts, next run time, lock time, last error |
setting |
key, value. Holds feed controls and every other setting |
HTTP surface
- Feeds: everything under
/feeds/(§5.11), plus/api/health. The feeds move under/feeds/<token>/when a feed token is set. - UI and API: REST endpoints for sources, rules, articles, stories, editions, digests, settings, the feed preview, jobs, and the playground. The app has no login of its own; Pangolin controls who reaches them (§9).
8. Delivery
- Image:
git.mariosem.es/msemes/rssfeed, tagged:latestand:<short-sha>. - CI:
.gitea/workflows/release.ymladapted from EPGMaster. Self-hosted runner, mirroredactions/checkoutanddocker/login-action, theREGISTRY_TOKENrepo secret, build on pull request, build plus Trivy scan plus push onmain. - Dockerfile: multi-stage on
node:24-slim. Build stage installs dependencies and builds the web UI. Runtime stage runs as the unprivilegednodeuser, exposes one port, declares/dataas a volume, and has a health check against/api/health. - Compose: a
docker-compose.ymlin the repo with the image, a named volume, and the environment variables below. The port only needs to be reachable by Pangolin, never published straight to the internet.
| Variable | Purpose | Default |
|---|---|---|
RSSFEED_PORT |
HTTP port | 34500 |
RSSFEED_DATA_DIR |
Data directory | /data |
TZ |
Timezone for schedules, editions, the digest, and displayed times | Europe/Zagreb |
RSSFEED_LLM_BASE_URL |
Initial LLM endpoint, editable later in the UI | none |
RSSFEED_LLM_API_KEY |
Optional key for the LLM endpoint | none |
RSSFEED_PUBLIC_URL |
Public base URL as reached through Pangolin, for example https://rss.example.com. Used for the feeds' self links |
Derived from forwarded headers |
RSSFEED_FEED_TOKEN |
When set, the feeds are served only under the tokenized path | none |
RSSFEED_ALLOW_PRIVATE_URLS |
Allow source URLs that resolve to private addresses | off |
Note: the LLM endpoint is itself a private address. The private-address block applies to source and article URLs, not to the configured LLM base URL.
9. Non-functional requirements
LLM capacity. Not a design constraint: the owner provides an LLM server with enough capacity. The pipeline still avoids unnecessary calls (§4), which keeps the feed fresh, and the Activity screen shows latency and queue depth so a backlog is visible.
Freshness. In Live mode, with the default 15-minute poll and an idle queue, a new story should appear in the feed within about 30 minutes of publication. An article joining an existing story shows up as an extra source within minutes, since it needs only triage and an embedding. In Editions mode, stories appear at the next release, except breaking ones.
Schedules. Editions are released within 5 minutes of their set time. At the default schedule, the digest is published by 07:00 when the LLM is reachable, and without its opening by 12:30 otherwise.
Back-pressure. The queue processes newest first. When a backlog builds, for example after an outage, recent news is not stuck behind stale items, and items past the maximum age are skipped.
Resilience. Restart-safe at every stage. An LLM outage, a broken feed, or a failed extraction never loses an article and never stops the other sources.
Retention. Published stories and their articles are kept 90 days by default, rejected articles 14 days, and bookmarks indefinitely. Extracted full text is pruned after 30 days. Embeddings are kept as long as their articles, so the v2 archive search can use them. All configurable.
Security.
- Access control is Pangolin's job. The app is reached from outside only through Pangolin, which handles authentication and TLS. The app has no user accounts or login of its own. It reads the proxy's forwarded headers to build correct links when no public base URL is set.
- RSS reader apps cannot complete a browser login. Keeping every feed under
/feeds/means one Pangolin rule can treat the feeds differently from the UI, and the optional feed token adds a secret path if the feeds are opened without a login. - Source and article URLs are checked against private and loopback ranges to prevent server-side request forgery, with the opt-out variable above.
- Article text is untrusted input to the LLM. It is passed as clearly delimited data, the model has no tools and can only return a fixed JSON schema, and the worst outcome of a prompt-injection attempt is one wrong verdict, one wrong grouping, or one bad title and description.
- Everything from feeds and from the LLM is rendered as text or passed through an HTML sanitizer. No raw HTML from either reaches the browser or the output feed.
- The container runs as a non-root user. No secrets are baked into the image.
Observability. Structured logs to stdout, the Activity screen from §5.13, and /api/health reporting app version, database status, queue depth, LLM reachability, and the times of the last edition and digest.
10. Success criteria
Targets to check after two weeks of real use:
| Measure | Target |
|---|---|
| Unwanted articles reaching the feed | Under 5% of published stories, judged by "should have been filtered" marks |
| Wanted articles wrongly filtered | Under 2% of rejected articles, judged by rescues |
| The same event appearing as two separate stories | Fewer than 3 a day, judged by Merge actions |
| Different events wrongly grouped | Under 2% of multi-source stories, judged by Split actions |
| Importance scores you override | Under 10% of the stories you open |
| Stories entering the feed each day | Within the limits you set, apart from breaking stories |
| Digest published | Every day with at least 3 eligible stories |
| Titles longer than one sentence, or descriptions longer than two, in the feed | Zero |
| Stories in the feed that are not in Croatian | Zero |
| Stories in the feed without an AI title and description | Zero |
| Articles lost during an LLM outage | Zero |
| Median time from publication to feed, Live mode, idle queue | Under 30 minutes |
| Time spent maintaining it after the first two weeks | A few minutes a week |
11. Milestones
| Milestone | Scope | Result |
|---|---|---|
| M0 — Skeleton | Repo, workspaces, database migrations, health endpoint, Dockerfile, release workflow | An empty app that builds and publishes its image |
| M1 — Merge | Sources screen, fetcher, parser, deduplication, raw merged /feeds/all.xml |
A working feed merger, no AI yet |
| M2 — AI pipeline | Extraction; LLM client with test connection; playground; keyword and AI rules with triage; title and description with the importance score; format check; language guard; circuit breaker | The Croatian, filtered feed, one item per article |
| M3 — Stories and feed controls | Embeddings, clustering with same-event confirmation, lead selection, regeneration, coverage boost; feed controls in Live and Editions modes with every limit and cap; min_importance; daily digest with fallback and /feeds/digest.xml |
One item per event, at the volume you choose, with a morning digest |
| M4 — Reader and trust | Feed screen with the pinned digest, "All stories" view, and Add to feed; Story screen with Split, Merge, override, and Regenerate; feed-controls preview; Filtered screen with Rescue; dry run; feedback examples; Activity; PWA | Feature-complete MVP |
| M5 — v1 | Per-category feeds, reprocessing | The remaining v1 items from §6 |
Each milestone is usable on its own through the output feed. M2 is where the Croatian quality of the chosen model gets checked. M3 is where the clustering thresholds get calibrated on real data.
12. Risks
| Risk | Mitigation |
|---|---|
| Croatian quality of the model. Wrong dialect forms, awkward phrasing, or mistranslated names | Playground and model comparison before going live, the language guard, the glossary, an explicit standard-Croatian instruction, and the Regenerate action |
| A title or description states something the articles did not | Grounded prompt, low temperature, a format too short to hold much beyond the core facts, the original always one click away, and an "AI summary" label on every story |
| Two sentences leave out context that matters | The description sticks to who, what, where, and when; the original article, unchanged, is one tap away; the story view lists every source |
| The model ignores the length limit | The format check retries once and then trims to the first sentence of the title and the first two of the description; retries are counted on Activity, so a model that keeps failing is visible |
| Over-filtering hides something important | Reject only above the confidence threshold, the Filtered view, Rescue, dry run, and per-rule hit counts that expose a rule firing too often |
| Feed limits keep out an important story | The breaking level bypasses every limit, Editions mode picks by importance, left-out stories are re-evaluated when their importance rises, the digest ignores daily limits, and the "All stories" view and Add to feed are always available |
| Clustering merges different events or misses duplicates | Two thresholds with an LLM check in between, an explicit definition of "same event", the 48-hour window, Split and Merge, and corrections stored as examples |
| A regenerated description mixes up facts from different reports | It states disagreements instead of choosing, every source is linked, and regenerations are capped per story |
| Importance scores bunch up and stop meaning anything | A concrete neutral rubric in the prompt, the coverage boost, the score distribution on Activity, and manual override. This matters because feed limits depend on the score |
| The digest opening states something not in the stories | It may use only the selected titles and descriptions, the language guard applies, and everything after the opening is text already published |
| Sites block extraction or sit behind paywalls | Fall back to the feed excerpt and label the story partial |
| Feeds rate-limit or ban the fetcher | Conditional requests, per-host concurrency of one, backoff, honest User-Agent |
| Prompt injection through article text | No tools, fixed output schema, delimited input, sanitized output |
| RSS reader apps cannot pass Pangolin's login | Every feed under one /feeds/ prefix so a single Pangolin rule covers them, with the optional feed token as a secret path |
| Copyright | Personal use, the article never copied or changed, only RSSFeed's own two-sentence text plus links to the originals |
13. Assumptions
- The LLM server speaks the OpenAI-compatible API, is reachable from the container over the LAN, and provides chat models and a multilingual embedding model with enough capacity for the volume. Confirmed by the owner in v0.3.
- One merged feed plus the digest feed is enough for MVP.
- Outside access goes through Pangolin, which the owner configures. Confirmed in v0.5.
- Planning volume is a few dozen sources and a few hundred articles a day. Sources are added in the UI, and nothing in the design depends on an exact number.
14. Open questions
None. Every question has been answered.
Resolved
| Decision | Resolution | Version |
|---|---|---|
| Outside access | Through Pangolin, configured by the owner. Pangolin handles authentication and TLS; the app has no login of its own | v0.5 |
| Croatian sources | Get RSSFeed's own title and description, like every other source | v0.5 |
| Source list | Added in the UI after deployment, not defined in the PRD | v0.5 |
| Product name | RSSFeed | v0.5 |
| UI language | English. Story content stays Croatian | v0.4 |
| Title and description length | One-sentence title, description of at most two sentences, enforced by a format check | v0.4 |
| Changes to the original article | None. The article is only linked as published; RSSFeed adds its own title and description | v0.4 |
| LLM server, model, hardware, availability, embedding support | The owner's concern. The PRD treats the LLM as capable and available | v0.3 |
| Importance perspective | Neutral: scale of consequences and number of people affected, regardless of country | v0.3 |
| Feed size and composition | Owner-controlled through feed controls (§5.10) | v0.3 |
| Digest time and size | Settings, defaulting to 06:30 and 10 stories | v0.3 |
| Scope of the six recommended additions | All in MVP | v0.2 |