# RSSFeed — Product Requirements Document
**Version:** 0.5 (Ready for build)
**Date:** 2026-11-15
**Status:** Ready for build kickoff — no open questions remain (§14)
**Repository:** `https://git.mariosem.es/msemes/RSSFeed.git`
**Deliverable:** one Docker image, `git.mariosem.es/msemes/rssfeed`, built and published by Gitea Actions
**Reference project for stack and CI conventions:** EPGMaster (`git.mariosem.es/msemes/EPGMaster`)
> **v0.5 changes:** all open questions resolved. Outside access goes through **Pangolin**, which handles authentication and TLS, so the built-in admin token is removed. All feeds move under one `/feeds/` prefix so a single Pangolin rule can cover them, and a public base URL setting keeps feed links correct behind the proxy (§5.11, §8, §9). Stories from Croatian sources get RSSFeed's own title and description like all others. Sources are added in the UI, and the name RSSFeed is confirmed.
>
> **v0.4 changes:** (1) Every story gets a fixed format: a **one-sentence title** and a **description of at most two sentences** (§5.6). Key points and the summary-length setting are removed, and the digest reuses each story's title and description as published (§5.8). (2) **The original article is never edited, rewritten, or republished.** RSSFeed adds only its own title and description and links to the article as published. (3) The UI language is **English**. Filter reasons are now in English too, since they are part of the UI. Story content stays Croatian.
>
> **v0.3 changes:** feed controls (§5.10); importance judged neutrally; the LLM treated as given.
>
> **v0.2 changes:** story clustering, importance, and the daily digest moved into MVP; feed items became stories.
---
## 1. Summary
RSSFeed is a self-hosted, single-user news pipeline. You add RSS/Atom sources; it fetches them and has a **locally hosted LLM** read every article. It throws away what you have said you do not want, groups articles about the same event into one story, and writes a **short Croatian title and description** for each story. The result is published two ways: as **one merged RSS feed** you can subscribe to from any reader, and as a clean dark-mode web reader built into the app.
Three things make it different from a plain feed merger:
1. **Own title and description.** The LLM reads the full article, not just the feed excerpt, and writes a one-sentence title that says what happened plus a description of at most two sentences. The article itself is never changed: every item links to it exactly as published.
2. **Everything in Croatian.** Sources can be in any language; the output is always Croatian.
3. **Negatives.** You describe in plain language what you never want to see ("celebrity gossip", "horoscopes", "transfer rumours"). The LLM judges each article against those rules, and only qualified articles reach the feed.
On top of those:
- **One story, not five copies.** When several outlets report the same event, the feed shows it once, with links to every source.
- **Importance.** Every story is rated from 1 to 5 on a neutral scale.
- **You set the volume.** A daily limit, a minimum importance, caps per source and per category, or a few editions a day instead of a constant stream.
- **Daily digest.** Each morning, one item lists the previous day's most important stories.
Everything runs on your own hardware. No article text, rule, or reading habit leaves your network.
---
## 2. Goals and non-goals
### Goals
| # | Goal | Phase |
|---|------|-------|
| G1 | Add, edit, pause, and remove RSS/Atom sources from the UI | MVP |
| G2 | Merge all sources into a single valid RSS 2.0 feed at a stable URL | MVP |
| G3 | Give each story a one-sentence title and a description of at most two sentences, written by a local OpenAI-compatible LLM from the full article text | MVP |
| G4 | Write every title and description in Croatian regardless of source language | MVP |
| G5 | Filter articles against user-defined negatives before they reach the feed | MVP |
| G6 | Show what was filtered and why, and let the user overrule it | MVP |
| G7 | Simple, clean, dark-mode web UI in English that is comfortable to read | MVP |
| G8 | Ship as a single Docker image built by Gitea Actions | MVP |
| G9 | Group articles about the same event from different sources into one story | MVP |
| G10 | Rate every story's importance on a neutral scale | MVP |
| G11 | Publish a daily digest of the most important stories | MVP |
| G12 | Let the owner control how many stories the feed gets, and of what kind | MVP |
| G13 | Per-topic feeds and reprocessing after a model change | v1 |
### Non-goals
- **Not an article editor.** The original article is never edited, rewritten, shortened, or republished. RSSFeed adds its own title and description, and always links to the article as published.
- **Not a public republishing service.** The feed carries RSSFeed's titles and descriptions plus links to the originals. It never carries article text, and it is meant for personal use.
- **Not multi-user.** One owner, one set of sources and rules. No accounts, roles, or sharing.
- **Not a cloud-AI product.** No calls to hosted LLM APIs. The LLM endpoint is whatever OpenAI-compatible server you run locally.
- **Not a paywall or anti-bot bypass.** If a site blocks extraction, the title and description are written from the feed excerpt and the story is marked as partial. No headless browser in MVP.
- **Not a fact-checker.** When sources in a story disagree, the description says so. It does not decide which source is right.
- **Not a full RSS reader replacement.** The built-in reader covers reading, searching, and bookmarking. Third-party reader apps get the content through the output feed; there is no reader-sync API.
---
## 3. User
One persona: **the owner**. Runs a homelab with Docker and Gitea, already hosts a local LLM, reads news from a mix of Croatian and foreign outlets, and is tired of wading through filler to find what matters. Wants to set it up once, tune the negatives and feed controls over a week or two, and then just read.
### User stories
- As the owner, I want to paste a site or feed URL and have the feed found for me, so adding a source takes seconds.
- As the owner, I want one feed URL that contains everything, so my phone reader needs a single subscription.
- As the owner, I want each item to be a one-sentence Croatian title and at most two sentences of description, so I can decide in a few seconds whether to open the original.
- As the owner, I want the original article left exactly as published, so what I open is the source's own work.
- As the owner, I want to write negatives in plain language, so I do not have to maintain keyword lists.
- As the owner, I want to see what got filtered and which rule caused it, so I can trust the filter and fix it when it is wrong.
- As the owner, I want to test a new negative against recent articles before saving it, so I know what it will remove.
- As the owner, I want one item per event, listing every outlet that covered it, so the feed does not repeat itself.
- As the owner, I want to set how many stories my feed gets per day and what kinds, so the feed fits the time I have.
- As the owner, I want to see what my feed settings would have let through last week before I save them, so I can tune them without guessing.
- As the owner, I want a morning overview of yesterday's top stories, so I have one thing to read when I have time for only one.
- As the owner, I want nothing lost when the LLM server restarts, so articles wait and get processed when it comes back.
---
## 4. How it works
Each article moves through a fixed pipeline. Every stage is persisted, so a restart or an LLM outage resumes where it stopped.
```
Sources ─► Fetch ─► Dedupe ─► Extract ─► Keyword ─► Triage ─► Cluster ─► Summarize ─► Web reader
rules (LLM) (embed) (LLM) │
│ │ │ ▼
└────┬─────┘ │ Feed controls
▼ ▼ │
Filtered view Same event as an ▼
rule, reason, existing story: RSS feed
Rescue button added as a source
Daily: stories from the last 24 h ─► top 10 by importance ─► Digest (LLM) ─► Web reader and RSS
```
"Summarize" produces the story's title and description (§5.6). Every summarized story appears in the web reader. **Feed controls** (§5.10) then decide which of them enter the RSS feed, and when.
**Why the work is split into stages.** Each stage keeps expensive calls away from articles that do not need them, which keeps the feed fresh:
- Keyword rules cost nothing and run first.
- Triage reads only the title, excerpt, and opening of the text, so its prompt is roughly a tenth the size of a summarize prompt. Rejected articles never reach the summarize call.
- Clustering uses an embedding, which is far cheaper than a chat call. An article that joins an existing story is not summarized on its own; it is listed as an additional source.
- Importance comes out of the summarize call, so it costs nothing extra. The digest is one small call a day.
### Article states
| State | Meaning |
|-------|---------|
| `new` | Fetched from a feed, not yet processed |
| `extracted` | Full text obtained, or fallen back to the feed excerpt |
| `rejected` | Matched a negative. Visible in the Filtered view, never in the feed |
| `kept` | Passed triage, waiting to be clustered |
| `grouped` | Assigned to a story. The story carries the title and description and is what appears in the feed |
| `failed` | A stage failed after all retries. Visible in Activity with the error |
### Story states
| State | Meaning |
|-------|---------|
| `pending` | Created from its first article, waiting for its title and description |
| `published` | Title and description written; visible in the web reader |
| `updating` | Gained sources and has a refreshed title and description queued. The current version stays visible meanwhile |
| `failed` | Summarize failed after all retries. Visible in Activity with the error |
Separately, every published story has a **feed status**: in the feed, waiting for the next edition, or left out with a reason (§5.10).
Two rules hold at every stage: **no article is dropped silently**, and **no story is published without a Croatian title and description**. If the LLM is unreachable, articles and stories wait in their current state.
---
## 5. Functional requirements
### 5.1 Sources
- Add a source by feed URL or site URL. For a site URL, discover the feed from `` tags and common paths, and let the user pick if there are several.
- Supported formats: RSS 2.0, RSS 1.0, Atom 1.0, JSON Feed.
- Per-source settings: display name, enabled, poll interval (default 15 minutes), "fetch full article" on or off (default on), language hint (default auto-detect).
- Per-source health: last successful fetch, last error, consecutive failures, articles in the last 24 hours, and the share of them that were filtered.
- A source that fails repeatedly backs off exponentially up to 6 hours and shows a warning. It is never disabled automatically.
- OPML import and export.
### 5.2 Fetching and deduplication
- Polite fetching: conditional GET with `ETag` and `Last-Modified`, honor `Retry-After`, 20-second timeout, an identifiable User-Agent, and at most one concurrent request per host.
- Deduplicate on canonical URL (tracking parameters stripped), then on feed GUID, then on a normalized title hash within a 48-hour window.
- Deduplication catches the same article seen twice. Recognizing different articles about the same event is clustering (§5.5).
- On first add, a source imports only items from the last 24 hours, so a new source does not flood the queue with its whole backlog. A source added on a quiet day still gets its 10 newest items from within the maximum age, so it never looks empty.
- Removing a source also forgets its article URLs, so adding the same feed again imports its current items.
- Quiet hours (optional, e.g. 23:00–05:00 local time): no scheduled fetching and no background processing, so no AI calls. New items wait in the feeds and are fetched and processed when the window ends. Editions, digest creation and cleanup still run on time (they cost no AI), and a digest due in the window gets its opening when it ends. Things started by hand (Fetch now, Playground, connection test) still run. A banner shows when quiet hours are on.
- Items older than a configurable maximum age (default 72 hours) at fetch time are stored but skipped by the pipeline.
### 5.3 Full-text extraction
- Fetch the article page and extract the main content with a Readability-style extractor.
- If extraction fails, returns too little text, or the page is blocked, fall back to the feed's own content and mark the article `partial`. A partial article is still processed, and the UI labels its story as based on the excerpt only.
- **Extracted text is input to the LLM and nothing else.** It is stored locally for triage, clustering, and regeneration, but never shown in the reader, served in a feed, or modified. Reading the article always means opening the original.
- Also capture the lead image URL, author, and publication date where available.
### 5.4 Negatives (the filter)
Two kinds of rule, both managed on one screen:
| Kind | How it works | Cost |
|------|--------------|------|
| **AI rule** | A plain-language description, such as "Celebrity gossip, showbiz, and influencer news with no public-interest angle". The LLM judges each article against all active AI rules. | One short LLM call per article |
| **Keyword rule** | Words, phrases, or a regex matched against title and feed categories. Also covers "skip this category from this source". | Free, runs before the LLM |
Requirements:
- Each rule has: text, enabled toggle, optional scope (all sources or selected ones), and a hit counter with the date of the last hit.
- Rules can be written in any language. The model applies them to articles in any language.
- Keyword rules run first. An article they reject never costs an LLM call.
- The triage call returns structured JSON: decision, the matching rule, a one-sentence reason in English for the Filtered view, and a confidence from 0 to 1.
- **An article is rejected only when confidence is at or above the threshold** (default 0.7, adjustable as a "strictness" setting). When the model is unsure, the article is kept. A wrongly hidden important story costs more than one piece of filler getting through.
- **Exceptions ("always keep").** Optional positive rules that override negatives, for example "Anything about Croatia's national football team" alongside a negative about sports.
- **Dry run.** Before saving a new or edited rule, run it against the articles published within the feed's time window (at least 24 hours, up to 400) and show what it would remove. Enabled "always keep" rules are honoured, so the result is what the filter would really do.
- **Apply to the current feed.** Saving a negative rule also removes its matches from the feed right away (on by default; one checkbox). A finished dry run of the same rule is reused, so nothing is judged twice. A saved rule's dry run can be applied with one click. Removed articles appear under Filtered with the rule and reason, a story left without articles disappears, and **Put back all** restores them without recording a correction.
- **Filtered view.** Every rejected article is listed with its rule and reason. **Rescue** sends it on to clustering and summarize, like any kept article.
- **Feedback as examples.** A rescued article, or a published story the user marks "should have been filtered", is stored as a labeled example. The most recent examples (default 10) are included in the triage prompt, so the filter adapts to corrections without retraining anything.
- Rejected articles never join a story.
- Rejected articles are kept for 14 days for review, then pruned.
Negatives decide what is worth processing at all. Feed controls (§5.10) decide how much of what survives goes into the RSS feed.
### 5.5 Story clustering
A **story** is one news event as reported by one or more sources. Every kept article belongs to exactly one story. Most stories have a single source. Clustering is what lets the feed show an event once when several outlets cover it.
**Matching an article to a story**
1. Embed the article's original title and the first ~1,000 characters of its text with a **multilingual** embedding model, so a Croatian and an English article about the same event can match.
2. Compare the embedding against the articles of every story that started within the clustering window (default 48 hours). At the expected volume that is about 1,000 vectors, compared in memory.
3. Decide by the best similarity score:
| Similarity | Decision |
|------------|----------|
| At or above the merge threshold | Join that story |
| Between the review and merge thresholds | Ask the triage model a short yes-or-no question: is this the same event? Join if yes |
| Below the review threshold | Start a new story |
4. **"Same event" means one specific occurrence:** the same incident, decision, match, or announcement. Two articles on the same topic are different stories, for example two separate developments in the same war on the same day. This definition is written into the confirmation prompt.
The thresholds depend on the embedding model. Defaults are calibrated on real data during milestone M3, and the playground (§5.9) shows the similarity between any two articles so they can be tuned.
**What a story shows**
- One Croatian title and description (§5.6); every source with its original headline, unchanged, and a link to the article as published; and when the story was first and last reported.
- The story's main link points to its **lead article**. By default the lead is the first non-partial article, preferring Croatian-language sources so the original is readable without translation. A setting turns off the Croatian preference.
**Keeping the title and description current**
- An article that joins an existing story does not get its own title and description.
- The story's title and description are regenerated from up to 4 member articles, truncated to fit the context window, when:
- the story reaches 3 sources, or
- a new member arrives more than 6 hours after the current version was written, which keeps developing stories up to date.
- At most 3 regenerations per story. The source count, the 6-hour gap, and the cap are all settings.
- Regenerated text keeps the same format limits. Where member articles disagree on a key fact, the description says so (for example "Izvori se razlikuju oko broja žrtava.") instead of picking one.
- The story keeps its feed GUID through every regeneration and every new source, so readers update the item instead of showing a new one.
**Corrections**
- **Split** removes an article from a story and makes it a story of its own.
- **Merge** joins two stories that clustering kept apart.
- Both are stored as examples for the confirmation prompt, the same way filter corrections are (§5.4).
### 5.6 Title and description
One LLM call per new story produces structured JSON:
| Field | Description |
|-------|-------------|
| `title_hr` | **Exactly one sentence** in Croatian that says what happened. Written as a headline, with no full stop at the end. This is RSSFeed's own title; the article's headline is left as published and shown in the source list |
| `description_hr` | **At most two sentences** in Croatian with the core facts: who, what, where, when, and the key number if there is one |
| `category` | One value from a fixed list (§5.12) |
| `importance` | Integer from 1 to 5, scored against the rubric in §5.7 |
| `source_lang` | Detected language of the lead article |
Requirements:
- **The article itself is never changed.** The title and description are RSSFeed's own text, shown next to a link to the article exactly as published. The article is never rewritten, shortened, or re-hosted.
- **The format is fixed and checked.** After generation, the app verifies that the title is one sentence and the description is no more than two. The sentence counter recognizes Croatian ordinal dates ("15. studenoga"), numbers, and common abbreviations ("npr.", "tj.", "dr."), so it does not count them as sentence ends. On a violation, the call is retried once with a stricter prompt; if it still fails, the app keeps the first sentence of the title and the first two of the description.
- The prompt requires **standard Croatian** (hrvatski standardni jezik, ijekavica) and forbids Serbian or Bosnian forms. Models commonly blur these, so this is a stated rule and a checked one, not an assumption.
- **Language guard.** After generation, verify the output is Croatian with a cheap heuristic: diacritics, stop-word profile, and the absence of Cyrillic and of typical ekavian forms. On failure, retry once with a stricter prompt, then mark the story `failed` so it never reaches the feed in the wrong language. The guard applies to every Croatian text the app produces, including regenerations and the digest opening.
- **Grounding.** The prompt allows only facts present in the provided articles. Names and numbers are carried over exactly. No commentary, no speculation, and no clickbait: the title states what happened rather than teasing it.
- Stories from Croatian-language sources skip translation but get their own title and description like every other story. A source's own headline and excerpt are never used as the feed text.
- A user **glossary** of fixed translations and of terms never to translate (names of institutions, products, clubs).
- Every version is stored with the model, prompt version, token counts, and duration. This makes it possible to **reprocess** the last N days after changing model or prompt (v1).
- A **Regenerate** action on any story re-runs its title and description.
### 5.7 Importance
Every story gets an importance score from 1 to 5, **judged neutrally**: by the scale of the consequences and the number of people affected, regardless of country or region. A Croatian story and a foreign story of the same scale get the same score.
| Score | Rubric |
|-------|--------|
| 5 | Major event with wide, lasting consequences: a war starts or ends, a government falls, a disaster with many casualties, a decision that affects millions |
| 4 | Significant news with broad consequences that many people will follow |
| 3 | Solid news of general interest |
| 2 | Minor or niche: local events, routine announcements, small updates |
| 1 | Barely news |
Requirements:
- The model's score comes from the summarize call (§5.6). The rubric lives in the editable prompt.
- **Coverage boost.** A story covered by 3 or more sources gets +1, capped at 5. The app computes this; it is not asked of the model. It corrects for a model that underrates big stories.
- The score is recalculated whenever the story is regenerated or its source count changes.
- **Override.** The owner can set a story's score by hand in the story view. The app never changes an overridden score afterwards.
- **Importance alone never hides a story.** It narrows the RSS feed only through the feed controls you set (§5.10), and the web reader can always show everything.
- Used by: feed controls (§5.10), the `min_importance` feed parameter (§5.11), the importance filter in the reader (§5.13), and digest selection (§5.8).
- The Activity screen shows the distribution of scores over the last 7 days, which makes a model that rates nearly everything 3 easy to spot.
### 5.8 Daily digest
Each morning, RSSFeed publishes one item that lists the previous day's most important stories.
- **Schedule.** Default 06:30 local time (`TZ`, default Europe/Zagreb). Covers stories first reported since the previous digest.
- **Selection.** The top 10 stories by importance, with ties broken by number of sources and then by recency. Only stories at importance 3 or above, or at the feed's minimum importance if that is higher, are eligible. Categories excluded in feed controls are left out. Daily and per-source limits do not apply, so an important story that a limit kept out of the feed can still lead the digest. If fewer than 3 stories qualify, no digest is published that day. Time, size, and the minimum importance are settings.
- **Content.** The title "Jutarnji pregled — 15. studenoga 2026.", a two-sentence opening written by the LLM, then each selected story's title and description exactly as already published, in importance order, each linking to the story's lead article. The digest adds no new text about individual stories.
- **Grounding.** The opening may use only facts from the selected titles and descriptions. The language guard (§5.6) applies.
- **Fallback.** If the LLM is unreachable, retry every 15 minutes for up to 6 hours, then publish without the opening. Everything else in the digest already exists, so it still arrives.
- **Publishing.** The digest is pinned at the top of the Feed screen until the next one replaces it, and is always available at `/feeds/digest.xml`. Whether it also appears in the main feed, with the category Pregled, is a feed control (§5.10).
### 5.9 LLM connection
The app talks to any server that exposes the OpenAI-compatible API (LM Studio, Ollama, llama.cpp server, vLLM, LocalAI, and similar). Model choice, hardware, capacity, and availability are the owner's concern; the app only needs the endpoint and model names.
| Setting | Notes |
|---------|-------|
| Base URL | For example `http://192.168.1.50:1234/v1` |
| API key | Optional. Most local servers ignore it |
| Triage model | Judges negatives and answers the same-event question in clustering. A small, fast model is enough |
| Summary model | Writes titles, descriptions, regenerations, and the digest opening. May be the same as the triage model |
| Embedding model | A multilingual embedding model, used for clustering (§5.5) |
| Concurrency | Default 1. Raise it if your server handles parallel requests |
| Request timeout | Default 120 seconds |
| Max article characters | Default 12,000 per article. Longer text is truncated to fit the context window |
Requirements:
- Uses `POST /v1/chat/completions`, `GET /v1/models` to populate the model pickers, and `POST /v1/embeddings` for clustering.
- **Test connection** button: checks reachability, lists models, and runs one sample triage, one sample title and description, and one embedding. Reports latency and whether the server honors JSON-schema structured output.
- Structured output uses `response_format` with a JSON schema where the server supports it, falls back to JSON mode, then to prompt-only with validation and one repair retry. Reasoning blocks (`