diff --git a/TECHNICAL.md b/TECHNICAL.md index 2d01e601a..11461a2bf 100644 --- a/TECHNICAL.md +++ b/TECHNICAL.md @@ -283,19 +283,23 @@ Lists every built file with its content hash: ```json { "version": "", - "files": { "assets/javascripts/bundle.min.js": "abc123...", "index.html": "def456..." } + "files": { "assets/javascripts/bundle.min.js": "abc123...", "index.html": "def456..." }, + "sizes": { "assets/javascripts/bundle.min.js": 116474, "index.html": 8212 } } ``` Used by the SW for hash-based invalidation: on sync, files whose hash changed -are re-fetched; files no longer in the manifest are evicted. +are re-fetched; files no longer in the manifest are evicted. The `sizes` map +records the exact byte count of every built file on disk, so quota eviction +accounts entries at their true size instead of guessing. ### Service worker (`src/assets/javascripts/sw.js`) - **Caches**: `docsforge-` (content), `docsforge-meta` (manifest + previous-files list). - **Constants**: `BUILD_HASH`, `BASE_URL` (with trailing-slash normalization), - `ORIGIN_BASE`, `SYNC_CONCURRENCY = 6`, quota margins. + `ORIGIN_BASE`, `SYNC_CONCURRENCY = 6`, `DOWNLOAD_COST_BYTES = 20 MiB` + (per-download budget reservation), `QUOTA_MARGIN_RATIO = 0.1`. - **IndexedDB**: `docsforge-i18n` (locale preference). - **Messages**: - `DOCSFORGE_RELOAD_DETECTED` (from the page, on Navigation Timing @@ -308,7 +312,12 @@ are re-fetched; files no longer in the manifest are evicted. - **Manifest sync**: fetch `cache-manifest.json` (`cache: 'no-cache'`), diff against the previous files list, fetch changed URLs (concurrency 6), evict orphaned entries (LRU by access time when quota exceeded, plus - manifest-driven eviction for files no longer tracked). + manifest-driven eviction for files no longer tracked). The sync is + budgeted against the free space reported by `storage.estimate()`: each + download reserves a flat `DOWNLOAD_COST_BYTES` (20 MiB — the usage + estimate lags behind in-flight writes) and the sync stops once the budget + is exhausted, so files that cannot possibly fit are never downloaded in + the first place. Unbudgeted files are cached on demand when visited. - **Fetch strategy**: pages served from cache with background revalidation; static assets cache-first; navigation requests matched against the manifest (`manifestHasFile`) so pages absent from the manifest never 404 as @@ -316,7 +325,13 @@ are re-fetched; files no longer in the manifest are evicted. (`nav.type === 'back_forward'`) are deliberately **not** treated as revalidation triggers (design decision). - **Quota handling**: on QuotaExceeded, evicts LRU entries (using - `docsforge-access-times`), with a configurable margin, and retries. + `docsforge-access-times`) and retries. Eviction accounts each entry at its + **measured** byte size (Content-Length, falling back to reading the body), + frees until `available + freed` covers the required bytes plus a + proportional 10%-of-quota margin, and drops evicted entries from the + persisted previous-files list (`docsforge-manifest-files`) so a later sync + re-fetches them instead of believing they are cached. A single resource + larger than the whole quota is never cached. --- diff --git a/docs/docs/changelog/index.md b/docs/docs/changelog/index.md index 3a441f661..a846ecc30 100644 --- a/docs/docs/changelog/index.md +++ b/docs/docs/changelog/index.md @@ -1,3 +1,38 @@ +## [12.5.5] — 2026-08-19 + +### Added + +- **File sizes in `cache-manifest.json`** — the manifest now ships a `sizes` + map with the exact byte count of every built file. The service worker uses + it (plus the response's `Content-Length`, measured from the body when + missing) to account quota evictions at their real size instead of the old + guessed 5 MiB per entry. + +### Changed + +- **Service worker quota eviction is exact** — when the browser cache is + full, the SW evicts least-recently-used entries until the actually-freed + bytes (measured, not estimated) cover the required space plus a + proportional 10%-of-quota margin, then retries. The old flat 20 MiB free + margin is gone. Evicted entries are removed from the persisted + previous-files list, so a file that was evicted is never falsely recorded + as cached — the next sync re-fetches it instead of skipping it. +- **Manifest sync is budget-aware** — the background sync now checks the + free space reported by `storage.estimate()` before downloading: every + changed file reserves a flat 20 MiB of budget (the usage estimate lags + behind in-flight writes), and the sync stops once the budget is exhausted + instead of downloading files that would only be evicted again. Files that + don't fit are cached on demand when actually visited. A single resource + larger than the whole quota is never cached. + +### Fixed + +- **Safari (and any small-quota browser) no longer thrashes** — previously + the sync downloaded the whole manifest, hit the quota wall, evicted the + just-cached files, and recorded them as cached anyway; offline coverage + silently degraded to a few tail files. Now the sync stops early and the + tracked-file list stays truthful. + ## [12.5.4] — 2026-08-18 ### Added diff --git a/docs/docs/changelog/index.zh.md b/docs/docs/changelog/index.zh.md index cbc5c1e18..6fe7ab5f7 100644 --- a/docs/docs/changelog/index.zh.md +++ b/docs/docs/changelog/index.zh.md @@ -1,3 +1,33 @@ +## [12.5.5] — 2026-08-19 + +### 新增 + +- **`cache-manifest.json` 附带文件大小** —— manifest 现在带有一个 + `sizes` 映射，记录每个构建文件的精确字节数。Service Worker 用它 + （以及响应的 `Content-Length`，缺失时直接测量响应体）按真实大小进行 + 配额逐出，取代旧的"每个条目按 5 MiB 估算"的做法。 + +### 变更 + +- **Service Worker 配额逐出精确化** —— 浏览器缓存满时，SW 按 + 最近最少使用（LRU）顺序逐出条目，直到实际释放的字节数（真实测量， + 不再估算）覆盖所需空间并留出配额 10% 的成比例余量，然后重试。旧的 + 固定 20 MiB 空闲余量已移除。被逐出的条目会从持久化的"上次同步文件 + 列表"中删除，被逐出的文件绝不会被错误记录为已缓存 —— 下次同步会 + 重新拉取，而不是跳过。 +- **manifest 同步带预算控制** —— 后台同步在下载前先查询 + `storage.estimate()` 报告的空闲空间：每个变更文件预占 20 MiB 预算 + （用量估算落后于在途写入），预算耗尽后同步即停止，不再下载那些注定 + 会被再次逐出的文件。放不下的文件在真正访问时才按需缓存。单个资源 + 大于整个配额时永远不会被缓存。 + +### 修复 + +- **Safari（以及任何小额配额的浏览器）不再抖动** —— 此前同步会下载 + 整个 manifest，撞上配额上限后把刚缓存的文件逐出，却仍把它们记录为 + 已缓存；离线覆盖范围静默退化为 manifest 末尾的少量文件。现在同步会 + 提前停止，已跟踪文件列表始终真实可信。 + ## [12.5.4] — 2026-08-18 ### 新增 diff --git a/docsforge/build.py b/docsforge/build.py index 0d18a7926..fe4047cf1 100644 --- a/docsforge/build.py +++ b/docsforge/build.py @@ -1185,8 +1185,13 @@ def _generate_cache_manifest(site_dir: str, page_urls: list[str], files: Files | index, sitemap, PWA manifest, fonts, etc.). Hashes are computed from the Markdown SOURCE file when one exists, otherwise from the built file on disk. The SW uses this manifest to cache everything directly, without parsing HTML. + + The ``sizes`` map records the byte size of every built file on disk (the + exact number of bytes the SW stores in the browser cache), so quota + eviction can free precisely the space it claims instead of guessing. """ manifest_files = {} + manifest_sizes = {} # Build a lookup from page URL to source Markdown path. Multiple URL forms # can map to the same source (e.g. 'second/', 'second', 'second/index.html'). @@ -1241,10 +1246,12 @@ def _generate_cache_manifest(site_dir: str, page_urls: list[str], files: Files | h = hashlib.sha256(f.read()).hexdigest()[:16] manifest_files[url] = h + manifest_sizes[url] = os.path.getsize(abs_path) manifest = { "version": hashlib.sha256(json.dumps(manifest_files, sort_keys=True).encode()).hexdigest()[:12], "files": manifest_files, + "sizes": manifest_sizes, } manifest_path = os.path.join(site_dir, 'cache-manifest.json') diff --git a/docsforge/templates/assets/javascripts/sw.js b/docsforge/templates/assets/javascripts/sw.js index 24ca1b338..0392e6fa1 100644 --- a/docsforge/templates/assets/javascripts/sw.js +++ b/docsforge/templates/assets/javascripts/sw.js @@ -1 +1 @@ -"use strict";const BUILD_HASH="__DOCSFORGE_BUILD_HASH__",CACHE_NAME=`docsforge-${BUILD_HASH}`,META_CACHE="docsforge-meta",MANIFEST_URL="cache-manifest.json",FILES_KEY="docsforge-manifest-files",ACCESS_KEY="docsforge-access-times",BASE_URL="__DOCSFORGE_BASE_URL__".replace(/\/?$/,"/")||self.location.pathname.replace(/sw\.js$/,""),ORIGIN_BASE=self.location.origin+BASE_URL,I18N_DB_NAME="docsforge-i18n",I18N_DB_STORE="preferences",I18N_LOCALE_KEY="preferred_locale",SYNC_CONCURRENCY=6,QUOTA_MARGIN_BYTES=20*1024*1024,QUOTA_MARGIN_RATIO=.1;let _manifest=null,_manifestRefresh=null,_syncPromise=null,_preferredLocale=null;function log(...e){console.log("[SW]",...e)}function openI18nDB(){return new Promise((e,t)=>{const n=indexedDB.open(I18N_DB_NAME,1);n.onupgradeneeded=a=>{a.target.result.createObjectStore(I18N_DB_STORE)},n.onsuccess=a=>e(a.target.result),n.onerror=a=>t(a)})}async function readPreferredLocale(){try{const n=(await openI18nDB()).transaction(I18N_DB_STORE,"readonly").objectStore(I18N_DB_STORE);_preferredLocale=await new Promise((c,i)=>{const s=n.get(I18N_LOCALE_KEY);s.onsuccess=()=>c(s.result),s.onerror=()=>i(s.error)})||""}catch{_preferredLocale=""}return _preferredLocale}self.addEventListener("message",e=>{if(e.data&&e.data.type==="DOCSFORGE_RELOAD_DETECTED"&&(log("Reload detected by page; refreshing manifest in background"),refreshManifest().catch(()=>{})),e.data&&e.data.type==="DOCSFORGE_SET_LOCALE"){const t=e.data.locale||"";_preferredLocale=t,openI18nDB().then(n=>{const c=n.transaction(I18N_DB_STORE,"readwrite").objectStore(I18N_DB_STORE);t?c.put(t,I18N_LOCALE_KEY):c.delete(I18N_LOCALE_KEY)}).catch(()=>{})}});async function loadManifestFromCache(){if(_manifest)return _manifest;try{const t=await(await caches.open(META_CACHE)).match(MANIFEST_URL);if(t)return _manifest=await t.json(),log("Manifest loaded from meta cache:",_manifest.version),_manifest}catch{}return null}async function refreshManifest(){return _manifestRefresh||(_manifestRefresh=(async()=>{try{log("Fetching manifest...");const e=await fetch(MANIFEST_URL,{cache:"no-cache"});if(e.ok){const t=e.clone(),n=await e.json();await(await caches.open(META_CACHE)).put(MANIFEST_URL,t),_manifest=n,log("Manifest fetched:",n.version),await syncCacheFromManifest(n)}else log("Manifest fetch returned non-ok status:",e.status)}catch(e){log("Manifest refresh failed:",e.message)}return _manifest})().finally(()=>{_manifestRefresh=null}),_manifestRefresh)}async function getManifest(){return await loadManifestFromCache()}async function readPrevFiles(){const t=await(await caches.open(META_CACHE)).match(FILES_KEY);if(!t)return{};try{return await t.json()||{}}catch{return{}}}async function writePrevFiles(e){await(await caches.open(META_CACHE)).put(FILES_KEY,new Response(JSON.stringify(e)))}async function readAccessTimes(){const t=await(await caches.open(META_CACHE)).match(ACCESS_KEY);if(!t)return{};try{return await t.json()||{}}catch{return{}}}async function writeAccessTimes(e){await(await caches.open(META_CACHE)).put(ACCESS_KEY,new Response(JSON.stringify(e)))}async function touchAccessTime(e){try{const t=await readAccessTimes();t[e]=Date.now(),await writeAccessTimes(t)}catch{}}async function runWithConcurrency(e,t){const n=[],a=[];for(const[c,i]of e.entries()){const s=Promise.resolve().then(()=>i());n[c]=s;const o=s.then(()=>{});if(a.push(o),a.length>=t){await Promise.race(a);const r=a.findIndex(l=>l===o);r!==-1&&a.splice(r,1)}}return await Promise.all(a),Promise.all(n)}function keyToUrl(e){return new URL(e,ORIGIN_BASE).href}function urlToKey(e){const t=typeof e=="string"?e:e.href;return t.startsWith(ORIGIN_BASE)?t.slice(ORIGIN_BASE.length)||"./":null}function manifestHasFile(e,t){if(!e||!e.files)return!0;let n=urlToKey(t);return n===null?!0:(n.endsWith("/index.html")&&(n=n.slice(0,-10)||"./"),Object.prototype.hasOwnProperty.call(e.files,n))}async function makeSpaceIfNeeded(e=0){if(!navigator.storage||!navigator.storage.estimate)return;let t;try{t=await navigator.storage.estimate()}catch{return}if(!t||typeof t.usage!="number"||typeof t.quota!="number")return;const n=t.quota-t.usage,a=Math.max(e+QUOTA_MARGIN_BYTES,Math.floor(t.quota*QUOTA_MARGIN_RATIO));if(n>=a)return;const c=await caches.open(CACHE_NAME),i=await readAccessTimes(),s=[];for(const r of await c.keys()){const l=urlToKey(r.url);l&&(l===MANIFEST_URL||l==="sw.js"||s.push({url:r.url,key:l,time:i[r.url]||0}))}s.sort((r,l)=>r.time-l.time);let o=0;for(const r of s){if(n+o>=a)break;await c.delete(r.url)&&(delete i[r.url],o+=5*1024*1024)}await writeAccessTimes(i)}async function putWithQuotaHandling(e,t,n){try{await e.put(t,n.clone()),await touchAccessTime(t.url)}catch(a){if(a&&a.name==="QuotaExceededError"){log("Quota exceeded, evicting LRU entries...");const c=n.headers.get("content-length");await makeSpaceIfNeeded(c?parseInt(c,10):0);try{await e.put(t,n.clone()),await touchAccessTime(t.url)}catch(i){log("Still failed after eviction:",i.message)}}else throw a}}async function deleteOrphans(e,t){const n=await caches.open(CACHE_NAME),a=await n.keys();let c=0;for(const i of a){const s=urlToKey(i.url);s&&(s===MANIFEST_URL||s==="sw.js"||Object.prototype.hasOwnProperty.call(t,s)&&(Object.prototype.hasOwnProperty.call(e,s)||(await n.delete(i.url),c++)))}c>0&&log("Deleted",c,"orphaned cache entries")}async function syncCacheFromManifest(e){if(e)return _syncPromise||(_syncPromise=(async()=>{const t=await readPrevFiles(),n=e.files||{},a=await caches.open(CACHE_NAME);let c=0;const i=Object.keys(n);log("Syncing",i.length,"files from manifest...");const s=i.map(o=>async()=>{const r=n[o];if(t[o]!==r)try{const l=keyToUrl(o);log("Caching:",o);const f=await fetch(l,{cache:"no-cache"});f&&f.ok&&(await putWithQuotaHandling(a,l,f),t[o]=r,c++)}catch(l){log("Failed to cache:",o,l.message)}});await runWithConcurrency(s,SYNC_CONCURRENCY),await writePrevFiles(t),await deleteOrphans(n,t),log("Sync complete:",c,"files updated"),c>0&&self.clients.matchAll({includeUncontrolled:!0}).then(o=>o.forEach(r=>r.postMessage({type:"DOCSFORGE_UPDATE_READY",count:c}))).catch(()=>{})})().finally(()=>{_syncPromise=null}),_syncPromise)}async function respond404(){const e=await caches.open(CACHE_NAME),t=await readPreferredLocale(),n=[];t&&n.push(BASE_URL+"404."+t+".html"),n.push(BASE_URL+"404.html");for(const a of n){const c=await e.match(a).catch(()=>null);if(c){const i=await c.text();return new Response(i,{status:404,headers:{"Content-Type":"text/html"}})}}return new Response("

404 Not Found

",{status:404,headers:{"Content-Type":"text/html"}})}function buildPageCandidates(e,t){const n=[];return t&&(e.pathname.endsWith("/")?n.push(new URL(e.pathname+"index."+t+".html",e.origin).href):e.pathname.endsWith(".html")?n.push(new URL(e.pathname.slice(0,-5)+"."+t+".html",e.origin).href):n.push(new URL(e.pathname+"."+t+".html",e.origin).href)),n.push(e.href),e.pathname.endsWith("/")&&n.push(new URL(e.pathname+"index.html",e.origin).href),n}async function servePage(e){const t=await caches.open(CACHE_NAME),n=new URL(e.url),a=await readPreferredLocale(),c=await loadManifestFromCache(),i=buildPageCandidates(n,a);for(const s of i){const o=await t.match(s);if(o)return log("Serving page from cache:",s),await touchAccessTime(s),o}if(navigator.onLine===!1)return respond404();for(const s of i){if(!manifestHasFile(c,s)){log("Skipping page candidate not in manifest:",s);continue}log("Fetching page candidate:",s);try{const o=await fetch(s);if(o&&o.ok)return await putWithQuotaHandling(t,s,o),o}catch{}}return log("Page unavailable, returning 404:",e.url),respond404()}async function serveAsset(e){const t=await caches.open(CACHE_NAME),n=await t.match(e);if(n)return await touchAccessTime(e.url),n;try{const a=await fetch(e);if(a&&a.ok)return await putWithQuotaHandling(t,e,a),a}catch{}return new Response("Not found",{status:404})}function isPageRequest(e){return e.destination==="document"||e.mode==="navigate"?!0:e.method!=="GET"?!1:e.headers.get("X-DocsForge-Instant-Nav")==="1"?!0:(e.headers.get("accept")||"").includes("text/html")}self.addEventListener("install",e=>{log("Installing..."),e.waitUntil(self.skipWaiting())}),self.addEventListener("activate",e=>{log("Activating..."),e.waitUntil((async()=>{await readPreferredLocale();const t=await caches.open(CACHE_NAME);try{const a=await self.clients.matchAll({includeUncontrolled:!0,type:"window"}),c=a.find(i=>i.visibilityState==="visible")||a[0];if(c){log("Priming visible page:",c.url);const i=await servePage(new Request(c.url));i&&i.ok?log("Primed visible page:",c.url):log("Failed to prime visible page:",c.url,i.status)}}catch(a){log("Error priming visible page:",a.message)}await self.clients.claim(),log("Clients claimed"),await caches.keys().then(a=>Promise.all(a.filter(c=>c!==CACHE_NAME&&c!==META_CACHE).map(c=>caches.delete(c)))),log("Fetching manifest and syncing all files..."),await refreshManifest()||log("No manifest available after activation")})())}),self.addEventListener("fetch",e=>{const{request:t}=e;if(new URL(t.url).origin===self.location.origin){if(isPageRequest(t)){e.respondWith((async()=>(await getManifest(),servePage(t)))());return}e.respondWith(serveAsset(t))}}); +"use strict";const BUILD_HASH="__DOCSFORGE_BUILD_HASH__",CACHE_NAME=`docsforge-${BUILD_HASH}`,META_CACHE="docsforge-meta",MANIFEST_URL="cache-manifest.json",FILES_KEY="docsforge-manifest-files",ACCESS_KEY="docsforge-access-times",BASE_URL="__DOCSFORGE_BASE_URL__".replace(/\/?$/,"/")||self.location.pathname.replace(/sw\.js$/,""),ORIGIN_BASE=self.location.origin+BASE_URL,I18N_DB_NAME="docsforge-i18n",I18N_DB_STORE="preferences",I18N_LOCALE_KEY="preferred_locale",SYNC_CONCURRENCY=6,DOWNLOAD_COST_BYTES=20*1024*1024,QUOTA_MARGIN_RATIO=.1;let _manifest=null,_manifestRefresh=null,_syncPromise=null,_preferredLocale=null,_evicted=new Set;function log(...e){console.log("[SW]",...e)}function openI18nDB(){return new Promise((e,t)=>{const n=indexedDB.open(I18N_DB_NAME,1);n.onupgradeneeded=a=>{a.target.result.createObjectStore(I18N_DB_STORE)},n.onsuccess=a=>e(a.target.result),n.onerror=a=>t(a)})}async function readPreferredLocale(){try{const n=(await openI18nDB()).transaction(I18N_DB_STORE,"readonly").objectStore(I18N_DB_STORE);_preferredLocale=await new Promise((c,o)=>{const i=n.get(I18N_LOCALE_KEY);i.onsuccess=()=>c(i.result),i.onerror=()=>o(i.error)})||""}catch{_preferredLocale=""}return _preferredLocale}self.addEventListener("message",e=>{if(e.data&&e.data.type==="DOCSFORGE_RELOAD_DETECTED"&&(log("Reload detected by page; refreshing manifest in background"),refreshManifest().catch(()=>{})),e.data&&e.data.type==="DOCSFORGE_SET_LOCALE"){const t=e.data.locale||"";_preferredLocale=t,openI18nDB().then(n=>{const c=n.transaction(I18N_DB_STORE,"readwrite").objectStore(I18N_DB_STORE);t?c.put(t,I18N_LOCALE_KEY):c.delete(I18N_LOCALE_KEY)}).catch(()=>{})}});async function loadManifestFromCache(){if(_manifest)return _manifest;try{const t=await(await caches.open(META_CACHE)).match(MANIFEST_URL);if(t)return _manifest=await t.json(),log("Manifest loaded from meta cache:",_manifest.version),_manifest}catch{}return null}async function refreshManifest(){return _manifestRefresh||(_manifestRefresh=(async()=>{try{log("Fetching manifest...");const e=await fetch(MANIFEST_URL,{cache:"no-cache"});if(e.ok){const t=e.clone(),n=await e.json();await(await caches.open(META_CACHE)).put(MANIFEST_URL,t),_manifest=n,log("Manifest fetched:",n.version),await syncCacheFromManifest(n)}else log("Manifest fetch returned non-ok status:",e.status)}catch(e){log("Manifest refresh failed:",e.message)}return _manifest})().finally(()=>{_manifestRefresh=null}),_manifestRefresh)}async function getManifest(){return await loadManifestFromCache()}async function readPrevFiles(){const t=await(await caches.open(META_CACHE)).match(FILES_KEY);if(!t)return{};try{return await t.json()||{}}catch{return{}}}async function writePrevFiles(e){await(await caches.open(META_CACHE)).put(FILES_KEY,new Response(JSON.stringify(e)))}async function readAccessTimes(){const t=await(await caches.open(META_CACHE)).match(ACCESS_KEY);if(!t)return{};try{return await t.json()||{}}catch{return{}}}async function writeAccessTimes(e){await(await caches.open(META_CACHE)).put(ACCESS_KEY,new Response(JSON.stringify(e)))}async function touchAccessTime(e){try{const t=await readAccessTimes();t[e]=Date.now(),await writeAccessTimes(t)}catch{}}async function runWithConcurrency(e,t){const n=[],a=[];for(const[c,o]of e.entries()){const i=Promise.resolve().then(()=>o());n[c]=i;const l=i.then(()=>{});if(a.push(l),a.length>=t){await Promise.race(a);const f=a.findIndex(s=>s===l);f!==-1&&a.splice(f,1)}}return await Promise.all(a),Promise.all(n)}function keyToUrl(e){return new URL(e,ORIGIN_BASE).href}function urlToKey(e){const t=typeof e=="string"?e:e.href;return t.startsWith(ORIGIN_BASE)?t.slice(ORIGIN_BASE.length)||"./":null}function manifestHasFile(e,t){if(!e||!e.files)return!0;let n=urlToKey(t);return n===null?!0:(n.endsWith("/index.html")&&(n=n.slice(0,-10)||"./"),Object.prototype.hasOwnProperty.call(e.files,n))}async function measuredSize(e,t){try{const n=await e.match(t);if(!n)return 0;const a=parseInt(n.headers.get("content-length"),10);return Number.isFinite(a)&&a>0?a:(await n.clone().arrayBuffer()).byteLength}catch{return 0}}async function storageEstimate(){if(!navigator.storage||!navigator.storage.estimate)return null;try{const e=await navigator.storage.estimate();if(e&&typeof e.usage=="number"&&typeof e.quota=="number"&&e.quota>0)return e}catch{}return null}async function availableBytes(){const e=await storageEstimate();return e?Math.max(0,e.quota-e.usage):null}async function makeSpaceIfNeeded(e=0){const t=await storageEstimate();if(!t)return!1;const n=Math.max(0,t.quota-t.usage);if(e>0&&e>t.quota)return!1;const a=Math.max(e,Math.floor(t.quota*QUOTA_MARGIN_RATIO));if(n>=a)return!0;const c=await caches.open(CACHE_NAME),o=await readAccessTimes(),i=[];for(const s of await c.keys()){const r=urlToKey(s.url);r&&(r===MANIFEST_URL||r==="sw.js"||i.push({url:s.url,time:o[s.url]||0}))}i.sort((s,r)=>s.time-r.time);let l=0;for(const s of i){if(n+l>=a)break;const r=await measuredSize(c,s.url);await c.delete(s.url)&&(delete o[s.url],_evicted.add(s.url),l+=r)}await writeAccessTimes(o);const f=[];for(const s of _evicted){const r=manifestKeyOf(s);r&&f.push(r)}if(f.length>0){const s=await readPrevFiles();let r=!1;for(const u of f)Object.prototype.hasOwnProperty.call(s,u)&&(delete s[u],r=!0);r&&await writePrevFiles(s)}return n+l>=a}function manifestKeyOf(e){let t=urlToKey(e);return t===null||t===MANIFEST_URL||t==="sw.js"?null:(t.endsWith("/index.html")&&(t=t.slice(0,-10)||"./"),t)}async function putWithQuotaHandling(e,t,n){try{return await e.put(t,n.clone()),await touchAccessTime(t.url),!0}catch(a){if(a&&a.name==="QuotaExceededError"){log("Quota exceeded, evicting LRU entries (measured)...");const c=n.headers.get("content-length"),o=c?parseInt(c,10):0;if(await makeSpaceIfNeeded(o))try{return await e.put(t,n.clone()),await touchAccessTime(t.url),!0}catch(l){log("Still failed after eviction:",l.message)}else log("Could not free enough space for",t.url,"(required",o,"bytes)")}else throw a;return!1}}async function deleteOrphans(e,t){const n=await caches.open(CACHE_NAME),a=await n.keys();let c=0;for(const o of a){const i=urlToKey(o.url);i&&(i===MANIFEST_URL||i==="sw.js"||Object.prototype.hasOwnProperty.call(t,i)&&(Object.prototype.hasOwnProperty.call(e,i)||(await n.delete(o.url),c++)))}c>0&&log("Deleted",c,"orphaned cache entries")}async function syncCacheFromManifest(e){if(e)return _syncPromise||(_syncPromise=(async()=>{const t=await readPrevFiles(),n=e.files||{},a=await caches.open(CACHE_NAME);let c=0,o=await availableBytes();const i=Object.keys(n);log("Syncing",i.length,"files from manifest...");const l=[];for(const s of i){const r=n[s];if(t[s]!==r){if(o!==null){if(o{const r=n[s];try{const u=keyToUrl(s);log("Caching:",s);const h=await fetch(u,{cache:"no-cache"});h&&h.ok&&(await putWithQuotaHandling(a,u,h)?(t[s]=r,c++):log("Not cached (quota):",s))}catch(u){log("Failed to cache:",s,u.message)}};await runWithConcurrency(l.map(s=>()=>f(s)),SYNC_CONCURRENCY);for(const s of _evicted){const r=manifestKeyOf(s);r&&Object.prototype.hasOwnProperty.call(t,r)&&(delete t[r],log("Unmarked evicted file:",r))}_evicted.clear(),await writePrevFiles(t),await deleteOrphans(n,t),log("Sync complete:",c,"files updated"),c>0&&self.clients.matchAll({includeUncontrolled:!0}).then(s=>s.forEach(r=>r.postMessage({type:"DOCSFORGE_UPDATE_READY",count:c}))).catch(()=>{})})().finally(()=>{_syncPromise=null}),_syncPromise)}async function respond404(){const e=await caches.open(CACHE_NAME),t=await readPreferredLocale(),n=[];t&&n.push(BASE_URL+"404."+t+".html"),n.push(BASE_URL+"404.html");for(const a of n){const c=await e.match(a).catch(()=>null);if(c){const o=await c.text();return new Response(o,{status:404,headers:{"Content-Type":"text/html"}})}}return new Response("

404 Not Found

",{status:404,headers:{"Content-Type":"text/html"}})}function buildPageCandidates(e,t){const n=[];return t&&(e.pathname.endsWith("/")?n.push(new URL(e.pathname+"index."+t+".html",e.origin).href):e.pathname.endsWith(".html")?n.push(new URL(e.pathname.slice(0,-5)+"."+t+".html",e.origin).href):n.push(new URL(e.pathname+"."+t+".html",e.origin).href)),n.push(e.href),e.pathname.endsWith("/")&&n.push(new URL(e.pathname+"index.html",e.origin).href),n}async function servePage(e){const t=await caches.open(CACHE_NAME),n=new URL(e.url),a=await readPreferredLocale(),c=await loadManifestFromCache(),o=buildPageCandidates(n,a);for(const i of o){const l=await t.match(i);if(l)return log("Serving page from cache:",i),await touchAccessTime(i),l}if(navigator.onLine===!1)return respond404();for(const i of o){if(!manifestHasFile(c,i)){log("Skipping page candidate not in manifest:",i);continue}log("Fetching page candidate:",i);try{const l=await fetch(i);if(l&&l.ok)return await putWithQuotaHandling(t,i,l),l}catch{}}return log("Page unavailable, returning 404:",e.url),respond404()}async function serveAsset(e){const t=await caches.open(CACHE_NAME),n=await t.match(e);if(n)return await touchAccessTime(e.url),n;try{const a=await fetch(e);if(a&&a.ok)return await putWithQuotaHandling(t,e,a),a}catch{}return new Response("Not found",{status:404})}function isPageRequest(e){return e.destination==="document"||e.mode==="navigate"?!0:e.method!=="GET"?!1:e.headers.get("X-DocsForge-Instant-Nav")==="1"?!0:(e.headers.get("accept")||"").includes("text/html")}self.addEventListener("install",e=>{log("Installing..."),e.waitUntil(self.skipWaiting())}),self.addEventListener("activate",e=>{log("Activating..."),e.waitUntil((async()=>{await readPreferredLocale();const t=await caches.open(CACHE_NAME);try{const a=await self.clients.matchAll({includeUncontrolled:!0,type:"window"}),c=a.find(o=>o.visibilityState==="visible")||a[0];if(c){log("Priming visible page:",c.url);const o=await servePage(new Request(c.url));o&&o.ok?log("Primed visible page:",c.url):log("Failed to prime visible page:",c.url,o.status)}}catch(a){log("Error priming visible page:",a.message)}await self.clients.claim(),log("Clients claimed"),await caches.keys().then(a=>Promise.all(a.filter(c=>c!==CACHE_NAME&&c!==META_CACHE).map(c=>caches.delete(c)))),log("Fetching manifest and syncing all files..."),await refreshManifest()||log("No manifest available after activation")})())}),self.addEventListener("fetch",e=>{const{request:t}=e;if(new URL(t.url).origin===self.location.origin){if(isPageRequest(t)){e.respondWith((async()=>(await getManifest(),servePage(t)))());return}e.respondWith(serveAsset(t))}}); diff --git a/src/assets/javascripts/sw.js b/src/assets/javascripts/sw.js index 2f1423cb4..cda86244e 100644 --- a/src/assets/javascripts/sw.js +++ b/src/assets/javascripts/sw.js @@ -30,7 +30,12 @@ const I18N_DB_STORE = 'preferences'; const I18N_LOCALE_KEY = 'preferred_locale'; const SYNC_CONCURRENCY = 6; -const QUOTA_MARGIN_BYTES = 20 * 1024 * 1024; +// Conservative per-download cost when budgeting the manifest sync against +// free space: each changed file reserved at a flat 20 MiB regardless of its +// real size, because the browser's quota-usage estimate lags behind in-flight +// writes. The sync stops downloading once the budget is exhausted instead of +// fetching files that will only be evicted again. +const DOWNLOAD_COST_BYTES = 20 * 1024 * 1024; const QUOTA_MARGIN_RATIO = 0.1; // In-memory manifest + dedupe promises. @@ -38,6 +43,10 @@ let _manifest = null; let _manifestRefresh = null; let _syncPromise = null; let _preferredLocale = null; +// URLs evicted by makeSpaceIfNeeded() since the last reconciliation. The +// manifest sync removes them from the persisted previous-files list before +// writing, so evicted files are never falsely recorded as cached. +let _evicted = new Set(); function log(...args) { console.log('[SW]', ...args); @@ -227,16 +236,52 @@ function manifestHasFile(manifest, url) { return Object.prototype.hasOwnProperty.call(manifest.files, key); } -async function makeSpaceIfNeeded(requiredBytes = 0) { - if (!navigator.storage || !navigator.storage.estimate) return; - let estimate; +// Actual byte size of a cached entry. Content-Length is exact for network +// responses (the stored body is exactly what the server sent); synthesized +// responses (404 pages, etc.) have no header, so measure the body directly. +async function measuredSize(cache, url) { try { - estimate = await navigator.storage.estimate(); - } catch (e) { return; } - if (!estimate || typeof estimate.usage !== 'number' || typeof estimate.quota !== 'number') return; - const available = estimate.quota - estimate.usage; - const targetFree = Math.max(requiredBytes + QUOTA_MARGIN_BYTES, Math.floor(estimate.quota * QUOTA_MARGIN_RATIO)); - if (available >= targetFree) return; + const resp = await cache.match(url); + if (!resp) return 0; + const header = parseInt(resp.headers.get('content-length'), 10); + if (Number.isFinite(header) && header > 0) return header; + const body = await resp.clone().arrayBuffer(); + return body.byteLength; + } catch (e) { + return 0; + } +} + +// Storage estimate (usage + quota), or null when unavailable/unusable (the +// caller then relies on reactive quota handling). +async function storageEstimate() { + if (!navigator.storage || !navigator.storage.estimate) return null; + try { + const est = await navigator.storage.estimate(); + if (est && typeof est.usage === 'number' && typeof est.quota === 'number' && est.quota > 0) return est; + } catch (e) { /* ignore */ } + return null; +} + +// Free space per storage.estimate(), or null when unavailable. +async function availableBytes() { + const est = await storageEstimate(); + return est ? Math.max(0, est.quota - est.usage) : null; +} + +// Evict least-recently-used entries until `available + freed` covers +// requiredBytes, accounting each entry at its ACTUAL byte size (no guesses). +// Evicted entries are removed from the persisted previous-files list so a +// later manifest sync re-fetches them; returns true when enough space was +// freed for the caller to retry its cache.put(). +async function makeSpaceIfNeeded(requiredBytes = 0) { + const est = await storageEstimate(); + if (!est) return false; + const available = Math.max(0, est.quota - est.usage); + // A single resource larger than the whole quota can never be cached. + if (requiredBytes > 0 && requiredBytes > est.quota) return false; + const targetFree = Math.max(requiredBytes, Math.floor(est.quota * QUOTA_MARGIN_RATIO)); + if (available >= targetFree) return true; const cache = await caches.open(CACHE_NAME); const times = await readAccessTimes(); @@ -246,41 +291,81 @@ async function makeSpaceIfNeeded(requiredBytes = 0) { if (!key) continue; // Never evict the manifest itself or sw.js. if (key === MANIFEST_URL || key === 'sw.js') continue; - entries.push({ url: req.url, key, time: times[req.url] || 0 }); + entries.push({ url: req.url, time: times[req.url] || 0 }); } entries.sort((a, b) => a.time - b.time); let freed = 0; for (const entry of entries) { if (available + freed >= targetFree) break; + const size = await measuredSize(cache, entry.url); const deleted = await cache.delete(entry.url); if (deleted) { delete times[entry.url]; - // We don't know the real byte size; assume a modest chunk and keep evicting. - freed += 5 * 1024 * 1024; + _evicted.add(entry.url); + freed += size; } } await writeAccessTimes(times); + + // Drop evicted entries from the persisted previous-files list so the next + // manifest sync re-fetches them instead of believing they are cached. + const evictedKeys = []; + for (const url of _evicted) { + const key = manifestKeyOf(url); + if (key) evictedKeys.push(key); + } + if (evictedKeys.length > 0) { + const prev = await readPrevFiles(); + let changed = false; + for (const key of evictedKeys) { + if (Object.prototype.hasOwnProperty.call(prev, key)) { + delete prev[key]; + changed = true; + } + } + if (changed) await writePrevFiles(prev); + } + return available + freed >= targetFree; +} + +// Normalize a URL to its manifest key form ('./', 'second/', ...), matching +// how the build emits directory-index pages. Returns null for URLs outside +// the site or for the manifest/sw.js files themselves. +function manifestKeyOf(url) { + let key = urlToKey(url); + if (key === null) return null; + if (key === MANIFEST_URL || key === 'sw.js') return null; + if (key.endsWith('/index.html')) key = key.slice(0, -'index.html'.length) || './'; + return key; } async function putWithQuotaHandling(cache, request, response) { try { await cache.put(request, response.clone()); await touchAccessTime(request.url); + return true; } catch (e) { if (e && (e.name === 'QuotaExceededError')) { - log('Quota exceeded, evicting LRU entries...'); + log('Quota exceeded, evicting LRU entries (measured)...'); const sizeHint = response.headers.get('content-length'); - await makeSpaceIfNeeded(sizeHint ? parseInt(sizeHint, 10) : 0); - try { - await cache.put(request, response.clone()); - await touchAccessTime(request.url); - } catch (e2) { - log('Still failed after eviction:', e2.message); + const required = sizeHint ? parseInt(sizeHint, 10) : 0; + const freedEnough = await makeSpaceIfNeeded(required); + if (freedEnough) { + try { + await cache.put(request, response.clone()); + await touchAccessTime(request.url); + return true; + } catch (e2) { + log('Still failed after eviction:', e2.message); + } + } else { + log('Could not free enough space for', request.url, '(required', required, 'bytes)'); } } else { throw e; } + return false; } } @@ -313,28 +398,64 @@ async function syncCacheFromManifest(manifest) { const cache = await caches.open(CACHE_NAME); let updated = 0; + // Budget the sync against the free space available NOW: every download + // costs a flat DOWNLOAD_COST_BYTES (the estimate lags behind in-flight + // writes), and the sync stops once the budget is exhausted instead of + // fetching files that would only be evicted again. Files skipped here are + // cached on demand when actually visited. A null budget (no usable + // estimate) falls back to reactive quota handling. + let budget = await availableBytes(); const entries = Object.keys(newFiles); log('Syncing', entries.length, 'files from manifest...'); - const tasks = entries.map(key => async () => { + const tasks = []; + for (const key of entries) { const newHash = newFiles[key]; - if (prevFiles[key] === newHash) return; + if (prevFiles[key] === newHash) continue; + if (budget !== null) { + if (budget < DOWNLOAD_COST_BYTES) { + log('Quota budget exhausted, stopping sync at', key); + break; + } + budget -= DOWNLOAD_COST_BYTES; + } + tasks.push(key); + } + log('Syncing', tasks.length, 'changed files within quota budget...'); + const runTask = async (key) => { + const newHash = newFiles[key]; try { const fullUrl = keyToUrl(key); log('Caching:', key); const resp = await fetch(fullUrl, { cache: 'no-cache' }); if (resp && resp.ok) { - await putWithQuotaHandling(cache, fullUrl, resp); - prevFiles[key] = newHash; - updated++; + const ok = await putWithQuotaHandling(cache, fullUrl, resp); + if (ok) { + prevFiles[key] = newHash; + updated++; + } else { + log('Not cached (quota):', key); + } } } catch (e) { log('Failed to cache:', key, e.message); } - }); + }; + + await runWithConcurrency(tasks.map(k => () => runTask(k)), SYNC_CONCURRENCY); - await runWithConcurrency(tasks, SYNC_CONCURRENCY); + // Never record evicted files as cached: drop every URL evicted during + // this sync (or while it was running) from the previous-files list so the + // next sync re-fetches what actually got evicted. + for (const url of _evicted) { + const key = manifestKeyOf(url); + if (key && Object.prototype.hasOwnProperty.call(prevFiles, key)) { + delete prevFiles[key]; + log('Unmarked evicted file:', key); + } + } + _evicted.clear(); await writePrevFiles(prevFiles); await deleteOrphans(newFiles, prevFiles); log('Sync complete:', updated, 'files updated'); diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py index 86b468b02..682130300 100644 --- a/tests/e2e/conftest.py +++ b/tests/e2e/conftest.py @@ -56,8 +56,13 @@ def has_browser() -> bool: return _HAS_BROWSER -def _build_fixture(tmp_path: Path, site_name: str = "E2E", with_nav: bool = True, language: str | None = None) -> tuple[Path, Path]: - """Build a small docsforge site and return (root, site_dir).""" +def _build_fixture(tmp_path: Path, site_name: str = "E2E", with_nav: bool = True, language: str | None = None, quota_assets: int = 0) -> tuple[Path, Path]: + """Build a small docsforge site and return (root, site_dir). + + quota_assets: number of 2 MiB static binaries to drop into docs/assets + (used by the quota/eviction tests to force the SW against the browser's + storage limit). + """ from docsforge.config_base import load_config from docsforge.build import build @@ -68,6 +73,10 @@ def _build_fixture(tmp_path: Path, site_name: str = "E2E", with_nav: bool = True (docs / "second.md").write_text("# Second\n\nAnother page with searchable content.\n\nUniqueTokenSecond\n") (docs / "guide").mkdir() (docs / "guide" / "intro.md").write_text("# Introduction\n\nIntro material.\n") + if quota_assets: + (docs / "assets").mkdir() + for i in range(quota_assets): + (docs / "assets" / f"big{i + 1}.bin").write_bytes(b"x" * (2 * 1024 * 1024)) nav_block = ( "nav:\n - Home: index.md\n - Second: second.md\n - Guide:\n - guide/intro.md\n" if with_nav else "" @@ -133,6 +142,21 @@ def served_site_i18n(tmp_path_factory): httpd.shutdown() +@pytest.fixture(scope="module") +def served_site_quota(tmp_path_factory): + """A fixture site with 2x 2 MiB static binaries for quota/eviction tests.""" + if not has_browser(): + pytest.skip("Playwright/Chromium unavailable — E2E tests skipped") + _, site_dir = _build_fixture(tmp_path_factory.mktemp("e2e-quota"), quota_assets=2) + httpd = socketserver.ThreadingTCPServer(("127.0.0.1", 0), lambda *a: _Handler(*a, directory=str(site_dir))) + httpd.daemon_threads = True + port = httpd.server_address[1] + t = threading.Thread(target=httpd.serve_forever, daemon=True) + t.start() + yield f"http://127.0.0.1:{port}/" + httpd.shutdown() + + @pytest.fixture def context_page(served_site): """A fresh Playwright context/page (isolated SW state per test).""" diff --git a/tests/e2e/test_browser.py b/tests/e2e/test_browser.py index 05ba5d7ce..b971aaaf6 100644 --- a/tests/e2e/test_browser.py +++ b/tests/e2e/test_browser.py @@ -134,6 +134,166 @@ def test_dev_server_matches_deployed(served_dev): p.stop() +# --- Quota / eviction tests ------------------------------------------------- +# +# These run Chromium with a tiny per-origin quota (--quota-override-size-mb) +# so the SW's quota handling actually fires. The service worker budgets the +# manifest sync at a flat 20 MiB per download, so with a 25 MiB quota exactly +# one file gets synced; and on-demand caching of a 2 MiB binary evicts the +# LRU entries with measured byte accounting. Tests skip when the override is +# not honoured (quota still huge), keeping the suite green on Chromium builds +# without the flag. + +_QUOTA_OVERRIDE_FLOOR = 100 * 1024 * 1024 + + +def _launch_quota(quota_mb: int): + from playwright.sync_api import sync_playwright + from _browser import launch_opts + + opts = launch_opts() + opts["args"] = [f"--quota-override-size-mb={quota_mb}"] + p = sync_playwright().start() + return p, p.chromium.launch(**opts) + + +def _quota_override_applied(page, quota_mb: int) -> bool: + """True when the quota override took effect (estimate reports ~quota_mb).""" + return page.evaluate( + "async (mb) => (await navigator.storage.estimate()).quota <= mb * 1024 * 1024", + quota_mb, + ) + + +def _content_cache_name(page): + """Name of the SW content cache (docsforge-), not the meta cache.""" + return page.evaluate( + """async () => { + const names = await caches.keys(); + return names.find(n => n.startsWith('docsforge-') && n !== 'docsforge-meta'); + }""" + ) + + +def _cached_urls(page, cache_name: str): + return page.evaluate( + "async (name) => (await (await caches.open(name)).keys()).map(r => r.url)", + cache_name, + ) + + +def _tracked_files(page): + """The SW's persisted previous-files list (docsforge-manifest-files).""" + return page.evaluate( + """async () => { + const cache = await caches.open('docsforge-meta'); + const resp = await cache.match('docsforge-manifest-files'); + return resp ? await resp.json() : {}; + }""" + ) + + +def test_manifest_sync_budget_stops_when_quota_small(served_site_quota): + """With a 25 MiB quota the flat 20 MiB-per-download budget must stop the + manifest sync after one file, and every tracked file must really be in + the cache (no 'marked cached but evicted' lies).""" + base_url = served_site_quota + p, browser = _launch_quota(25) + context = browser.new_context() + page = context.new_page() + try: + page.goto(base_url, wait_until="networkidle") + _sw_ready(page) + if not _quota_override_applied(page, 25): + pytest.skip("quota override not honoured by this Chromium") + page.wait_for_function( + """async () => { + const cache = await caches.open('docsforge-meta'); + const resp = await cache.match('docsforge-manifest-files'); + if (!resp) return false; + const files = await resp.json(); + return Object.keys(files).length === 1; + }""", + timeout=15000, + ) + tracked = _tracked_files(page) + assert len(tracked) == 1, f"expected exactly 1 tracked file, got {list(tracked)}" + cache_name = _content_cache_name(page) + cached = set(_cached_urls(page, cache_name)) + for key in tracked: + expected = base_url + ("" if key == "./" else key) + assert expected.rstrip("/") in {u.rstrip("/") for u in cached}, ( + f"tracked file {key!r} is not actually cached" + ) + # The site ships many more files than the budget allowed to download. + manifest = page.evaluate( + "async () => (await (await caches.open('docsforge-meta')).match('cache-manifest.json')).json()" + ) + assert len(manifest["files"]) > 1, "fixture site is too small for this test" + usage = page.evaluate("async () => (await navigator.storage.estimate()).usage") + assert usage <= 25 * 1024 * 1024 + finally: + context.close() + browser.close() + p.stop() + + +def test_eviction_uses_measured_sizes(served_site_quota): + """On-demand caching of a 2 MiB binary under a 3 MiB quota must evict the + LRU entries (with real byte accounting), keep the cache within quota, and + never leave a tracked file that is missing from the cache.""" + base_url = served_site_quota + p, browser = _launch_quota(3) + context = browser.new_context() + page = context.new_page() + try: + page.goto(base_url, wait_until="networkidle") + _sw_ready(page) + if not _quota_override_applied(page, 3): + pytest.skip("quota override not honoured by this Chromium") + + # Fetch both 2 MiB binaries through the SW. The first fits; the second + # exceeds the remaining space, forcing measured LRU eviction. + for asset in ("assets/big1.bin", "assets/big2.bin"): + page.evaluate( + "async (a) => { const r = await fetch(a); if (!r.ok) throw new Error(r.status); await r.arrayBuffer(); }", + asset, + ) + page.wait_for_function( + """async (url) => { + const names = await caches.keys(); + const name = names.find(n => n.startsWith('docsforge-') && n !== 'docsforge-meta'); + const cache = await caches.open(name); + return !!(await cache.match(url)); + }""", + base_url + "assets/big2.bin", + timeout=15000, + ) + + cache_name = _content_cache_name(page) + cached = {u.rstrip("/") for u in _cached_urls(page, cache_name)} + assert base_url.rstrip("/") + "/assets/big2.bin" in cached, "big2.bin must be cached" + assert base_url.rstrip("/") + "/assets/big1.bin" not in cached, ( + "big1.bin must have been evicted to make room for big2.bin" + ) + assert base_url.rstrip("/") not in cached, "home page must have been evicted first (LRU)" + + # Every tracked file must still be present in the content cache. + tracked = _tracked_files(page) + for key in tracked: + expected = base_url + ("" if key == "./" else key) + assert expected.rstrip("/") in cached, ( + f"tracked file {key!r} was evicted but still recorded as cached" + ) + + usage = page.evaluate("async () => (await navigator.storage.estimate()).usage") + assert usage <= 3 * 1024 * 1024, f"cache usage {usage} exceeds the 3 MiB quota" + finally: + context.close() + browser.close() + p.stop() + + def test_i18n_translates_ui(served_site_i18n): """A site built with theme.language='fr' must render and translated UI strings (not the English defaults).""" diff --git a/tests/integration/test_build_e2e.py b/tests/integration/test_build_e2e.py index ab7cd6fd1..99de788c2 100644 --- a/tests/integration/test_build_e2e.py +++ b/tests/integration/test_build_e2e.py @@ -84,6 +84,17 @@ def test_generates_cache_manifest(self, tmp_project, monkeypatch): "cache-manifest.json must not be in cache manifest" ) + # Sizes map: exact byte count of every built file, matching disk. + sizes = cm["sizes"] + assert set(sizes.keys()) == set(files.keys()), ( + "sizes must cover exactly the manifest files" + ) + for key, size in sizes.items(): + assert isinstance(size, int) and size > 0 + assert sizes["404.html"] == (tmp_project / "site" / "404.html").stat().st_size, ( + "size must be the exact built-file byte count" + ) + def test_second_build_is_incremental(self, tmp_project, monkeypatch): """The second build must not rewrite an unchanged page's output.""" _build_once(monkeypatch, tmp_project) diff --git a/tests/regression/test_regressions.py b/tests/regression/test_regressions.py index fb6718e06..37dc77a0a 100644 --- a/tests/regression/test_regressions.py +++ b/tests/regression/test_regressions.py @@ -171,6 +171,40 @@ def test_regression_manifest_key_is_files_not_Files(tmp_path: Path, monkeypatch) assert "Files" not in data +def test_regression_manifest_sizes_match_built_files(tmp_path: Path, monkeypatch): + """_generate_cache_manifest must ship a `sizes` map with the exact byte + size of every built file (the SW evicts by measured size, no guessing).""" + import json + + import docsforge.build as build_mod + + monkeypatch.chdir(tmp_path) + (tmp_path / "site").mkdir() + src = tmp_path / "docs" / "p.md" + src.parent.mkdir() + src.write_text("# P") + out = tmp_path / "site" / "p" / "index.html" + out.parent.mkdir() + out.write_text("" + ("x" * 4096) + "") + + from docsforge.files import Files + + files = Files([]) + build_mod._generate_cache_manifest(str(tmp_path / "site"), ["p/"], files) + data = json.loads((tmp_path / "site" / "cache-manifest.json").read_text()) + + assert "sizes" in data, "cache manifest must carry a sizes map" + assert set(data["sizes"].keys()) == set(data["files"].keys()), ( + "every manifest file must have a size" + ) + for key, size in data["sizes"].items(): + built = tmp_path / "site" / (key.replace("./", "").rstrip("/") + "/index.html" if key.endswith("/") else key) + assert built.is_file(), f"size key {key!r} must resolve to a built file" + assert size == built.stat().st_size, ( + f"size for {key!r} must be the exact built-file byte count" + ) + + # --------------------------------------------------------------------------- # v11.1.5 (found while writing tests) — find_orphaned_outputs vs dir URLs # ---------------------------------------------------------------------------