From a multi-agent bug-hunt + adversarial verification pass (0 critical/
high; 15 mediums). Fixed 13; 2 deferred as bigger refactors.
Correctness:
- srs: "Hard" no longer overshoots "Good" for reviewed cards (reps>=2). It
compounded ease AND x1.2; now grows x1.2 off the previous interval only.
+ regression test.
- enrich/dates: stop reading the modal verb "may" as the month May (phantom
calendar events). "may" needs an ordinal/year/date-preposition now.
- enrich/bib: disambiguate colliding BibTeX keys (smith2020, smith2020a, …)
— duplicates corrupted reference-manager imports.
- learn/glossary: junk-term guard used && (dead); now || so all-stopword
terms like "there" are actually skipped.
- transcription/engineImpl.web: don't collapse Whisper's null end-timestamp
to a zero-length [t,t] segment; estimate from the next chunk or window
duration (fixes citation/seam anchors).
- transcription/pipeline: re-check the abort signal AFTER each chunk so
Cancel works on single-chunk audio (it previously still saved).
- db/repo.native: use withExclusiveTransactionAsync for reassign /
deleteCourse / upsertVectors / createFlashcards (withTransactionAsync is
not isolated on a shared connection → interleaved/half-applied writes).
- db/repo.native: don't cache a rejected open/migrate promise — a transient
first-open failure no longer bricks storage for the whole session.
- stores/transcriptsStore: sequence-guard refresh() so overlapping
focus/typing/filter refreshes can't resolve out of order and show stale
results.
- audio/wav: decode WAVE_FORMAT_EXTENSIBLE (0xFFFE) via its SubFormat GUID
+ add 24-bit PCM — common ffmpeg/Windows WAVs no longer hard-fail native
import. + tests.
Performance / footprint:
- audio/decode.native: decode straight to 16kHz (decodeAudioData sampleRate
hint) so the JS side never holds a full-rate buffer or runs the resample
loop — big memory/OOM win on long lectures.
- models/catalog display: Settings model sizes are now backend-aware (the
no-GPU WASM path pulls ~2x fp32 weights; was advertising ~half).
Feature gap:
- download: native exports were silent no-ops. New download.native.ts writes
to cache + opens the share sheet (expo-sharing); transcript/ICS/Anki-CSV/
BibTeX/RIS exports now work on device.
Deferred (bigger): lexical-search recall ceiling (needs a full-corpus rank
path on both repos); triple in-memory copy of the encoded file during
transcribe (media is keyed by a not-yet-existing transcript id).
Validated: tsc clean, 282 tests pass, web export clean (native deps not
bundled), arm64 APK compiles with expo-sharing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The WASM backend couldn't create a session — "TransposeDQWeightsFor
MatMulNBits Missing required scale" — so transcription failed to start on
mobile (which we route to WASM) and on any no-WebGPU device. Root cause:
the default quantized Whisper decoder (q8/q4, decoder_model_merged) uses
MatMulNBits ops that the onnxruntime-web bundled with transformers.js
4.2.0 cannot load on WASM. Reproduced and bisected in a browser harness:
across both Xenova and onnx-community repos, q8/q4/string-fp32 all fail on
WASM with this error, while an EXPLICIT per-file fp32 decoder
({ encoder_model: 'fp32', decoder_model_merged: 'fp32' }) loads and runs
(2/2 chunks, no error) on a no-GPU machine.
So WASM now requests that explicit fp32 decoder; WebGPU keeps fp16. Larger
download on the WASM path, but it actually loads and runs — q8 never did
on WASM. (4.2.0 is the latest transformers.js, so bumping isn't an option.)
tsc clean, 279 tests pass, web export OK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two issues behind "it just shows downloaded 100% / loads forever / crashes
after a couple seconds":
1. Progress was invisible during transcription. The 'transcribing' stage
only fired AFTER the first (slow) chunk, so the UI sat at "Loading
model… 100%" through the entire first inference and looked frozen.
- pipeline now emits a transcribing kickoff (progress 0) BEFORE the
first chunk and carries chunkIndex/chunkCount on every event.
- transcribeStore threads those through; the Library job card shows a
spinner, "Transcribing… N%", and "part i of N" so a long file's
progress is legible and obviously advancing.
2. Web crash on mobile. We picked WebGPU + fp16 whenever navigator.gpu
existed, but mobile WebGPU drivers crash on sustained fp16 Whisper
inference (transcribes a chunk, then the GPU process dies). Mobile web
now uses cross-origin-isolated multi-threaded WASM (slower but stable);
desktop keeps WebGPU.
Native's "loads forever" was the same missing-feedback problem — it was
grinding through chunks with no UI signal; the chunk counter now shows it.
tsc clean, 279 tests pass (pipeline progress test updated for the kickoff
event), web export OK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Whisper ".en" models reject `task`/`language`; transformers.js throws
"Cannot specify `task` or `language` for an English-only model" if either
is passed — and the web engine passed `task: 'transcribe'` (plus a
possibly-undefined `language`) unconditionally. Since the default model is
tiny.en, web/PWA transcription failed for every default user.
Only add `task`/`language` for multilingual models now (translation also
requires a multilingual model), and never pass an explicit `undefined`
language (which trips the same check). Mirror the same gate on native
(whisper.rn) so a .en model is never asked to do a language/translate it
can't.
tsc clean, 279 tests pass, web export OK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two native-only fixes behind the "transcription failed [object Object]"
report on the APK:
1. Model download. engineImpl.native.loadModel assumed the ggml .bin was
already on disk (the downloader was a TODO), so on a real device it
rejected — and since in-app recording is web-only, no model was ever
present. Now loadModel fetches the model from Hugging Face
(ggerganov/whisper.cpp) into <documentDirectory>/models on first use,
reporting 0..1 progress through the existing "Loading model… X%" UI.
Downloads to a .part file and renames on success so an interrupted
download can't leave a truncated model; if initWhisper still rejects,
the file is deleted so the next try re-downloads cleanly. Model URLs
verified (tiny.en 77.7MB / base.en 148MB / small.en 488MB).
2. Error surfacing. transcribeStore did `String(err)`, which renders a
non-Error native rejection (whisper.rn / file system throw plain
objects) as "[object Object]". New shared errorMessage() pulls
message/reason/code (or JSON) out of whatever was thrown, so failures
are actionable instead of opaque.
Progress downloads aren't in the new expo-file-system OO API yet, so the
fetch uses the still-supported expo-file-system/legacy resumable
downloader; native-only, not bundled on web.
Validated: tsc clean, 279 tests pass, web export unaffected (legacy not
in the web bundle), arm64 APK builds and the native JS bundle resolves
the legacy import. On-device download + transcribe still needs a real
phone to confirm end to end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On native, importing anything but WAV failed with "Only WAV is supported
on native" — ffmpeg-kit-react-native was retired in 2025 and a real
decoder was left as a follow-up. Since in-app recording is web-only, the
native app's only way to add audio is importing a phone recording — which
is m4a/aac — so transcription was effectively broken on the APK.
Add react-native-audio-api (Software Mansion, maintained) and route
non-WAV files through its standalone decodeAudioData(ArrayBuffer), which
uses platform codecs + bundled FFmpeg. WAV keeps its pure-JS fast path.
The decoded channels go through the existing toMono16k, mirroring the web
AudioContext.decodeAudioData path exactly. Bytes are read locally and
passed as an ArrayBuffer, so it doesn't matter whether the picker yields
file:// or content://. WAV is now sniffed by RIFF/WAVE content, not just
the extension.
The lib's config plugin is intentionally NOT enabled: it only sets up
playback (background audio mode, a media-playback foreground service, mic
permission), none of which decoding needs — and adding those would
contradict the app's on-device/no-extra-permissions stance.
Validated locally (arm64 Docker build): react-native-audio-api compiles
and autolinks against RN 0.85 / new arch; the arm64-v8a APK ships
libreact-native-audio-api.so + libav{codec,format,util}/libswresample.so
+ liboboe.so (48MB -> 58MB). tsc clean, 279 tests pass, web export does
NOT bundle the native dep. Runtime decode still needs an on-device check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Android CI job recompiled all native C++ (whisper.cpp, reanimated,
worklets, gesture-handler) from scratch every push — ~41 min on the
shared runner — because nothing survived the `COPY . .` layer
invalidation, and `--load` exported the whole ~15GB image just to copy
one APK out.
Dockerfile (docker/android.Dockerfile):
- Install ccache and patch the NDK CMake toolchain file so every module's
externalNativeBuild routes C/C++ compiles + links through ccache (the
third-party modules don't honor ccache on their own). ccache dir is a
persistent BuildKit cache mount, so objects survive across builds.
- Mount GRADLE_USER_HOME as a cache and pass --build-cache --parallel, so
Kotlin/Java/resource/dex tasks and resolved deps persist too.
- Cache-mount the bun install dir; move the nodejs install into the cached
toolchain layer (was reinstalling on every source change).
- New `apk` stage (FROM scratch) holding just the signed APK, for
--output type=local extraction without loading the 15GB image.
CI (.gitea/workflows/ci.yml):
- build-apk now builds `--target apk --output type=local,dest=./out` and
streams ./out/app-release.apk to /srv/wisp/wisp.apk — no --load, no
docker create/cp, and the build image no longer piles up on the host.
Validated locally (arm64, real sign+assemble path):
cold gradle 5m5s -> warm gradle 57s; 61/61 C++ compiles served from
ccache (100% hit); 226/578 Gradle tasks from cache; export-only build
60s wall, APK valid + v2/v3 signed. The first server build after this
is still cold (~41 min) to populate the caches; pushes after that are
much faster.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Library crammed five text nav-links into a non-wrapping row with no
bottom navigation — unusable on phones. Restructure the primary screens
into a bottom tab bar:
- Add src/app/(tabs)/_layout.tsx: <Tabs> with 5 tabs (Library, Search,
Study, Ask, Settings) with emoji icons and the accent active tint.
- Move index/search/study/ask/settings into the (tabs) route group; the
parens keep URLs unchanged (/, /search, /study, /ask, /settings).
- Root _layout becomes a Stack hosting (tabs) (headerless) plus the
secondary pushed screens (record, transcript/[id], courses, quiz,
bibliography), each with a back-button header.
- Drop the per-screen <Stack.Screen> headers from the moved tabs (titles
now come from Tabs); relocate Study's "Export Anki" action into the body.
- Library: remove the cramped header link-row and duplicate title; add a
"nothing uploaded" subheader and a Courses link.
Verified at 375px: tab bar pinned to the bottom, 5 even tabs, navigation
between tabs works. tsc clean, web export OK, 279 tests pass.
CI: build-apk now runs automatically on every push to master (after
deploy-web), so the APK at /wisp.apk tracks master instead of being
frozen at a stale tag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Strictly opt-in, gated, with the deterministic features as the always-present floor:
- GenerationEngine: WebLLM (Qwen2.5-1.5B, WebGPU, CDN-loaded) + BYO-key cloud
(OpenAI-compatible); native stub. Pure grounding prompt builder (4 tests).
- rag.askLectures: retrieve Phase-1 hits -> grounded prompt -> answer with
citations; refuses when nothing relevant; falls back to search-only when no
engine is available. Never sends raw audio/transcripts — only question + snippets.
- aiStore (BYO key persisted in localStorage on web), Ask screen (answer +
tappable citation chips that jump to the audio + honest "verify" disclaimer),
Settings AI section (engine status + bring-your-own-key form).
279 tests green, 0 tsc errors, web export builds. ROADMAP phases 0-5 complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- useRecorder: MediaRecorder mic capture with elapsed timer, live input-level
meter (AnalyserNode), pause/resume, and a screen Wake Lock; returns the
recording as an ArrayBuffer that flows into the existing transcribe pipeline.
- Record screen: big timer + level meter + start/pause/resume/stop; on stop the
pre-capture sheet collects title/course/date, then it transcribes + saves +
indexes (reusing Phase 0/1). Home gets Record + Import buttons.
- Web-only for now; native realtime (whisper.rn), OPFS crash-recovery,
tab-audio capture, and the per-course dashboard are follow-ups.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- android.Dockerfile: build arm64-v8a only (--max-workers=2) → ~48MB APK, ~4x
less compile (the 4-ABI build timed out on the shared prod runner).
- ci.yml: build-apk runs only on version tags (v*), timeout 120m — normal pushes
no longer peg the box; web still auto-deploys on every master push.
- wisp.compose.yml: mount /srv/wisp/wisp.apk read-only so the download survives
web container rebuilds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- docker/android.Dockerfile: install nodejs (image ships Bun; ci-android-sign.sh
patches build.gradle with a Node script).
- buffer polyfill: whisper.rn -> safe-buffer requires Node's `buffer`, absent in
Metro/RN. Add `buffer` dep, alias it in metro.config.js, set the global in
_layout. Web export unaffected (verified).
Validated locally end-to-end: assembleRelease BUILD SUCCESSFUL, 117MB APK across
all 4 ABIs with librnwhisper.so, signed (CN=Wisp, O=briggen.dev).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- .gitea/workflows/ci.yml rewritten for a self-hosted runner co-located with
the host Docker daemon + Traefik: push to master -> test -> deploy-web
(docker compose up --build in place; no SSH, no registry). APK job documented
as the next iteration (socket-safe build+extract; signing secrets already set).
- .gitignore: never commit the release keystore (*.keystore + wisp-release.keystore).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NilsBriggen