Ultrasongs：本地生成 Ultrastar Deluxe 歌曲文件的开源工具

Generate Ultrastar Deluxe song files from an audio or video file and its lyrics.
Everything runs on your own machine — no external AI services.

Work in progress — functional, but rough around the edges.

Pipeline: Demucs vocal separation → torchcrepe pitch detection → Whisper/WhisperX transcription → fuzzy lyric alignment → BPM-based beat mapping → Ultrastar .txt, with a visual timeline editor to fine-tune the result.

Hardware matters. With an NVIDIA GPU a 4-minute song takes about a minute. Without one the pipeline runs on the CPU and is slow — roughly 5× the length of the audio in our test (a 4-minute song: 20 minutes or more). It works, but plan accordingly. See Performance.

Contents

• Screenshots
• Features
• Windows installer
• Run from source
• Settings and languages
• How it works
• Performance
• Configuration reference
• Tech stack
• Ultrastar .txt format
• Credits · License

Screenshots

The interface comes in English and Spanish; both sets of screenshots (with an original demo song) are in docs/screenshots.

Features

• Any source: upload audio or video (MP3, MP4, MKV, FLAC, WAV, …) or paste a YouTube link (optional, can be disabled at build time).
• Automatic vocal separation with Demucs htdemucs.
• Per-word pitch with torchcrepe, a neural estimator tuned for singing.
• Transcription with faster-whisper, biased by your lyrics, plus optional WhisperX forced alignment for tighter word timing.
• Lyric alignment — fuzzy Levenshtein match of the transcription to your lyrics, syllable splitting for 20+ languages, BPM detection and beat-accurate note placement.
• Timeline editor — piano roll with audio and video playback, per-note editing, separate music/notes volume, and real-time microphone pitch tracking.
• Songs library — browse and import the songs already in your Ultrastar Songs folder, and write finished songs straight into it.
• Drafts — every processed song is saved automatically so you can come back to it.
• English and Spanish interface.

Windows installer

pnpm tauri:build produces src-tauri/target/release/bundle/nsis/Ultrasongs_<version>_x64-setup.exe (about 50 MB). It is a Tauri shell that starts the bundled Next.js server (with its own Node) and shows it in a native window.

The installer is small on purpose. Python, PyTorch and the AI models are downloaded the first time the app starts, from a progress screen:

An internet connection is needed for that first run only.

• No NVIDIA GPU: the first-run screen says so and asks you to confirm before downloading anything. Processing then runs on the CPU, about 5× slower than the audio.
• Not code-signed. Windows SmartScreen warns on first launch ("Windows protected your PC"): choose More info → Run anyway.
• Where things live: the program in %LOCALAPPDATA%\Ultrasongs; Python, models, drafts and settings in %LOCALAPPDATA%\com.ultrasongs.app. They are kept apart so uninstalling the program does not delete your drafts or the multi-GB models.
• Songs folder: choose your Ultrastar Songs folder in Settings. Finished songs are written there as Artist - Title/; folders the app did not create are never overwritten.

Build flags: NEXT_PUBLIC_ENABLE_YOUTUBE=0 removes the YouTube downloader from the build; ULTRASONGS_FORCE_DEVICE=cpu makes the installer skip the CUDA build (testing, or a broken NVIDIA driver).

Run from source

Requirements

• Node.js 20+ and pnpm
• Python 3.12 (the versions in python/requirements.lock.txt were tested on it)
• An NVIDIA GPU is strongly recommended; any CUDA-capable card works (development used a GTX 1080)
• Disk: about 7.5 GB for the Python environment (PyTorch CUDA is ~3 GB) plus 0.5–3 GB per Whisper model

FFmpeg needs no separate install: it comes with @ffmpeg-installer/ffmpeg.

Setup

# 1. Node dependencies
pnpm install

# 2. Python environment (one virtualenv runs everything)
python -m venv python/.venv
python/.venv/Scripts/python -m pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu126   # CPU: .../whl/cpu
python/.venv/Scripts/python -m pip install -r python/requirements.lock.txt

# 3. Configuration (the defaults work out of the box)
cp .env.example .env.local

The app uses python/.venv automatically when it exists and falls back to python on PATH.

Run

pnpm dev:all   # Next.js (port 3000) + Python service (port 8001)

or separately:

pnpm dev                                                           # Next.js only
cd python && python -m uvicorn transcribe_service:app --port 8001  # Python only

pnpm dev:all checks whether port 8001 is already listening before starting uvicorn, so it is safe to run repeatedly. For a production build run pnpm build and then pnpm start:all (or start.bat on Windows).

Settings and languages

The sidebar has a Settings entry with two options:

• Language — Español / English. The choice is kept in a cookie, so it survives restarts and the first paint is already in the right language. With no choice made, the browser's language is used (Spanish by default).
• Songs folder — your Ultrastar Songs folder. The desktop app opens the native folder picker. SONGS_DIR in .env.local overrides it and locks the control.

Translations

All interface text lives in app/i18n/es.json and app/i18n/en.json (flat "area.key": "text" pairs with {name} placeholders). Components use const { t } = useI18n(); t("key", { name }).

To add a language, create app/i18n/<code>.json with the same keys and register it in app/i18n/index.ts (Lang, LANGUAGES, messages); the build fails if a language is missing a key. Messages produced by the Python service and the first-run installer travel as keys and are translated in the browser. API errors are translated on the server from the language cookie: routes answer { message, code } through apiError() / apiFailure() (app/lib/apiResponse.ts) and library code throws AppError(key, vars) (app/lib/appError.ts) instead of fixed text. Raw technical details (pip output, third-party messages) are shown as they come.

How it works

Upload / YouTube
  └─ FFmpeg: any format → mono 128 kbps MP3

Transcribe (Python service, streamed to the browser over SSE)
  ├─ Demucs:      separate vocals and accompaniment
  ├─ torchcrepe:  pitch analysis on the vocals
  ├─ Pauses:      RMS silence regions
  └─ Whisper / WhisperX: transcribe the vocals, prompted by your lyrics

Generate
  ├─ Auto:   alignLyrics() → BPM detection → ms to beats → .txt
  └─ Editor: timeline notes → .txt  (line breaks inferred from pauses)

WhisperX forced alignment. Plain Whisper timestamps drift by ±100–300 ms. With ALIGN_ENGINE=whisperx the service runs WhisperX (wav2vec2 forced alignment) in a worker process that shares the service's environment (WHISPERX_PYTHON points it at a separate interpreter instead). The worker uses CUDA when available; WHISPERX_DEVICE and WHISPERX_COMPUTE_TYPE override that, and pre-Volta cards such as the GTX 1080 default to int8.

Performance

The CPU figure comes from one machine and varies with the processor; read it as an order of magnitude. Demucs and Whisper are the slow parts. For tighter VRAM budgets use WHISPER_MODEL=large-v3 (int8_float16 compute).

Configuration reference

Set these in .env.local (see .env.example). All are optional.

Tech stack

Ultrastar .txt format

#TITLE:Song Title
#ARTIST:Artist Name
#MP3:song.mp3
#BPM:120
#GAP:1200
: 0 4 60 Hel-
: 4 4 62 lo
- 16
: 20 4 60 World
E

• : normal note · * golden note (star) · F freestyle (not sung or scored) · R rap · G golden rap · - line break · E end of song
• Note: [type] [start_beat] [duration_beats] [midi_pitch] [syllable]
• GAP = milliseconds before beat 0; beat formula: ((ms − GAP) / 1000) × (BPM / 60) × 4

Support

If Ultrasongs is useful to you, you can support its development through GitHub Sponsors.

Credits

Built by Pablo Pramparo.

Powered by Demucs (Meta Research), faster-whisper, WhisperX, torchcrepe and yt-dlp.

License

MIT