dfeen87/Acoustic-Projection-Microphone-System：实时翻译声学投射麦克风系统

Production-grade implementation of an advanced acoustic projection microphone system with real-time translation capabilities.

Table of Contents

• Features
• 🌍 Local Translation (100% Private)
• Architecture
• 🚀 Quick Start
• 🌐 REST API with Global Node Access
• 📊 Production Launcher Features
• 🧪 System Validation
• 📁 Project Structure Overview
• 🐳 Docker Deployment
• 🛠️ Troubleshooting
• 📊 Performance
• 💻 API Documentation
• 🧪 Testing
• ⚙️ Configuration
• 🤝 Contributing
• 📜 System Requirements
• 🎯 What's Next
• License
• 🙏 Acknowledgments
• 📚 References
• 📖 Citation
• 📧 Support

Features

• Advanced Beamforming: Delay-and-sum, superdirective, and adaptive null-steering algorithms
• Deep Noise Suppression: LSTM-based neural network for speech enhancement
• Acoustic Echo Cancellation: NLMS adaptive filter with double-talk detection
• Voice Activity Detection: Energy and zero-crossing rate based VAD with hangover mechanism
• Real-time Translation: Local Whisper + NLLB pipeline (200+ languages, fully offline); TensorFlow Lite engine also supported
• Directional Audio Projection: Phased array synthesis for targeted audio delivery
• End-to-End Encryption: ChaCha20-Poly1305 + X25519 key exchange via libsodium; Argon2id password-based key derivation and file encryption
• Push-to-Talk (PTT) Controller: Hardware/software PTT with keyboard, mouse, external pedal, and software-controlled modes
• Call Signaling: UDP-based call setup, teardown, and session management with peer discovery
• Real Audio I/O: PortAudio integration for live microphone capture and speaker playback (when PortAudio is installed)
• WAV File I/O: Read and write WAV files for offline processing and testing
• High Performance: FFTW-optimized FFT with STFT support, multi-threaded processing, SIMD-ready
• Production Launcher: Enterprise-grade startup system with automatic health checks and monitoring
• REST API with Global Node Access: FastAPI-based REST API for peer discovery and session management across all network nodes

BEDROCK baseline: v11.0.0

Version 11 preserves the DSP/control/UI architecture while enforcing validation, authentication, persistence and test-runner contracts. See release notes and invariant map and the Bedrock 1.1 adversarial verification and release gaps. API-key-enabled deployments require credentials in the dashboard Settings; loading the page does not grant API access. Software validation does not certify microphone hardware, acoustic output safety or model accuracy.

New in Version 10.0.0 (V10)

• Smartglasses Acoustic-Projection-Microphone-System (A-P-M-S):
  • 2-mic temple frame topology (14–16 cm baseline) with geometric acoustic projection onto the virtual mouth vector.
  • Phase-aware spectral masking and phase coherence index ($\gamma$) for spatial filtering without bulky front-frame physical arrays.
  • Ultra-low-power recursive IIR spectral gain smoothing (< 15–20 mW DSP power budget).
  • Sub-10 ms processing latency operating on native 16 kHz, 10 ms frames (160 samples per channel).
• AI Assistant Front-End Integration Interface:
  • Direct 16 kHz mono PCM output contract for downstream wake-word detection (Sensory, Porcupine/PvRecorder) and streaming ASR (Whisper, Kaldi, ONNX Runtime).
  • Embedded frame metadata output including VAD confidence (speech_confidence), real-time SNR (estimated_snr_db), and speech activity decisions (is_speech_active).
• Comprehensive C++ DSP & Test Suite Additions:
  • Dedicated DSP pipeline module in src/dsp/smartglasses_apm/smartglasses_apm.cpp.
  • Mathematical projection correctness tests (tests/test_smartglasses_apm_projection.cpp).
  • Pipeline latency, MACs/cycle benchmark, and numerical edge-case robustness tests (tests/test_smartglasses_apm_dsp_pipeline.cpp).
• Detailed Specification & Pseudocode Documentation:
  • Complete architecture spec at docs/architecture/smartglasses_apm_spec.md.
  • Step-by-step algorithm pseudocode at docs/architecture/smartglasses_apm_pseudocode.md.

🌍 Local Translation (100% Private)

APM System includes fully local speech recognition and translation using state-of-the-art AI models. Your conversations never leave your device.

Features

• 🔒 100% Private - No cloud APIs, all processing on-device
• 🌐 200+ Languages - Powered by Meta's NLLB translation model
• 🎤 Accurate Speech Recognition - OpenAI Whisper for transcription
• ⚡ Real-time Performance - 2-4 seconds per sentence (GPU) or 5-8 seconds (CPU)
• 🚫 No Internet Required - Works completely offline after initial setup

Quick Setup

# One-command setup
./scripts/setup.sh --full

# Or step-by-step: activate venv and run translation bridge
source venv/bin/activate
python3 scripts/translation_bridge.py audio.wav --source en --target es

Platform Notes

• Supported on Linux and macOS (Intel & Apple Silicon)
• Text translation fallback uses portable <cctype> classification (no locale-dependent behavior) for cross-platform correctness
• CI covers Linux and macOS; local validation evidence and platform limits are recorded in the release reports rather than implying every hosted job passed.

Supported Languages

English, Spanish, French, German, Italian, Portuguese, Russian, Chinese, Japanese, Korean, Arabic, Hindi, and 180+ more.

See docs/translation/ for complete translation documentation.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      APM System Pipeline                     │
└─────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────┐
│  Microphone     │───▶│   Beamforming    │───▶│    Echo     │
│  Array (4-16)   │    │   Engine         │    │ Cancellation│
└─────────────────┘    └──────────────────┘    └─────────────┘
                               │                       │
                               ▼                       ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────┐
│   Directional   │◀───│   Translation    │◀───│    Noise    │
│   Projector     │    │   Engine         │    │ Suppression │
└─────────────────┘    └──────────────────┘    └─────────────┘
        │                                              │
        ▼                                              ▼
┌─────────────────┐                          ┌─────────────┐
│  Speaker Array  │                          │     VAD     │
│    (3-8)        │                          │   Engine    │
└─────────────────┘                          └─────────────┘

Architecture Notes

The codebase is intentionally split into two layers:

• APMCore (src/apm_core.cpp): lightweight DSP + text fallback suitable for embedding
• APMSystem (src/core/apm_system.cpp): full real-time pipeline (beamforming, NS, AEC, VAD, translation, projection)

Each component is defined exactly once to ensure clean builds across Linux and macOS.

🚀 Quick Start

Prerequisites

• Node.js 14+ (for launcher) - Download
• CMake 3.18+ - Download
• C++20 Compiler - GCC 10+, Clang 11+, or MSVC 2019+
• FFTW3 (optional) - sudo apt-get install libfftw3-dev (Linux) or brew install fftw (Mac)
• libsodium (optional, for encryption) - sudo apt-get install libsodium-dev (Linux) or brew install libsodium (Mac)
• PortAudio (optional, for live audio I/O) - sudo apt-get install portaudio19-dev (Linux) or brew install portaudio (Mac)

One-Command Launch

# Linux/Mac
./scripts/start-apm.sh

# Windows
scripts\start-apm.bat

That's it! The launcher will:

• ✅ Validate your environment
• ✅ Install Node.js dependencies (if needed)
• ✅ Build the C++ backend (if needed)
• ✅ Start the APM system
• ✅ Open the dashboard in your browser

Manual Setup

If you prefer step-by-step control:

# 1. Install launcher dependencies
cd launcher
npm install

# 2. Build C++ backend
cd ..
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release

# 3. Start the system
cd launcher
npm start

The system starts on:

• Backend API: http://localhost:8080
• Dashboard UI: http://localhost:4173

Custom Configuration

# Use different ports
APM_BACKEND_PORT=9000 APM_UI_PORT=5000 npm start

# Enable debug logging
DEBUG=1 npm start

# Combined
APM_BACKEND_PORT=9000 DEBUG=1 npm start

🌐 REST API with Global Node Access

The APM system includes a FastAPI-based REST API server that enables peer discovery and session management across all nodes in the network.

Quick Start - REST API Only

Start the REST API server independently:

# Linux/Mac
./scripts/start-api.sh

# Windows
scripts\start-api.bat

# With custom configuration
python3 backend/main.py --host 0.0.0.0 --port 8080

Global Node Access

By default, the API binds to 0.0.0.0, making it accessible from any node in the network. This enables:

• Peer Discovery: Other nodes can query this node's peer list
• Session Management: Remote nodes can initiate and manage sessions
• Health Monitoring: Network-wide health checks

API Endpoints

Interactive API Documentation

Once running, visit:

• Swagger UI: http://localhost:8080/docs
• ReDoc: http://localhost:8080/redoc

Configuration Options

# Environment variables
export APM_API_HOST=0.0.0.0      # Global access (default)
export APM_API_PORT=8080         # API port (default)

# Command-line arguments
python3 backend/main.py --host 0.0.0.0 --port 8080 --reload

For complete REST API documentation, see backend/README.md

📊 Production Launcher Features

The APM launcher is enterprise-grade with:

🔍 Pre-flight Validation

• Checks for required executables and files
• Validates port availability
• Verifies build artifacts
• Ensures proper file permissions

🏥 Health Monitoring

• Automatic backend health checks every 300ms
• 60-second timeout with informative error messages
• Real-time process monitoring
• Captures backend stdout/stderr for debugging

🛡️ Robust Error Handling

• Graceful shutdown on SIGINT/SIGTERM
• Force-kill after 5-second timeout
• Port conflict detection
• Detailed error messages with solutions

📝 Production Logging

[2025-01-15T10:30:45.123Z] [INFO] Validating environment...
[2025-01-15T10:30:45.456Z] [SUCCESS] Environment validation passed
[2025-01-15T10:30:45.789Z] [INFO] Starting C++ backend...
[2025-01-15T10:30:46.012Z] [Backend] Server listening on port 8080
[2025-01-15T10:30:47.345Z] [SUCCESS] Backend healthy after 3 checks (1234ms)
[2025-01-15T10:30:47.678Z] [SUCCESS] UI server listening on http://localhost:4173
[2025-01-15T10:30:48.901Z] [SUCCESS] APM System is fully operational! 🚀

🔐 Security

• UI server binds to 127.0.0.1 only (localhost)
• Security headers enabled (X-Frame-Options, X-XSS-Protection, X-Content-Type-Options)
• No external file system access from UI server
• 404 for all non-root paths

⚡ Performance

• Fast startup: < 5 seconds typical
• Minimal overhead: ~30MB RAM for launcher
• Automatic process cleanup
• Multi-platform support (Windows/Mac/Linux)

🧪 System Validation

Health Check Script

# Validate your entire setup
node scripts/healthcheck.js

Checks:

• ✅ Node.js version (14+)
• ✅ CMake installation
• ✅ C++ compiler availability
• ✅ File structure integrity
• ✅ Backend binary exists
• ✅ Dependencies installed
• ✅ Runtime status (if running)
• ✅ Port availability

Integration Tests

# Run full integration test suite
node tests/integration/integration.test.js

Tests include:

• Backend health endpoint
• Response time benchmarks
• Concurrent request handling
• UI server functionality
• Security headers
• Load testing (100 sequential, 50 concurrent requests)

📁 Project Structure Overview

Acoustic-Projection-Microphone-System/
├── backend/                     # Python FastAPI backend and signaling/control modules
├── cmake/                       # CMake package/config helpers
├── config/                      # Optional/runtime dependency configuration
├── docker/                      # Container build assets
├── docs/                        # Architecture, build, deployment, security, and user docs
├── examples/                    # C++ and Python usage examples
├── frontend/                    # Frontend adapters/hooks/components
├── include/apm/                 # Public C++ API headers
├── installers/                  # Installer assets (e.g., Inno Setup)
├── launcher/                    # Node.js launcher/orchestration entrypoint
├── scripts/                     # Setup, startup, and utility scripts
├── src/                         # Core C++ implementation (DSP, signaling, translation, tools)
├── tests/                       # C++ and Node integration tests
├── tools/                       # Utility tools and scripts
├── ui/                          # React + Vite web UI
├── CMakeLists.txt               # Root CMake build configuration
├── README.md                    # Project overview and usage
├── TESTING_AND_VALIDATION.md    # Validation matrix and test guidance
└── SYSTEM_MANIFEST.md           # Repository/system manifest

🐳 Docker Deployment

Quick Start

# Build the existing API/native image
docker build -f docker/Dockerfile -t apm-system:11.0.0 .

# Set a deployment key in your environment before exposing protected APIs.
: "${APM_API_KEY:?Set APM_API_KEY to your deployment key}"
docker run --rm -p 127.0.0.1:8080:8080 \
  -e APM_API_KEY -e APM_DB_PATH=/data/apm.sqlite \
  -v apm-data:/data apm-system:11.0.0

Production Deployment

The maintained Dockerfile starts the Python API and includes the native binary; it does not package the React dashboard, inference models or PortAudio capture. /health proves API liveness. Offline/fallback metrics are expected without a running native telemetry source and must not be interpreted as working audio. Provision TLS/ingress and credentials explicitly for network deployments.

scripts/validate-docker-context.sh checks that local credentials and database files are excluded before publishing an image. The runtime validation script checks API authentication and SQLite restart persistence; neither check certifies model quality or hardware behavior. See the release reports for the exact image build conditions, including proxy/CA limitations.

🛠️ Troubleshooting

Launcher Issues

Error: Backend executable not found

# Rebuild the backend
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release

Error: Backend port 8080 is already in use

# Find and kill the process
lsof -i :8080          # Linux/Mac
netstat -ano | findstr :8080  # Windows

# Or use a different port
APM_BACKEND_PORT=8081 npm start

Error: Backend health check timed out

# Run backend directly to see errors
./apm_backend  # or ./build/apm_backend

# Check for:
# - Firewall blocking localhost
# - Missing dependencies
# - Port conflicts

Error: UI file not found

# Verify file exists in parent directory
ls -la ../apm-dashboard.html

# File must be at: apm/apm-dashboard.html
# Launcher must be at: apm/launcher/apm_launcher.js

Build Issues

Q: Build fails with "fftw3.h not found"
A: Install FFTW:

sudo apt-get install libfftw3-dev  # Ubuntu/Debian
brew install fftw                   # macOS
vcpkg install fftw3                 # Windows

Q: Tests fail with "Segmentation fault"
A: Check audio frame sizes match across components. Ensure FFT size ≤ frame size.

Q: Poor beamforming performance
A: Verify microphone spacing matches speed of sound. Calibrate microphone positions.

Q: High CPU usage
A: Reduce sample rate from 48kHz to 16kHz for lower quality requirements.

Q: Echo cancellation not working
A: Ensure speaker reference signal is provided. Check for timing synchronization.

Getting Help

1. Check logs: Enable debug mode with DEBUG=1 npm start
2. Run health check: node scripts/healthcheck.js
3. Verify prerequisites: Node.js 14+, CMake 3.18+, C++20 compiler
4. Check ports: Ensure 8080 and 4173 are available

📊 Performance

Benchmarked on Intel i7-12700K, 32GB RAM, Ubuntu 22.04:

Memory usage:

• Backend: ~15MB (without TFLite models)
• Launcher: ~30MB
• Total: ~45MB baseline

💻 API Documentation

Public Facade

APMCore

Lightweight public facade for initializing the system and performing audio processing and text translation. Suitable for embedding.

APMCore core;
core.initialize(48000, 1);           // sample_rate, num_channels
core.set_source_language("en");
core.set_target_language("es");

std::vector<float> out = core.process(input_samples);

APMCore::TextTranslationResult r = core.translate_text("Hello");
// r.success, r.translated_text, r.processing_time_ms

Note: The C++ text fallback in APMCore supports only en→es and en→fr. Full 200+ language support requires the Python translation bridge (scripts/translation_bridge.py).

Core Classes

AudioFrame

Encapsulates audio data with metadata.

AudioFrame(size_t samples, int sample_rate, int channels);
std::span<float> samples();           // Access audio data
void compute_metadata();              // Calculate peak, RMS, clipping
std::vector<float> channel(int ch);   // Extract single channel

BeamformingEngine

Spatial filtering for directional audio capture.

BeamformingEngine(int num_mics, float spacing_m);

AudioFrame delay_and_sum(
    const std::vector<AudioFrame>& mic_array,
    float azimuth_rad,
    float elevation_rad
);

AudioFrame superdirective(
    const std::vector<AudioFrame>& mic_array,
    float azimuth_rad
);

NoiseSuppressionEngine

Deep learning-based noise reduction.

AudioFrame suppress(const AudioFrame& noisy);
void reset_state();  // Reset LSTM state

EchoCancellationEngine

Adaptive echo cancellation with NLMS.

EchoCancellationEngine(int filter_length = 2048);

AudioFrame cancel_echo(
    const AudioFrame& microphone,
    const AudioFrame& speaker_reference
);

bool detect_double_talk(const AudioFrame& mic, const AudioFrame& ref);
void reset();  // Reset adaptive filter weights and reference buffer

VoiceActivityDetector

Speech/non-speech classification.

struct VadResult {
    bool speech_detected;
    float confidence;      // 0.0 to 1.0
    float snr_db;
    float energy_db;
};

VadResult detect(const AudioFrame& frame);
void adapt_threshold(float ambient_noise_db);
void reset();  // Reset hangover counter

FFTProcessor

High-performance FFT using FFTW (only available when built with FFTW3; throws at runtime otherwise).

FFTProcessor(int size);

void forward(const std::vector<float>& input,
            std::vector<std::complex<float>>& output);

void inverse(const std::vector<std::complex<float>>& input,
            std::vector<float>& output);

static void apply_window(std::vector<float>& data, WindowType type);
// WindowType: Hann, Hamming, Blackman, Kaiser

APMSystem

Complete processing pipeline.

struct Config {
    int num_microphones = 4;
    float mic_spacing_m = 0.012f;
    int num_speakers = 3;
    float speaker_spacing_m = 0.015f;
    int sample_rate = 48000;
    std::string source_language = "en-US";
    std::string target_language = "es-ES";
};

APMSystem(const Config& config);
APMSystem();  // Default constructor (uses Config defaults)

std::vector<AudioFrame> process(
    const std::vector<AudioFrame>& microphone_array,
    const AudioFrame& speaker_reference,
    float target_direction_rad
);

std::future<std::vector<AudioFrame>> process_async(...);
void reset_all();  // Reset echo canceller, noise suppressor, and VAD state

🧪 Testing

# Run all tests
cd build && ctest --output-on-failure

# Run specific test suite
./apm_test --gtest_filter=BeamformingTest.*

# Run with detailed output
./apm_test --gtest_output=xml:test_results.xml

# Memory leak check
valgrind --leak-check=full ./apm_test

# Performance profiling
perf record ./apm_bench
perf report

# Integration tests
node tests/integration/integration.test.js

Test coverage: 87% (lines), 92% (functions)

⚙️ Configuration

Hardware Setup

Microphone Array:

• Linear array: 4-8 microphones
• Spacing: 10-15mm (λ/2 at 11kHz)
• Recommended: omnidirectional electret or MEMS

Speaker Array:

• Linear array: 3-6 speakers
• Spacing: 15-20mm
• Recommended: full-range drivers, 85dB+ SPL

Software Configuration

// Low-latency configuration
config.num_microphones = 4;
config.mic_spacing_m = 0.012f;

// High-quality configuration
config.num_microphones = 8;
config.mic_spacing_m = 0.010f;

// Language support
config.source_language = "en-US";  // English
config.target_language = "es-ES";  // Spanish
// APMSystem accepts any IETF language tag; 200+ languages available via Python bridge
// C++ text fallback (APMCore) supports: en→es, en→fr

Environment Variables

# Launcher configuration
export APM_BACKEND_PORT=8080      # Backend API port
export APM_UI_PORT=4173           # Dashboard UI port
export DEBUG=1                     # Enable debug logging

# Backend configuration
export APM_NUM_MICS=4             # Number of microphones
export APM_MIC_SPACING=0.012      # Microphone spacing (meters)
export APM_NUM_SPEAKERS=3         # Number of speakers

🤝 Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit changes (git commit -m 'Add amazing feature')
4. Push to branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Code Style

• Follow C++20 Core Guidelines
• Format with clang-format (Google style)
• Add unit tests for new features
• Update documentation
• Run node scripts/healthcheck.js before submitting

📜 System Requirements

Minimum

• CPU: 2 cores
• RAM: 512MB
• Disk: 100MB
• Node.js: 18.0.0+ (the Vite 5 dashboard build requires Node 18 or later)
• CMake: 3.18+

Recommended

• CPU: 4+ cores with AVX2
• RAM: 2GB+
• Disk: 1GB+
• Node.js: 18.0.0+
• CMake: 3.20+
• OS: Linux or macOS (Windows supported via MSYS2/vcpkg)

🎯 What's Next

Now that the full APM pipeline builds cleanly and launches reliably, the next phase begins. This document outlines the upcoming milestones that will take the system from a validated prototype to a production‑ready acoustic intelligence engine.

✅ Completed Milestones

• Full DSP pipeline integrated (beamforming → echo cancellation → noise suppression → VAD → translation → projection)
• APMSystem orchestrator implemented and validated
• Clean Docker build with reproducible environment
• CI pipeline green across build and lint stages
• Production launcher with health monitoring
• Automated testing and validation
• Cross-platform startup scripts
• End-to-end encryption (ChaCha20-Poly1305 + X25519 via libsodium)
• Push-to-Talk (PTT) controller with keyboard/mouse/external/software modes
• UDP call signaling with peer discovery and session management
• PortAudio real audio I/O (live microphone and speaker support)
• Local Whisper + NLLB translation (200+ languages, fully offline)

🚀 Next Milestones

See docs/ROADMAP.md for detailed roadmap including:

1. Ring buffers & low-latency audio pipeline - Continuous real-time capture with PortAudio
2. TTS integration - Complete ASR → NMT → TTS speech-to-speech chain
3. DSP Optimization - SIMD acceleration, FFT-based beamforming
4. Developer Experience - CLI tools, config presets, documentation

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Author: Don Michael Feeney Jr.
• Dedicated to: Marcel Krüger
• Enhanced with: Claude (Anthropic), Google (Jules), Microsoft (Copilot)
• FFT: FFTW library by Matteo Frigo and Steven G. Johnson
• ML Framework: TensorFlow Lite by Google

I would like to acknowledge Microsoft Copilot, Anthropic Claude, Google Jules, and OpenAI ChatGPT for their meaningful assistance in refining concepts, improving clarity, and strengthening the overall quality of this code.

📚 References

1. Van Trees, H. L. (2002). Optimum Array Processing. Wiley-Interscience.
2. Benesty, J., et al. (2007). Springer Handbook of Speech Processing. Springer.
3. Paliwal, K. K., et al. (2010). "Speech Enhancement Using a Minimum Mean-Square Error Short-Time Spectral Modulation Magnitude Estimator." Speech Communication.
4. Valin, J. M. (2018). "A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement." IEEE MMSP.

📖 Citation

If you use this work in research, please cite:

@software{feeney2026apm,
  author = {Feeney, Don Michael Jr.},
  title = {Acoustic Projection Microphone System},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/dfeen87/Acoustic-Projection-Microphone-System}
}

📧 Support

• Email: dfeen87@gmail.com
• GitHub: Discussion Board
• Health Check: node scripts/healthcheck.js
• Logs: Enable with DEBUG=1 npm start

Status: Software baseline verified locally and in CI; delivery/hardware gaps are recorded in the Pass B report | Version: 11.0.0 | Last Updated: October 2026

Enterprise Consulting & Integration

This architecture is fully open-source under the MIT License. If your organization requires custom scaling, proprietary integration, or dedicated technical consulting to deploy these models at an enterprise level, please reach out at: dfeen87@gmail.com