Benchmarks & Profiling
Deterministic latency and VRAM allocation statistics
Empirical Real-Device Multi-Model Fleet Scorecard (Vulkan GPU & CPU Ground Truth)
Measured on physical Android devices under unrooted Termux Bionic libc environments across canonical 1.58-bit model families. All values represent verified hardware ground truth under steady thermal equilibrium with zero mocking, zero silent fallback, and validated per-device semantic outputs.
| Target Model | Size | Test Device & Hardware | Backend / Layers | Prompt Eval | Token Speed | Hardware Output Ground Truth | Status |
|---|---|---|---|---|---|---|---|
| BitNet-2B | 1.13 GB | Galaxy S25 (Snapdragon 8 Elite) | GPU Vulkan (30/30) | 180.8 ms | 19.37 tok/s | "Paris. The Eiffel Tower, located in the heart of Paris," |
PASS |
| BitNet-2B | 1.13 GB | Galaxy S20 (Turnip Adreno 650) | GPU Vulkan (30/30) | 601.3 ms | 7.71 tok/s | "Paris.\nIt is located in the region of Île-de-France," |
PASS |
| BitNet-2B | 1.13 GB | Galaxy A35 (Exynos 1380 Mali-G68) | GPU Vulkan (30/30) | 1,222.6 ms | 4.22 tok/s | " Paris. London has a very long name, but its people are friendly and" |
PASS |
| BitNet-2B | 1.13 GB | Galaxy A53 (Exynos 1280 Mali-G68) | GPU Vulkan (30/30) | 1,603.9 ms | 3.26 tok/s | " Paris. Cathy has a lot more money than David does. The new" |
PASS |
| Falcon-E-1B | 636 MB | Galaxy S25 (Snapdragon 8 Elite) | GPU Vulkan (24/24) | 130.0 ms | 34.35 tok/s | "situated in Paris, a city known for its rich history and cul" |
PASS |
| Falcon-E-1B | 636 MB | Galaxy S20 (Turnip Adreno 650) | GPU Vulkan (24/24) | 457.6 ms | 10.76 tok/s | "located in the Southern Italy, bordered by the Danube (D" |
PASS |
| Falcon-E-1B | 636 MB | Galaxy A35 (Exynos 1380 Mali-G68) | GPU Vulkan (24/24) | 904.1 ms | 5.81 tok/s | " a city in the heart of art and culture, where history and innovation meet" |
PASS |
| Falcon-E-1B | 636 MB | Galaxy A53 (Exynos 1280 Mali-G68) | GPU Vulkan (24/24) | 1,234.9 ms | 4.46 tok/s | " a complex and often subject to numerous laws and regulations, with different types of" |
PASS |
| Falcon3-7B | 3.05 GB | Galaxy S25 (Snapdragon 8 Elite) | GPU Vulkan (28/28) | 474.3 ms | 8.30 tok/s | "a sovereign state situated mainly in Western Europe..." |
PASS |
| Falcon3-7B | 3.05 GB | Galaxy S20 (Turnip Adreno 650) | GPU Vulkan (-ngl 8) | 2,194.0 ms | 2.62 tok/s | "a large city in the north-west region..." |
PASS |
| Falcon3-7B | 3.05 GB | Galaxy A53 (Exynos 1280, 6GB RAM) | GPU Chunked (4) | 7,976.8 ms | 1.74 tok/s | " the only power that can be considered to exist for" |
PASS |
| Falcon3-7B | 3.05 GB | Galaxy A35 (Exynos 1380, 6GB RAM) | GPU Chunked (4) | 15,820.1 ms | 0.78 tok/s | " the largest city in Europe by population, and its" |
PASS |