Running a GGUF Model on Android via Termux in 10 Minutes

Running language models on local hardware does not have to stop at your desktop or laptop. Modern smartphones pack surprisingly capable ARM SoCs and plenty of RAM to run small, quantized language models directly on-device.

Using an Android phone (in my setup, a vivo V27 with 8 GB RAM) and Termux, you can compile llama.cpp and serve an OpenAI-compatible endpoint right from your pocket.

Here is a quick walkthrough to get a model running via the command line in about 10 minutes.

1. Prerequisites and Installation

First, ensure you have Termux installed from F-Droid (avoid the deprecated Play Store build, as its package repositories are frozen).

Update packages and install the core build toolchain:

pkg update && pkg upgrade -y
pkg install git cmake clang wget -y

2. Compiling llama.cpp for ARM NEON

Clone the repository and build the binaries using CMake. Clang on Termux automatically leverages ARM NEON optimizations for matrix multiplications:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build
cmake ..
cmake --build . --config Release -j $(nproc)

Once the build finishes, you will have executable binaries inside build/bin/, notably:

• llama-cli (interactive terminal chat)
• llama-server (HTTP server exposing OpenAI-compatible endpoints)

3. Downloading a Compact GGUF Model

For an 8 GB Android device, a 1.5B to 3B parameter model quantized to 4-bit (Q4_K_M) strikes the best balance between coherent output, low memory footprint, and respectable token generation speed (typically 12–20 tokens/sec on modern mid-range chips).

A reliable choice is Qwen2.5-1.5B-Instruct-GGUF:

# Navigate to a models folder
mkdir -p ~/models && cd ~/models

# Download Qwen 2.5 1.5B Instruct Q4_K_M (~1.1 GB)
wget https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF/resolve/main/qwen2.5-1.5b-instruct-q4_k_m.gguf

4. Running Interactive Chat

Test the model directly inside your Termux terminal session:

cd ~/llama.cpp/build/bin

./llama-cli \
  -m ~/models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -p "You are a concise engineering assistant." \
  -cnv \
  -t 4 \
  -c 2048

Key flags explained:

• -cnv: Runs in conversational/chat mode.
• -t 4: Sets thread count. Mid-range mobile CPUs typically perform best when matching physical performance cores (usually 4 cores). Setting this too high increases thermal throttling without speeding up generation.
• -c 2048: Allocates a 2048-token context window to preserve mobile RAM.

5. Exposing an HTTP Server (127.0.0.1:8080)

If you want to query the model from other Android apps, Python automation scripts in Termux, or test local HTTP requests, start llama-server:

./llama-server \
  -m ~/models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -t 4 \
  -c 2048

You can now hit the endpoint from another Termux session via curl:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Explain DNS lookup in two bullet points."}
    ],
    "temperature": 0.2
  }'

Realistic Observations & Constraints

1. Thermals & Battery: Sustained generation over long prompts will warm up the device. Mobile OS task schedulers will throttle CPU clock speeds if temperatures climb. Keeping batch sizes and context limits reasonable prevents aggressive throttling.
2. Android OOM Killer: Android aggressively kills background tasks consuming substantial RAM. To keep Termux alive during inference, enable "Acquire Wakelock" from the Termux notification drawer and exclude Termux from aggressive battery optimization in Android system settings.
3. Model Sizing Sweet Spot: While a 7B model can theoretically load in 8 GB RAM using high quantization, it leaves almost no headroom for Android system services, resulting in frequent SIGKILL crashes. Stick to 1B–3B models for reliable on-device execution.

Conclusion

Running llama.cpp on Android demonstrates how accessible local inference has become. It requires zero cloud subscriptions, works completely offline, and provides a private testing sandbox for lightweight edge automation directly on your phone.