Building a Raspberry Pi Voice Assistant With AI
A Raspberry Pi with a camera and microphone is a surprisingly capable base for a small interactive AI assistant. The goal is not to build a full Alexa replacement immediately, but to create a practical voice interface that can listen, understand, respond, speak, and eventually see. The most efficient approach is to let the Raspberry Pi handle local device interaction while using cloud AI services for the expensive language and speech processing.
The project can start simple: press a key, speak into a microphone, send the audio for transcription, pass the text to an AI model, then play the spoken response through a speaker. From there it can grow into a wake-word assistant, camera-aware helper, local automation controller, or even a small household AI terminal.
What the Project Is
The project is an AI-powered voice assistant running on a Raspberry Pi.
At a high level, it does this:
- Captures speech from a USB microphone.
- Converts speech to text.
- Sends the text to an AI model.
- Receives a response.
- Converts the response to speech.
- Plays the answer through a speaker.
- Optionally captures images from a camera for visual questions.
The design is intentionally hybrid. The Raspberry Pi does the local hardware work, while the AI model runs in the cloud.
That gives the best balance of cost, performance, and simplicity.
Why Not Run Everything Locally?
A Raspberry Pi can do a lot, but it is not ideal for running a modern large language model locally. Small models may run, but they are usually slow, limited, or frustrating for natural conversation.
Speech recognition is also computationally expensive. Lightweight local models can work, but they may be slower or less accurate than cloud-based speech-to-text services.
The better early design is:
Raspberry Pi = local appliance
Cloud AI = intelligence layer
That means the Pi handles microphones, speakers, camera, wake word detection, and local actions. The cloud handles transcription, reasoning, and advanced language understanding.
This keeps the device responsive without needing expensive hardware.
Basic Architecture
flowchart TD
A["User speaks"] --> B["USB microphone"]
B --> C["Raspberry Pi audio capture"]
C --> D["Speech-to-text service"]
D --> E["AI chat model"]
E --> F["Text response"]
F --> G["Text-to-speech engine"]
G --> H["Speaker output"]
H --> I["User hears response"]
The important design decision is that not every stage has to use the same provider. For example, the project could use one service for speech-to-text, another for the AI model, and a local engine for text-to-speech.
That makes the system flexible and cost-conscious.
Recommended Version 1
The first version should avoid unnecessary complexity.
A practical first milestone is a push-to-talk assistant:
Press Enter → record speech → transcribe → ask AI → speak answer
This avoids wake-word detection, continuous listening, and streaming audio. Those can be added later once the basic loop works.
Version 1 Components
| Component | Recommended Choice | Reason |
|---|---|---|
| Operating system | Raspberry Pi OS | Stable, common, well-supported |
| Language | Python | Simple hardware and API integration |
| Microphone | USB webcam microphone or USB mic | Raspberry Pi headphone jack is output-only |
| Camera | USB camera or Pi camera | Optional at first |
| Speech-to-text | Cloud STT, such as OpenAI Whisper-style transcription | Better accuracy and lower Pi workload |
| AI model | Cost-effective cloud model | Faster and better than local Pi models |
| Text-to-speech | Piper running locally | Free, fast, and avoids paying for every spoken reply |
| Audio playback | ALSA / aplay |
Simple and already available on Raspberry Pi OS |
Important Hardware Lesson
A standard Raspberry Pi headphone jack does not accept microphone input. It is for audio output only.
That means a phone-style headset with an inline microphone will not appear as a capture device.
To capture speech, one of these is needed:
- USB microphone
- USB webcam with built-in microphone
- USB sound card with microphone input
- I2S microphone module
- Microphone HAT or array board
For a simple assistant, a USB webcam microphone or USB microphone is the easiest option.
Once the USB camera/microphone is plugged in, the Pi should show a capture device:
arecord -l
Example output:
**** List of CAPTURE Hardware Devices ****
card 3: gadget [USB Webcam gadget], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0
That tells us the microphone is available as:
plughw:3,0
A direct recording test can then be done with:
arecord -D plughw:3,0 -f S16_LE -r 16000 -c 1 -d 5 test.wav
aplay test.wav
If the recording plays back clearly, the microphone is working.
Proposed Tech Stack
flowchart LR
subgraph Hardware["Hardware"]
Mic["USB microphone"]
Cam["USB / Pi camera"]
Speaker["Speaker or headphones"]
end
subgraph Pi["Raspberry Pi"]
Python["Python assistant app"]
Audio["ALSA audio capture/playback"]
Piper["Piper local TTS"]
Skills["Local skills / commands"]
end
subgraph Cloud["Cloud AI Services"]
STT["Speech-to-text"]
LLM["AI chat model"]
Vision["Vision model"]
end
Mic --> Audio
Audio --> Python
Python --> STT
STT --> LLM
LLM --> Python
Python --> Piper
Piper --> Speaker
Cam --> Python
Python --> Vision
Vision --> LLM
Python --> Skills
The Raspberry Pi becomes the physical interface. The AI services provide the intelligence. Local skills can be added later for actions that should not need cloud reasoning.
Software to Install on the Raspberry Pi
Assuming Raspberry Pi OS is already installed, updated, and accessible via SSH, the basic dependencies are:
sudo apt update
sudo apt upgrade -y
sudo apt install -y \
git curl wget unzip jq \
python3 python3-venv python3-pip \
portaudio19-dev python3-pyaudio \
libasound2-dev alsa-utils \
ffmpeg sox \
mpg123 \
espeak-ng
For Python:
mkdir -p ~/pi-ai-assistant
cd ~/pi-ai-assistant
python3 -m venv --system-site-packages .venv
source .venv/bin/activate
pip install --upgrade pip wheel setuptools
pip install \
openai \
sounddevice \
scipy \
numpy \
python-dotenv \
webrtcvad \
requests
For camera support on Raspberry Pi OS:
sudo apt install -y python3-picamera2
The --system-site-packages option is useful because some Raspberry Pi camera packages are installed through the OS package manager rather than through pip.
Local Text-to-Speech With Piper
For cost control, spoken output should ideally be local.
Piper is a good fit because it runs locally and can generate natural enough speech without sending every AI response to a cloud text-to-speech provider.
A typical folder layout might look like this:
~/pi-ai-assistant/
├── assistant.py
├── camera_test.py
├── .venv/
├── piper/
│ ├── piper/
│ │ └── piper
│ └── voices/
│ ├── en_GB-semaine-medium.onnx
│ └── en_GB-semaine-medium.onnx.json
└── snapshots/
The assistant can send text into Piper and play the resulting .wav file using aplay.
Core Voice Flow
The first working Python version only needs a few functions:
flowchart TD
A["Start assistant"] --> B["Wait for user input"]
B --> C["Record 6 seconds of audio"]
C --> D["Save temporary WAV file"]
D --> E["Send audio for transcription"]
E --> F["Send text to AI model"]
F --> G["Receive answer"]
G --> H["Generate speech locally with Piper"]
H --> I["Play audio response"]
I --> B
This is deliberately simple. It avoids the harder problem of knowing exactly when the user has finished speaking.
Later, the fixed six-second recording can be replaced with voice activity detection.
Basic Assistant Script
A minimal assistant can be structured like this:
import os
import subprocess
import tempfile
from pathlib import Path
import sounddevice as sd
from scipy.io.wavfile import write
from openai import OpenAI
client = OpenAI()
BASE_DIR = Path.home() / "pi-ai-assistant"
PIPER_BIN = BASE_DIR / "piper" / "piper" / "piper"
PIPER_MODEL = BASE_DIR / "piper" / "voices" / "en_GB-semaine-medium.onnx"
SAMPLE_RATE = 16000
RECORD_SECONDS = 6
INPUT_DEVICE = None
def record_audio(path: str):
print(f"Recording for {RECORD_SECONDS} seconds...")
audio = sd.rec(
int(RECORD_SECONDS * SAMPLE_RATE),
samplerate=SAMPLE_RATE,
channels=1,
dtype="int16",
device=INPUT_DEVICE,
)
sd.wait()
write(path, SAMPLE_RATE, audio)
print("Recording complete.")
def transcribe_audio(path: str) -> str:
with open(path, "rb") as f:
result = client.audio.transcriptions.create(
model="gpt-4o-mini-transcribe",
file=f,
)
return result.text.strip()
def ask_ai(user_text: str) -> str:
response = client.responses.create(
model="gpt-4.1-mini",
input=[
{
"role": "system",
"content": (
"You are a concise voice assistant running on a Raspberry Pi. "
"Answer clearly and briefly."
),
},
{
"role": "user",
"content": user_text,
},
],
)
return response.output_text.strip()
def speak(text: str):
if not PIPER_BIN.exists():
print("Piper not found. Printing only.")
print(text)
return
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
output_path = tmp.name
try:
piper = subprocess.Popen(
[
str(PIPER_BIN),
"--model",
str(PIPER_MODEL),
"--output_file",
output_path,
],
stdin=subprocess.PIPE,
text=True,
)
piper.communicate(text)
subprocess.run(["aplay", output_path], check=False)
finally:
try:
os.remove(output_path)
except FileNotFoundError:
pass
def main():
print("Pi AI Assistant ready.")
print("Press Enter to speak. Type q then Enter to quit.")
while True:
command = input("\nPress Enter to record > ").strip().lower()
if command in {"q", "quit", "exit"}:
break
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
audio_path = tmp.name
try:
record_audio(audio_path)
user_text = transcribe_audio(audio_path)
if not user_text:
print("No speech detected.")
continue
print(f"You: {user_text}")
answer = ask_ai(user_text)
print(f"Assistant: {answer}")
speak(answer)
except Exception as e:
print(f"Error: {e}")
speak("Sorry, something went wrong.")
finally:
try:
os.remove(audio_path)
except FileNotFoundError:
pass
if __name__ == "__main__":
main()
This is not the final product, but it gives the project a working spine.
Once this works, every improvement becomes incremental.
Handling Audio Devices
One practical issue is that Linux audio devices can appear under different card numbers after reboot.
For example, a USB webcam microphone may appear as:
card 3, device 0
So the test command would be:
arecord -D plughw:3,0 -f S16_LE -r 16000 -c 1 -d 5 test.wav
But after reboot, the same device might become card 1 or card 2.
For early testing, this is acceptable. Later, the assistant should use a more stable device name or configuration.
A useful debugging command in Python is:
python - <<'PY'
import sounddevice as sd
print(sd.query_devices())
PY
This lists devices as Python sees them, which helps when choosing the correct microphone input.
Adding the Camera
The camera should not be treated as always-on intelligence. That increases complexity, cost, and privacy risk.
A better design is on-demand vision.
For example:
User: "What do you see?"
Assistant:
1. Captures a still image.
2. Sends the image and question to a vision model.
3. Speaks the answer.
The camera flow looks like this:
sequenceDiagram
participant User
participant Pi as Raspberry Pi
participant Camera
participant Vision as Vision Model
participant AI as Chat Model
User->>Pi: "What do you see?"
Pi->>Camera: Capture snapshot
Camera-->>Pi: snapshot.jpg
Pi->>Vision: Send image with question
Vision-->>Pi: Description / visual answer
Pi->>AI: Combine context and response
AI-->>Pi: Final answer
Pi-->>User: Spoken response
A basic camera test script could save a snapshot:
from picamera2 import Picamera2
import time
camera = Picamera2()
config = camera.create_still_configuration()
camera.configure(config)
camera.start()
time.sleep(1)
camera.capture_file("snapshot.jpg")
camera.stop()
print("Saved snapshot.jpg")
For a USB camera, the implementation may use OpenCV instead of Picamera2.
What the Assistant Could Do
Once the basic voice loop works, the project can become much more useful.
Possible capabilities include:
General Conversation
The assistant can answer ordinary questions:
"What is the difference between RAM and storage?"
"Explain Docker like I'm new to it."
"Give me three ideas for dinner."
Coding Helper
Since the Pi is accessible over SSH and can sit on a desk, it can become a voice-driven development companion:
"Summarise this error."
"What does this command do?"
"Remind me of the Git command to undo the last commit."
Local Device Controller
Local commands can be routed without needing the AI model to invent shell commands.
Examples:
"What is the CPU temperature?"
"How much disk space is left?"
"Restart the service."
"Take a photo."
"Check if Docker is running."
These should be implemented as safe predefined functions, not arbitrary AI-generated terminal commands.
Home Assistant Integration
The Pi could call a Home Assistant API or MQTT broker:
"Turn on the office light."
"Set the room to evening mode."
"Is the garage door open?"
In this design, the AI interprets the request, but the actual action is executed by a controlled integration.
Visual Assistant
With a camera, it can answer visual questions:
"What is on my desk?"
"Can you read this label?"
"Does this cable look plugged in?"
"What does the screen say?"
This could be especially useful as a physical troubleshooting assistant.
Personal Desk Console
It could become a small local AI terminal:
"What am I working on today?"
"Summarise my project notes."
"Create a checklist for setting up this Pi."
"Give me the next three steps."
If connected to local files or a project folder, it could answer questions about specific work.
Skill Routing
A serious assistant should not send every request blindly to a general-purpose model.
Instead, it should classify intent and route requests.
flowchart TD
A["User request"] --> B["Transcribe speech"]
B --> C["Intent router"]
C --> D["General AI chat"]
C --> E["Local system command"]
C --> F["Camera snapshot"]
C --> G["Home automation"]
C --> H["Timer / reminder"]
C --> I["Project notes lookup"]
D --> J["Spoken response"]
E --> J
F --> J
G --> J
H --> J
I --> J
This matters because it keeps the assistant safer, faster, and cheaper.
For example, asking “what time is it?” should not need a cloud AI call. Asking “what is the temperature of the Pi?” should run a local command. Asking “explain the difference between these two architecture choices” can go to the AI model.
Cost Control
The biggest cost risks are:
- Sending continuous audio to the cloud
- Using premium models for every request
- Using cloud text-to-speech for every response
- Sending camera frames continuously
- Allowing long conversations to grow without limits
The cost-effective design is:
Local wake word
Local silence detection
Cloud speech-to-text only after activation
Cheap/fast AI model by default
Local Piper text-to-speech
On-demand camera only
A premium model can still be used for harder questions, but it should not be the default for every interaction.
Development Roadmap
Phase 1: Prove the Audio Loop
Goal:
Record → transcribe → ask AI → speak answer
This confirms that the microphone, API key, Python environment, and speaker output all work.
Phase 2: Add Better Recording
Replace fixed six-second recording with silence detection.
This allows the user to speak naturally without needing to fit into a fixed time window.
Phase 3: Add Wake Word
Add a wake-word engine such as Porcupine or openWakeWord.
The assistant then behaves more like:
"Hey Pi" → listen → answer
This is the first point where it starts to feel Alexa-like.
Phase 4: Add Camera Awareness
Add commands such as:
"What do you see?"
"Take a photo."
"Read this."
This should be on-demand, not continuous.
Phase 5: Add Local Skills
Add controlled local actions:
Check CPU temperature
Check disk space
Restart a known service
Read a local note
Call a local HTTP endpoint
Control Home Assistant
Phase 6: Run as a Service
Once stable, run the assistant under systemd so it starts automatically on boot.
Example service shape:
[Unit]
Description=Raspberry Pi AI Voice Assistant
After=network-online.target sound.target
Wants=network-online.target
[Service]
Type=simple
User=dian
WorkingDirectory=/home/dian/pi-ai-assistant
Environment=OPENAI_API_KEY=your_api_key_here
ExecStart=/home/dian/pi-ai-assistant/.venv/bin/python /home/dian/pi-ai-assistant/assistant.py
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
The API key should eventually be handled more safely than hardcoding it directly in the service file, but this shape is enough to understand the moving parts.
Security Considerations
A voice assistant that can run commands must be treated carefully.
The assistant should not be allowed to execute arbitrary shell commands generated by an AI model.
Instead, use a whitelist:
ALLOWED_ACTIONS = {
"cpu_temperature": get_cpu_temperature,
"disk_space": get_disk_space,
"take_photo": take_photo,
"restart_known_service": restart_known_service,
}
The model can choose from known actions, but the code decides what those actions are allowed to do.
This distinction is important.
Bad design:
User asks → AI writes shell command → Pi executes it
Better design:
User asks → AI identifies intent → Pi runs predefined safe function
Privacy Considerations
The microphone should not constantly stream to the cloud.
A safer design is:
Local wake word detection
Then record
Then send only the captured request
The camera should also be explicit:
Only capture when the user asks a visual question
This keeps the assistant useful without turning it into a constant surveillance device.
Possible Final Form
The finished assistant could become a small desk-based AI companion:
flowchart TD
A["Raspberry Pi AI Assistant"] --> B["Voice chat"]
A --> C["Camera-based visual help"]
A --> D["Local system tools"]
A --> E["Home automation"]
A --> F["Project assistant"]
A --> G["Reminders and timers"]
A --> H["Developer helper"]
B --> B1["Ask questions"]
B --> B2["Have short conversations"]
C --> C1["Describe scene"]
C --> C2["Read text"]
C --> C3["Troubleshoot physical objects"]
D --> D1["CPU temperature"]
D --> D2["Disk space"]
D --> D3["Service status"]
E --> E1["Lights"]
E --> E2["Sensors"]
E --> E3["MQTT / Home Assistant"]
F --> F1["Read local notes"]
F --> F2["Summarise project state"]
F --> F3["Suggest next steps"]
The useful part is not merely that it talks. The useful part is that it bridges the physical room, local devices, project context, and cloud intelligence.
What Should Be Built First
The correct first version is intentionally modest.
Build this first:
USB microphone
Python script
Cloud speech-to-text
Cloud AI response
Local Piper speech output
Manual press-to-talk loop
Only after that works reliably should the project add:
Wake word
Silence detection
Camera vision
Local action routing
systemd startup
Home Assistant integration
That progression avoids wasting time on complicated assistant behaviour before the basic hardware and software path is proven.
Conclusion
A Raspberry Pi AI assistant is very achievable if the project is built in layers. The Pi should not try to be the entire AI system. It should be the local appliance: microphone, speaker, camera, wake word, and safe local actions. The cloud AI service should handle the heavy language and speech intelligence.
The most efficient and cost-effective path is:
Start simple.
Use the cloud where the Pi is weak.
Run locally where cloud cost is unnecessary.
Add intelligence in controlled layers.
That gives a practical assistant now, with a clear route toward something much more capable later.