Transcribe and Speak: Self-Hosted Speech AI on a VPS

Part 11 of the AI series. Speech is the area where a plain CPU server does best. If you are new to the vocabulary, read part 1 first.

Of all the things you can self-host in AI, speech is the one that feels most like a bargain. Turning recordings into text, and text into spoken audio, both work well on an ordinary processor. There is no graphics card involved, the models are small, and the results are good enough to use for real. You can transcribe interviews, meetings, lectures and voice notes without sending a word to anyone, and have your own server read things aloud. This guide covers both directions: which models to choose, a working transcription script, how to get better results, and what to watch out for.

What is in this guide

• Why speech suits a CPU
• Speech to text: the Whisper family
• Choosing a model size
• A transcription script
• Getting better transcripts
• Who said what?
• Text to speech
• Ten uses worth building
• Consent, licences and cloning
• Which plan do you need?

Why speech suits a CPU

The big language models are limited by memory. Speech models are small: the useful ones range from tens of megabytes to a few gigabytes. Transcription is also a batch job. Nobody is waiting for a recording to finish, so a server that takes twice as long as the audio is still useful if it runs overnight. Text-to-speech engines such as Piper were designed to run on weak hardware. These are the jobs where self-hosting makes the most sense.

Speech to text: the Whisper family

Whisper is an open speech-recognition model that handles many languages, accents and noisy recordings. It comes in several sizes. Two re-implementations make it run well on a CPU:

• faster-whisper is a Python library that runs Whisper much faster than the original and in less memory, using 8-bit arithmetic. It is the easiest to script.
• whisper.cpp is a C++ version with no Python needed, and it is popular for small machines.

Both download the model for you or from the project's instructions. The first run needs an internet connection to fetch it.

Choosing a model size

Size	Parameters	Character
tiny	about 39 million	Very fast, rough. Good for quick drafts of clear speech
base	about 74 million	Fast, fair accuracy. A good first try on any VPS
small	about 244 million	The usual best balance for a CPU. Noticeably better with accents and noise
medium	about 769 million	More accurate, several times slower. Suits a strong plan or patience
large	about 1.5 billion	Best accuracy, heavy. Realistic mostly on a dedicated server, and slow

There are also "turbo" and distilled variants of the larger models that trade a little accuracy for a lot of speed, and English-only versions of the smaller ones that are a bit more accurate for English. Names and versions change, so check the project's current list.

How fast is it? It depends on the CPU, so measure it. Transcribe a one-minute clip and see how long it takes: the script below prints the ratio. If a model takes about the same time as the audio, an hour of recording takes an hour. Anything under about two times real time is comfortable for overnight work.

A transcription script

Set up a clean Python environment and install the library:

sudo apt install python3-venv ffmpeg
python3 -m venv ~/speech
~/speech/bin/pip install faster-whisper

Save this as transcribe.py:

#!/usr/bin/env python3
"""Transcribe an audio or video file to text and subtitles.
Usage: python3 transcribe.py recording.mp3 [model] [language]
Example: python3 transcribe.py meeting.mp4 small en
"""
import os, sys, time
from faster_whisper import WhisperModel


def stamp(seconds):
    ms = int(round(seconds * 1000))
    h, ms = divmod(ms, 3600000)
    m, ms = divmod(ms, 60000)
    s, ms = divmod(ms, 1000)
    return "%02d:%02d:%02d,%03d" % (h, m, s, ms)


def main():
    if len(sys.argv) < 2:
        print(__doc__)
        return
    path = sys.argv[1]
    size = sys.argv[2] if len(sys.argv) > 2 else "base"
    language = sys.argv[3] if len(sys.argv) > 3 else None
    model = WhisperModel(size, device="cpu", compute_type="int8")
    started = time.time()
    segments, info = model.transcribe(path, language=language, vad_filter=True)
    base = os.path.splitext(path)[0]
    with open(base + ".txt", "w", encoding="utf-8") as txt, \
         open(base + ".srt", "w", encoding="utf-8") as srt:
        for n, seg in enumerate(segments, 1):
            text = seg.text.strip()
            txt.write(text + "\n")
            srt.write("%d\n%s --> %s\n%s\n\n" % (n, stamp(seg.start), stamp(seg.end), text))
            print("%s %s" % (stamp(seg.start), text), flush=True)
    audio = info.duration
    took = time.time() - started
    print("Audio %.0f s, took %.0f s (%.2fx real time)" % (audio, took, took / max(audio, 1)))


if __name__ == "__main__":
    main()

Run it on a file:

~/speech/bin/python transcribe.py interview.mp3 small en

It prints each line with its time as it goes, then saves interview.txt (plain text) and interview.srt (subtitles that video players and editors understand) next to the original. The last line shows how many times real time it took. Leave out the language to let it detect one. The first run downloads the model.

It works with video files too, since it reads the audio track. Long jobs should run inside tmux or as a background service, so closing your SSH window does not stop them.

Getting better transcripts

• Clean audio matters most. A close microphone and a quiet room beat any model upgrade.
• Set the language if you know it. Detection can guess wrong on short or noisy clips.
• Voice activity detection (turned on in the script) skips silence and reduces made-up text in the quiet parts, which Whisper models are known to produce.
• Give it a hint. The library accepts an initial prompt, such as a list of names and technical terms, which makes it spell them correctly more often.
• Move up one size before anything more complicated. Going from base to small is often the biggest single improvement.
• Proofread important text. Names, numbers and quotes are where errors hide. No model is perfect.
• Post-process with a language model. Ask a local model to fix punctuation, split into paragraphs or summarise. Do it in a second step, after the speech model has finished, so memory is not shared.

Who said what?

Whisper transcribes words but does not label speakers. Separating speakers is called diarisation, and the open tools for it are heavier, more fiddly to set up, and some require accepting model licence terms. If you need it, try it as a separate step on a bigger plan, and expect imperfect results with overlapping voices. For meetings, a cheap trick works well: record each person on a separate track if you can, and transcribe each track on its own.

Text to speech

The other direction turns text into audio. Options for a CPU server:

• Piper is a fast, lightweight engine with many ready-made voices in many languages. It runs in real time or faster even on small machines, and the voices are clear, though a little flat.
• Newer small neural voices (a few open models of well under a billion parameters have appeared) sound more natural and expressive, at the price of more computing. Check each one's licence and whether it runs well without a GPU.

With Piper, install it from the project's own instructions, download a voice (the "medium" quality ones are a good balance), and feed it text. The result is a WAV file you can convert to MP3 with ffmpeg:

ffmpeg -i reading.wav -b:a 96k reading.mp3

For long texts, split them into paragraphs and join the audio afterwards, so one mistake does not ruin an hour of work and memory stays steady.

Ten uses worth building

1. Meeting minutes. Transcribe, then summarise with a local model (see the 6-8 GB guide).
2. Searchable audio. Transcribe a podcast or lecture archive, then index the text with the document search from part 9.
3. Subtitles for your own videos, from the SRT file.
4. Transcripts on your website for your own audio and video, which helps readers and search.
5. Voice notes inbox, as in the automation guide.
6. Dictation for long documents, with a recording app on your phone and a nightly transcription.
7. Articles read aloud, from saved web pages or your own drafts, as audio for a walk or a commute.
8. Spoken alerts from your server or home automation.
9. Language practice, listening to your own speech transcribed, to see what the machine, and so a listener, hears.
10. A voice assistant that joins the pieces: speech in, a language model, speech out. Expect a pause of several seconds on a CPU.

Consent, licences and cloning

• Get consent to record, and know the rules where you live. Recording calls and meetings is regulated in many places.
• Voices have licences. Check the terms of each voice before publishing audio commercially.
• Do not clone a real person's voice without their permission. Some open models can do it from a short sample. Using it to imitate someone is unethical and often illegal, and can be used for fraud.
• Recordings are sensitive. Keep audio and transcripts private, with strong logins and backups, and delete what you no longer need.

Which plan do you need?

• 2 GB: tiny and base models, and Piper. Good for short recordings and voice notes.
• 4 GB: the small model comfortably, and a small language model for clean-up, run one after the other.
• 8 GB: the medium model, plus a 7-8B model for summaries, taking turns.
• A dedicated server: large models, many files in parallel on many cores, and big archives. See the dedicated server AI guide.

More CPU cores make transcription faster, so for speech work, cores matter more than they do for chat models. Compare plans on both.

For more, see the cheapest plans and the sales page.