What is Whisper AI? OpenAI’s speech model explained

Whisper in a nutshell

OpenAI’s README describes Whisper as “a general-purpose speech recognition model” that is “trained on a large dataset of diverse audio.” It is a multitasking model: one model handles multilingual speech recognition, speech translation into English, and language identification.

Whisper isn’t an app. It’s a model, plus code to run it. You use it through software built on top of it, from command-line tools for developers to dictation apps that type into other apps.

Who made Whisper, and when

Whisper was developed by researchers at OpenAI. According to the model card, the original series of models was released in September 2022, followed by large-v2 in December 2022, large-v3 in November 2023, and large-v3-turbo in September 2024.

The research behind it is described in the paper Robust Speech Recognition via Large-Scale Weak Supervision. The model card says the models were built to study the robustness of speech processing systems trained under large-scale weak supervision, and also notes that Whisper is “potentially quite useful” as a speech recognition tool for developers.

How Whisper was trained

The paper explains that Whisper was trained on 680,000 hours of multilingual and multitask audio paired with transcripts collected from the internet. The model card breaks that down: most of the audio is English, and the non-English portion covers 98 languages.

The paper reports that models trained this way generalize well to standard benchmarks without any fine-tuning, and that, compared with humans, they approach human accuracy and robustness. That broad, messy training data is the reason Whisper wasn’t built around one narrow kind of recording.

How Whisper works

Under the hood, Whisper is a Transformer sequence-to-sequence model. Audio goes in, and a decoder predicts the text as a sequence of tokens. The README explains that special tokens tell the model which task to perform, so a single model can replace many stages of a traditional speech-processing pipeline.

In practice, that means an app can ask the same model to transcribe in the spoken language, translate into English, or detect the language first, without bolting together separate systems.

Whisper model sizes

Source: the Whisper README. Smaller models are lighter and faster; larger ones are generally more accurate. The README says the English-only “.en” models tend to perform better for English, especially tiny.en and base.en.

Turbo is an optimized version of large-v3 that, per the README, transcribes faster with minimal loss of accuracy. It isn’t trained for translation, so use one of the multilingual models if you need speech translated into English.

What Whisper does well

• Robustness: the model card says Whisper shows improved robustness to accents, background noise, and technical language compared with many existing speech recognition systems.
• Translation: it can translate speech from multiple languages into English without a separate translation step.
• Running locally: because the weights are openly available under the MIT License, apps can run Whisper on your own hardware instead of sending audio to a server.

Known limitations

OpenAI is upfront about Whisper’s weak spots. The model card warns that, because it was trained on large-scale noisy data, Whisper can produce text that wasn’t actually spoken, which is known as hallucination. It can also generate repetitive text.

Accuracy is uneven. The model card says Whisper performs worse on languages with less training data, and that performance differs across accents and dialects. The README adds that performance varies widely by language. OpenAI recommends evaluating the models in your own context before relying on them, which is good advice for any speech tool: test it with your own voice and vocabulary.

Whisper vs other speech models

Whisper isn’t the only open speech recognition model. NVIDIA’s Parakeet family is another open option, built with NVIDIA’s NeMo toolkit. Our comparison Parakeet vs Whisper covers how the two differ in languages, licenses, and design.

Running Whisper on a Mac

Developers can install OpenAI’s official Python package and run Whisper from the command line. The README lists the setup, which includes installing the ffmpeg tool, for example with Homebrew on macOS. The README also lists approximate memory needs for each model size, so larger models ask more of your Mac.

Most people don’t need to touch the command line. Many Mac apps bundle Whisper models and run them for you. Because the model runs on your Mac, your audio doesn’t have to leave it. Read on-device vs cloud dictation to see why that matters, and speech to text on Mac for the other options.

Where Morning fits

Morning is private dictation for Mac that lives in the notch, and it transcribes on-device with local Whisper models. Hold Control + Space, speak, release, and your text pastes where you’re typing. Optional cloud AI is off by default, and it works on macOS 14 or later.