Press Record on your phone to capture the sounds around you — up to 10 seconds. When you stop, the audio is sent from your phone to a server running a sound classification model. The server analyses the recording and returns the most likely sound categories for the soundscape you selected. Results and a mel spectrogram of the recording are shown directly on this page.
Urban sound classification is a branch of machine learning and environmental acoustics that automatically identifies sound events occurring in city and indoor environments. By analyzing short audio recordings, a classifier can detect sounds such as traffic, sirens, speech, and machinery, turning raw audio into structured labels. Applications range from noise pollution monitoring and smart city dashboards to accessibility tools and ecological research.
A mel spectrogram is a two-dimensional representation of an audio signal in which the horizontal axis shows time, the vertical axis shows frequency, and color encodes amplitude. The frequency axis follows the mel scale, which mimics the non-linear way the human ear perceives pitch — spacing lower frequencies further apart than higher ones. This perceptual scaling makes mel spectrograms well-suited as visual features for audio machine learning models, since they highlight the aspects of sound that matter most for human hearing.
CLAP (Contrastive Language–Audio Pretraining) is a model developed by LAION that jointly embeds audio and text into a shared vector space, similar to how CLIP works for images and text. During zero-shot classification, the model compares the embedding of an incoming audio clip against embeddings of candidate text labels, and returns the best match — without requiring any labelled audio examples per class. This app uses the laion/larger_clap_general checkpoint, trained on a large and diverse collection of audio–text pairs.
We classify sounds in categories based on the selected soundscape. A soundscape is a collection of sounds that occur in a particular environment, such as the city or the home. As we have a dedicated sound sensor on Amsterdam's Marineterrein, this soundscape is also available here. More on soundscapes at Wikipedia.