Pro Audio Files

Train Your Ears Become a Member

The Basics of Sound Design (+ Essential Software)

Article Content

The notion of sound design as we know it has evolved over the last several decades to include musical, gaming, theatrical as well as cinematic applications. Dan Dugan, of the American Conservatory Theater (ACT) in San Francisco, was the first person to be called a sound designer. He devised ways of automatically adjusting levels to reduce feedback. In film, Walter Murch was the first person given the title of sound designer for his work on Apocalypse Now, a masterpiece of sound design and re-recording mix engineering. The helicopter scene is incredible! I have been privileged to collaborate with the great Don Preston in my group, TriAngular Bent. Don is an original member of Frank Zappa’s Mothers of Invention and the person who created the helicopter sound on his Moog for the film.

But even before the phrase existed, people have been using all sorts of mechanical and electronic methods to “design” new sounds. Luigi Russlo comes immediately to mind as the inventor of the intonarumori, a collection of noise-based machines for creating new kinds of sounds in the early 20th Century.

Luigi Russlo

Luigi Russlo

Sound design is now accessible and possible for anyone with a computer. In this article, I will focus on some of the fundamental principles and approaches to sound design, audio processing plugins used by the pros, common applications and related career paths.

Fundamental Principles and Approaches to Sound Design

Purpose

The function of a sound (or lack thereof) will dictate the methods used to create it in many circumstances. For example, if a sound designer is tasked with creating the sound of a fictional monster there is clearly no one sample that will suffice. Commonly, several animal recordings with various types of processing may be combined with synthesized sounds or even instrument samples. There are no rules for creating something that doesn’t exist. But there are expectations that are now embedded in the culture of media entertainment. Gunshots in films do not sound “realistic,” but rather they sound hyperrealistic with long reverb tails and perfectly balanced frequency spectrums. Even sports and nature are not immune to hyperrealistic sound design.

Enhancing a Narrative

One basic function of sound design is to enhance the narrative, whether it be for film, theater, gaming, etc. Some may suggest that sound is subservient to the image, but I prefer the word supportive. No one would deny that bad sound is one major hallmark of a bad film. While great sound might not be able to salvage a bad story, it will certainly make it less painful.

Sound in gaming is a crucial part of the immersive experience. Sound design creation and implementation in the gaming industry is one of the most, if not the most, viable career paths for aspiring audio content producers. In this context, procedural or adaptive audio is used to realize an experience defined by user actions. While the sound design content is created in a similar way to traditional mediums, the implementation is non-linear and requires the use of middleware such as Wwise or FMOD to implement audio assets.

The idea that sound needs to adapt to the actions of the user in realtime presents significant and unique challenges for designers due to the need to consider a dynamic spatial relationship between the user and the sound object. The more realistic the sounds as they relate to the imagery, the more immersive the experience. VR (virtual reality) adds yet another layer of complexity as head movements need to be considered as well.

Sound for live theater might be considered the ancestor of modern sound design. Various mechanical devices have been used for thousands of years in theatrical productions. The Greeks invented various devices to create storm sounds that used brass balls and fabric draped over a rotating wheel.

In modern theater productions, applications such as QLab or Ableton Live are used to trigger precomposed elements of music and sound design to enhance the theatrical action on stage. This could be as simple a dog barking in the distance, the sound of an urban soundscape or a gunshot.

Musical Context

In musical contexts, sound design can enhance a gesture or create novel textures not possible with traditional acoustic or electronic instrumentation. Using orchestral sounds combined with synthesized or sampled found objects is a way of creating hybrid sonorities that can be unique and unidentifiable while remaining linked to the real world in timbre and playability.

Synth programming, with preset and sample pack creation is another possible income source for sound designers. While amazing instruments such as Omnisphere, Serum and u-he’s line of incredible synths offer enormous sonic variety and potential for composers, many don’t have the time or inclination to fully explore these products. This creates an opportunity for sound designers to fill the void.

Sampling instruments are widely used in sound design since they can provide immediate access to many sounds from a simple MIDI keyboard or drum pad. Literally any sound can be mapped, layered, pitched, looped, sequenced and modulated using any one of a number of available samplers such as Native Instrument’s Kontakt, Sampler and Quick Sampler in Logic Pro, Sampler and Simpler in Ableton Live, etc. The market for third-party sampling instruments has exploded in the last decade for composers lacking the will or resources to create their own instruments, resulting in yet another opportunity for sound designers and virtual instrument makers.

The use of real-time sampling, DJ gear, alternative MIDI controllers, custom electronics, circuit bent instruments and laptops in live music performance can arguably be considered a form of live sound design. You could say the relationship between musical improvisation and the performance of pre-composed material is analogous to traditional sound design and performance-based work.

Sound for Sound’s Sake

This phrase relates to the idea of art for art’s sake, a phrase purportedly coined by the French poet Théophile Gautier in the early 19th century “that expresses the philosophy that the intrinsic value of art, and the only ‘true’ art, is divorced from any didactic, moral, political or utilitarian function.” Extrapolating this idea we might consider creating or designing sounds simply for the pure joy of it. Taking the idea even further, we could imagine sound being freed from traditional musical relationships and form. Karlheinz Stockhausen’s idea of “moment form,” the work of John Cage, and Pierre Schaeffer’s musique concrète provide numerous examples that can redefine our understanding of sound and how it can be appreciated independent of familiar musical constructs.

To say that sound for sound’s sake, without a corresponding teaching position, is a viable career path would be disingenuous. This is a creative direction that requires a day gig (at least for now). But these ideas have surely influenced contemporary sound design and mainstream music, and have gradually seeped into the consciousness of popular culture. Knowing the history of your medium is essential. With accessibility to landmark works that changed the course of music and our perception of sound just a few clicks away, there is every reason to explore.

Sound Design Techniques

Layering and Envelopes

It is logical to propose that every sound we hear has a beginning, a middle and an end. In a sound design context this is a useful way of thinking about, deconstructing and constructing sounds. Similar to the idea of a four-stage ADSR envelope (attack, decay, sustain and release), we can consider each stage as just one component of a sonic event. In terms of amplitude, the attack, decay and release stages have a temporal component while the sustain portion is more about level. But things can be much more complex than using simple ADSR amplitude envelopes to construct a sound. Envelopes can be sources of modulation for any parameter in the signal chain including source generation, filtering and other modulators. Envelopes can have an infinite number of stages and can include looping algorithms, sustain points and rhythmic flexibility. While combining or layering several sounds with a shared amplitude envelope can yield amazing results, employing unique envelopes for each sound component can produce far more complex and evolving sonic textures or rich timbres not otherwise possible.

Deconstruction and Reconstruction

The example of FFT processing (Fast Fourier Transform) is worth considering at this point. It involves deconstructing a sound so that it can be represented by a collection of sine waves describable in terms of frequency, amplitude and relative phase. Once deconstructed, mathematical modifications can be made to particular components independently. This is how pitch can be shifted without changing duration and vice versa. Granular synthesis is a similar process whereby grains or little chunks of a sound can be treated independently in myriad ways such as pitch shifting, reversing and reordering.

Consider the only pure sound in terms of harmonics (the sine wave), which includes just one measurable frequency — the fundamental. As soon as you combine just two of these pure waveforms you produce sum tones, difference tones and possible tremolo effects. Adding varying envelopes into the mix causes things get even more complex.

This idea of achieving complexity from a few simple components also lies at the heart of simple FM (frequency modulation) synthesis, where two simple oscillators (a carrier and a modulator) can produce an astounding array of sounds and timbres.

Synthesis Versus Sampling – Hybrid Approaches

The combination of real-world samples and synthesized sounds is ripe with possibilities for sound design. Using the layering and envelope ideas mentioned above as well as additional processing, you can create engaging and evolving soundscapes and hybrid gestural events that incorporate both organic and electronic-based sources. This lends itself nicely to genres like sci-fi and gaming.

Analog and Digital

The resurgence of the analog aesthetic has been underway for many years now and shows no sign of slowing down. In 2020, vinyl sales surpassed CDs for the first time since 1986. Modular synthesis has been enjoying a renaissance since the introduction of the standardized Eurorack format by Doepfer in the late 90’s. And analog sounds are being used more and more for music and sound design. Having been raised in an analog world, the digital revolution seemed miraculous at the time and I never envisioned a return to old technology. But now, my studio is riddled with both analog gear and hybrid devices. After being immersed in digits for decades it was a breath of fresh air to hear pure continuous voltage again. Now I allow myself to explore both worlds independently and perhaps more interestingly, in combination. Both technologies have a particular sound, as do different instruments, and I see no reason to use one exclusive of the other.

Spatialization

One of the most fundamental aspects of sound design is creating a space for a sound to exist or creating a sonic environment. Two temporal processes to consider are reverberation and delay, both of which can include some sort of filtering, panning, and modulation. Putting aside surround sound and Dolby’s ATMOS systems for the moment, amazing results can be accomplished in a standard stereo field. Objects can be moved not just left to right, but front to back. Using binaural algorithms (typically restricted to headphone implementations) such as in Logic Pro, sounds can be located not just front/back and left/right, but behind, below and above the listener. Doppler effects can be applied to movement in the stereo field to heighten realism or create hyperreal motion.

The contrast between sampled and synthesized sounds can be correlated in some ways with the difference between convolution and algorithmic reverb. The former uses impulse response (IR) files derived from samples taken in actual spaces while the latter relies on algorithmic computation to create spaces based on delay lines, filtering and other parameters. Convolution reverbs are arguably more appropriate for placing sounds in “real” spaces while algorithmic reverbs can be used to emulate vintage reverb technologies like plate and spring reverbs, or create impossible spaces that don’t exist in real life. Convolution reverbs and IR files can also be used to emulate vintage gear, speaker cabinets, amp circuits, etc. Algorithmic reverbs typically have more control over parameters and can offer extreme settings. Some engineers suggest cascading reverbs (placing them in series) to create smoother results.

The creative use of reverb alone has enormous possibilities in sound design and it’s a well-known fact that everything sounds better with a little verb!

Software

There are so many great applications and audio processing plugins out there that are appropriate for sound design; it would be impossible to mention them all in this context. Below are just a few of my favorites at the moment (in no particular order), all of which I can wholeheartedly recommend.

UVI

NOTE: When this article was first published, there was an immediate comment lamenting the fact that I didn’t include Falcon by UVI in this round-up. I am the first to admit I have not tried every piece of sound design software currently available and I only write about things I have personally used. After the initial publishing date, UVI was gracious enough to provide me a copy of Falcon 2 to explore and I have to say it is an impressive piece of software.

Falcon 2 by UVI can be classified as a hybrid synth since it has so many approaches to sound design in one box including sampler capabilities, granular synthesis, wavetable synthesis, FM synthesis, additive synthesis, subtractive, synthesis, phase distortion synthesis and component modeling. It comes packed with 16 oscillators, 90 full-featured processing effects, 9 modulation generators, scripting processors for MIDI events, an analysis section that includes chord recognition and pitch detection and really way more than can be covered in this short description. With all that functionality, you would expect a bit of a learning curve. Happily, UVI has published several deep-dive YouTube videos that methodically explore this beauty. A quick perusal of the presets will convince you of the enormous potential this instrument offers. The multi-part and multi-layer approach guarantees the possibility of an infinite variety of unique sounds, textures and timbres. It is a sound design behemoth!

UVI Falcon 2

Output

This LA-based company has established a reputation for elegant interface design and superb processing and instrument plugins. Some of my favorites include:

  • Portal a granular-based processor.
  • Thermal offers saturation and distortion.

Kontakt-based Hybrid Instruments:

  • Analog Strings
  • Analog Brass and Winds
  • Arcade is a subscription-based instrument that allows access to a huge collection of loop kits that can be downloaded as desired. It employs an easy-to-use interface in which a two octave span of white keys trigger loop slices while the black keys will activate various processing and playback functions. There’s a lot more to it than I can explain in this article, but I suggest you check it out for yourself. Recently they launched Kit Generator, a function that creates a user-defined kit based on samples in the Arcade library or your own via a simple drag and drop. While other samplers can do auto-mapping as well, this feature takes full advantage of Arcade’s robust effects processing and playback manipulation, giving the user instant access and the ability to customize the instrument with a few clicks. Drag and drop one of your own tracks in and hear the magic. Really great!

Output Arcade

Native Instruments

The ramifications of the recent merger of iZotope and Native Instruments remain unknown as of this writing, but hopefully they will combine their expertise in ways that benefit users through innovation. Below are a few of my go-to NI instruments and processors:

  • Kontakt the industry standard sampling platform that hosts hundreds of third-party libraries.
  • Rise & Hit contains user-programmable whooshes, impacts and transition effects.
  • Driver  a unique saturation and distortion plugin with modulation possibilities.
  • Kinetic Metal a sample-based instrument capable of raw metallic textures with a steampunk GUI.
  • Polyplex  a Reaktor-based percussion instrument with 8 voices and up to 4 layers per voice.

NI Polyplex

ADVERTISEMENT

iZotope

While NI’s forte has traditionally been focused on virtual instruments, their new partner, iZotope, has spent years carving out niches in the world of artificial intelligence and post-production. In terms of sound design these are my favorites:

  • Trash it’s ugly, it’s nasty, it’s deliciously angry. When you need to give a sound some hard-edged attitude, this thing is perfect!
  • Vocal Synth this plugin uses a variety of traditional vocoder-style processing that can be layered for unusual results. Far more useful than simple vocal processing; try it on rhythmic content or virtually anything and see what happens.
  • Vinyl this a freebie that does exactly as you’d expect: apply lo-fi vinyl effects to any sound.
  • RX8 the industry standard post-production audio restoration suite used by producers, post-production pros, and anyone dealing with noise reduction and sample tweaking.

iZotope RX8

Adobe

While this company is not particularly known for its audio software, I am a big fan of Audition.

Audition this is a responsive waveform editor with onboard noise reduction (using iZotope algorithms), and the ability to apply any plugin on your system. There is also a multitrack function which I find to be a bit too clunky to mess with. But it is my go-to waveform editor and offers great visualizations for amplitude, phase, and spectrum.

Adobe Audition

Apple Logic Pro

Apple released a major update for Logic Pro last year that included some Ableton-esque style features such as Live Loops. They also renamed and revamped their sampler instrument:

ADVERTISEMENT

Sampler (formerly known as the EXS24) is a full-blown sampler with multi-sample and multi-layer possibilities. While there are not anywhere near the number of third party libraries available as there are for Kontakt, Sampler’s re-designed interface is far more accessible and user-friendly when compared to the EXS24 or Kontakt, making it perfect for those wanting to create custom instruments.

Note: be sure not to confuse Logic’s Sampler or Quick Sampler with Ableton’s Sampler or Simpler. You would think with all the brain power over at Apple someone could come up with a few names that were not direct rip-offs.

Alchemy is a powerful sound design tool. It provides several ways to generate sound, access samples, apply effects & filtering and modulate parameters with LFOs, envelopes and step sequencing. Originally developed by Camel Audio before being bought out by Apple, this thing is a behemoth with an enormous library of presets searchable via a browsing matrix.

Apple Alchemy

Heavocity

This company never fails to impress. They make mostly Kontakt-based sampling instruments with sound design-driven content and highly programmable parameters that ensure a unique palette of sounds for any user that takes the time to do a little tweaking. Some of my favorites include:

Damage and Damage 2these are the original and updated versions of this amazing instrument. They’re must-haves for designing cinematic percussion.

Gravity  the ideal device for creating risers, pads, hits and sound design textures.

Heavyocity Gravity

Sample Logic

This is another company that continues to put out solid instruments year after year. I find their stuff particularly useful for creating evolving, hybrid textures of all sorts. Many of their instruments use the approach of morphing between several sample sources with playable XY pad-based GUI designs. A sort of vector synthesis model for sample material.

  • Xosphere 2 great for creating atmospheres and unworldly ambience.
  • Waterharp the waterphone is a sound design staple. If you don’t own one, this is the next best thing.
  • Thunder Springs this is a unique library of spring drum and thunder tube samples.
  • Stadium Instruments morphed stadium sports instruments useable in both musical and sound design contexts.
  • Morphestra based largely on cinematic instrument sounds.

Sample Logic Morphestra

Audio Ease

This company is known for developing arguably the best convolution reverb currently available.

Altiverb 7 the newest release of this recognized leader in convolution technology offers an expanded array of IR (Impulse Response) files captured from concerts halls and performance spaces, plus a wide selection of outdoor environments and interior spaces, scoring stages, studios and vintage reverb gear — all of which can be modified and tweaked to create the perfect space for a sound object to inhabit. In addition, you can import other IRs you make yourself or from third-party vendors. It is the ideal tool for music producers and engineers, post-production specialists and sound designers. The included IR’s were painstakingly recorded by the engineers at Audio Ease from sources all over the world.

The quality and richness possible with Altiverb are truly unmatched — as is the corresponding price. But in this instance you truly do get what you pay for: stunning reverb and spatialization.

Audio Ease Altiverb 7

Zynaptiq

This company makes several unique products specifically developed for sound designers. Their sound design bundle includes:

  • Adapitverb a reverb plugin with pitch shifting and drone generation possibilities.
  • Morph 2 allows morphing between two sources to create hybrid sounds and smooth transitions.
  • Wormhole a multi-effects device for warping and pitch-shifting audio that is immediately satisfying.

Zynaptiq Wormhole

Valhalla DSP

This company specializes in spatialization (say that 5 times fast) and reverbs in particular. They make effective algorithmic reverbs including plates, rooms, halls and all the standard spaces most reverbs have. They also create emulations of vintage gear. Their interfaces are clear and to the point without any fancy graphics or animated fluff, which I assume helps keeps the devices running efficiently in terms of CPU load. They have kept their prices at $50/device, which is really a steal for the quality of these things. My favorites include:

  • Vintage Verb an emulation of old school digital hardware.
  • Plate 1960s – 1970s style plate sounds.
  • Room – rooms, halls, chambers and a bunch of other algorithm-based verbs.
  • Shimmer this one I use a lot to create extremely long glistening tails that can go on forever.

The Basics of Sound Design (+ Essential Software)

It is also worth checking out the handful of free products on their site including Space Modulator and Super Massive.

Soundtoys

This developer has carved out a place in the hearts of sound designers and music producers alike. It’s a great suite of plugs for saturation, spatialization, filtering, pitch-shifting and modulating sounds in many ways. My most used toys are:

  • Decapitator saturation and distortion ranging from subtle to a punch in the face.
  • Tremolator a solid tremolo effect that can be set to groove with customizable amplitude curves, accented patterns and analog emulation effects.
  • Echoboy stereo delay with saturation and filtering that just seems to always sound good regardless of the input.

Be sure to explore the hidden controls on many of the Sound Toys plugins, exposed by clicking a switch sometimes called “Style Edit” or “Tweak.” That is where you can really customize a sound.

Soundtoys Tremolator

ADVERTISEMENT

Cycling 74

As a Max/MSP user for the last 25 years or so, I can say that no discussion on sound design would be complete without mentioning this amazing language.

Max/MSP a graphic object-oriented programming language for creating audio and MIDI processing tools in the form of patches, Max for Live devices (for the Ableton Live environment) and stand-alone applications. There is also a whole set of visual tools for video manipulation and 3D animation called Jitter. In fact, there is little you cannot do with Max/MSP if you have the desire. I use it to write everything from small utility type patches that can interact with my DAW in various ways (generating automation for instance), to full-blown stand-alone apps that are basically DAWs in their own right, but with a unique set of features based on the requirements of my artistic vision at the time. When I cannot achieve the desired effect elsewhere I reach for Max/MSP and it has never failed me. You can check out some miscellaneous demos on my YouTube channel here.

Max MSP

Krotos

This company specializes in sound design effects and offers several plugins that would be particularly useful for those working in gaming or film — particularly horror or sci-fi genres.

Weaponizer while I consider myself a pacifist, I have to admit that sculpting gunshots and weaponry sounds with this effect generator is great fun! You can specify the type of weapon and location in space and adjust the details of the various layers in the sound.

Krotos Weaponizer

Spectrasonics

The company produces the much-revered synth and sample-based instrument used by many serious sound designers and music producers.

Omnisphere this is an instrument that includes access to a ton of sample content created by some of the best sound designers in the business. It has multi-part and layer capabilities, robust effects, extensive modulation possibilities, and an enormous searchable collection of presets to explore. This might be the desert island choice for virtual instruments in terms of sound design in a musical context. It should be noted that the license for Omnisphere allows for use in a musical context, but non-music uses, such as embedding sounds in a video game would require special licensing. Covered musical uses under the license include, “…the ability to distribute commercial and non-commercial music releases, song placements, music libraries, beats, stems, trackouts, cues, melody/chord packs, etc.” Read their FAQ’s if you need clarification.

Omnisphere 2

u-he

This company’s strange hyphenated name is based on the name of the founding developer, Urs Heckmann. They develop amazing sounding virtual synths of all sorts.

Zebra 2 is their self-described workhorse instrument. It uses a configurable subtractive synthesis model with a modular-based GUI design. I can’t exactly describe how they do it, but the oscillator and filter sections are capable of a silky richness that is like nothing I’ve heard from analog or digital gear.

u-he Zebra

Tonstrum

Traveler this is a device that can spatialize and move sound sources around a virtual listening position in precise and incredible ways. Check out the demo video to get a taste.

Tonstrum Traveler

Cableguys

These guys have taken envelopes to the next level.

Shaperbox 2 can modulate pitch, amplitude, filter settings, distortion settings, time-based effects, and spatial effects with customizable, multi-segment envelopes.

Cableguys Shaperbox 2

Glitch Machines

Palindrome is appropriately billed as a “granular morph plotting sampler geared toward experimental sound design.” Loads of fun and unexpected results to be had with this device.

Glitch Machines Palindrome

SoundMagic Spectral

This is an amazing suite of freeware plugs designed by composer and developer Michael Norris that uses FFT processing and granular synthesis. The interfaces use default sliders and number boxes, but do not be fooled by the austere design. These devices are incredibly powerful!

The Basics of Sound Design (+ Essential Software)

iOS Apps

While I did not originally intend for this article to include iOS software, there are a few apps that need to be mentioned in the context of sound design.

Borderlands is a beast of an app that uses granular synthesis to process user imported samples. The interface is stunning and intuitive. This is the desert island iOS app for sound designers!

Borderlands

Hexaglyphics Noise Generator was one of my first apps and remains one of my favorites. The interface is like some alien language only understood by futuristic beings that communicate in some code-based alphabet. A great sound design and performance instrument!

Hexaglyphics Noise Generator

BitWiz this app is another code-based aesthetic, but it uses actual code to generate sound. Don’t know any code? No worries — this thing will generate random code with a swipe. Save what you like and dump the rest.

BitWiz

Samplr is a sophisticated sample playback device in which the user can define and manipulate multiple playback regions that can even overlap with each other. Several interface modes are accessible depending on your needs.

Samplr

AudioStretch is a really sweet tactile device that allows immediate and intuitive control over sample playback speeds, freezing, reversing and looping. When you experience the simple touch control possible with this device you will quickly realize there is nothing comparable in any traditional plugin.

Audiostretch

Most of these iOS devices are ideal and designed for live performance, but it seems to me their full potential has yet to be fully explored in terms of sound design in the studio. I’m not really sure why that is. There may be a misconception relating to price-quality relationships since these things are ridiculously cheap compared to typical DAW-based virtual instruments. It could be people are turned off by the limitations in terms of bit resolution (many apps deal exclusively with 16-bit files). But I’ve experienced great results when recording the output of these apps into a DAW at higher bit depths and sample rates. The sounds I typically use or create in the iOS world are somewhat raw and grungy to begin with, and things tend to get smoothed out with further post-processing in the DAW anyway, so for me lower bit depths at the initial stage are not really an issue.

The new M1 chips being used in Apple computers apparently use the same technology found in iOS devices, which in theory means these apps could someday be accessible directly on your laptop or desktop. It may have already happened. I have seen quite a bit of discussion around the idea and I am not completely aware of the ramifications for developers. I would love to hear any thoughts from active iOS developers on the issue. Please email me or comment where you see this article posted.


Check out my other articles, reviews, interviews and my video tutorial series, Synthesis 101 available exclusively on The Pro Audio Files.

Follow me on Twitter: @PMantione
Instagram: philipmantione

Philip Mantione

Philip Mantione is a composer, synthesist, guitarist, educator and sound artist active in the LA experimental music scene. His music has been presented in festivals, museums and galleries worldwide. His current project is TriAngular Bent, an electroacoustic trio featuring Don Preston (founding member of Frank Zappa’s Mothers of Invention) and circuit bending virtuoso, Jeff Boynton. Details at philipmantione.com