AI overview
In recent years, advancements in artificial intelligence (AI) have introduced new tools to a sound mixer's repertoire, automating certain processes and optimising workflow.
However, these technologies augment rather than replace the sound mixer’s technical and artistic abilities. Their deep understanding of sound physics, audio equipment and their contributions to creative decisions remain crucial, ensuring that technology serves to enhance the storytelling, without diminishing the human touch that is central to their craft.
Download a PDF version of this page.
What impact might AI have in the future?
Audio capture techniques are changing. Instead of simply single microphones, with ‘baked in’ spatial information, complex microphone arrays, higher resolution capture and other approaches are starting to be used. This gives the ability to make changes to the audio capture in post.
This technique is similar to the way that 'Raw’ video capture is currently used to capture greater visual detail than can easily be displayed on a monitor (it requires grading).
Similar approaches are arriving with audio, i.e. the captured audio cannot easily be played back over traditional stereo monitor speakers or headphones. This is primarily in a spatial sense. Instead of simply capturing a piece of audio with a single microphone, arrays of microphones will be used as normal, with the ability to selectively adjust ‘polar pattern’ or ‘capturing zones’ after the recording has taken place.
Context aware AI will be able to do this automatically, allowing a sound mixer to specify or choose which elements of a recording should be included or removed, based on their spatial position at the location. In effect, this gives the option to undo the decision that is typically baked-in with a single microphone capture. This is similar to the processing happening in a human brain with our two ears and outer ear shape.
Microphone arrays are also likely to have a visual capture device, allowing the AI to perfect lip sync and understand the context of a scene, informing this kind of decision making process.
These kind of approaches will allow the onsite sound department greater freedom in microphone positioning, and the ability to work in noisier and more acoustically challenging environments, such as large reverberation, without resorting to ADR.
It will also facilitate some automatic editing and labelling of audio recordings on a per take basis, improving and speeding up storage, archiving and retrieval processes.
What AI tools can I use right now?
There are many digital whiteboarding, brainstorming, charting and script annotation tools that are essential for the planning process of sound mixers.
Combining these with scheduling and planning, while also using a cloud-based solution can create visibility across the team and reduce the risk of errors and costly mistakes.
Many of these tools include AI features that considerably improve workflows and efficiencies. Text to system diagram, or automatic technical flow charts are just a few of the examples.
Kit list preparation, asset tracking and cost prediction are also starting to become available, in some case being able to cost and recommend solutions directly from a script.
The ability to control digital mixing consoles and prepare the workflows and automation in advance is steadily increasing, allowing a sound mixer to need less setup and soundcheck time on location.
Some modern field recorders utilise AI to automatically adjust gain levels, ensuring optimal recording quality. Additionally, tools like AR visualisation, virtual call sheets and coordination platforms, and remote collaboration and monitoring solutions can be employed to enhance workflows and efficiency.
In various broadcast and unscripted contexts AI auto-mixing solutions are starting to appear to deal with various elements of a mix, in effect giving sound mixers very sophisticated ‘context aware’ compression, expansion and limiting algorithms.
These can be used for real time level balancing, noise reduction or many other novel uses. Latency is the current challenge in many of these solutions, but that is rapidly improving with hardware acceleration.
New digital networking solutions are also changing this landscape and enabling PC-based mixing solutions to be incorporated. AI and machine learning is also changing the shape of the plugin market across the board.
Auto configuration: AI is enabling some consoles and digital workstations the ability to auto configure. Watch out for ‘given brief’ to direct configuration tools with auto setup solutions.
Advanced wireless audio systems: Innovations in wireless microphone systems technology, such as improved digital transmission and better interference reduction, enhancing on-set audio recording.
Cloud-based workflows and storage: Utilising cloud services for backing up recordings and sharing files increasing efficiency and security.
Virtual reality (VR) and augmented reality (AR) audio capture: As VR and AR content grows, understanding how to capture and mix audio for immersive experiences will be essential.
Autonomous robotics: Autonomous robots and physical manipulation robots can allow on set microphone movement and subject tracking to take place in locations that previously were not accessible.
Acoustic environment prediction and auralisation: Create 3D acoustic environments based on room scans prior to filming to determine room weak spots and best microphone locations.
Automatic metadata tagging and organisation at recording stage: AI-assisted tagging allows for faster searching and organisation of sound libraries by automatically categorising an transcription of audio files based on their characteristics.
Spatial capture technologies: Arrays of microphones able to capture multiple distinct sources at a location.
ScreenSkills offers a variety of training opportunities for people at all stages of their career. Explore all training, events and opportunities.
ScreenSkills resources:
- Sound mixer job profile
- Sound mixer skills checklist
Other resources:
- Shure Audio Institute, RF Venue and Sennheiser Sound Academy have resources on advanced wireless audio systemss
- Coursera, AWS Training and Certification, Google Cloud and Microsoft Azure have information and tutorials on cloud-base workflows and storage
- Udemy, Coursera, Audio Engineering Conferences (AES) and Ambisonics have resources on virtual reality and augmented reality audio capture
- Robotics Academy by Carnegie Mellon University, ROS (Robot Operating System) and Bow Robotics offer tutorials and/or free training sessions