Joint article by:
Molly Turner, orchestral conducting master’s student at Juilliard, New York City (40.7° N, 74.0° W)
Douglas Graham, computer science sophomore at Rice University, Houston (29.7° N, 95.3° W)
The caricature for a composer may go something like this: they’re sitting at a cluttered desk, staring at a blank piece of music in front of them. At the moment of inspiration, they furiously jot down notes as if dictating the sounds of the heavens. If only it were that easy! And perhaps our overdone stereotype of the computer scientist is a nerdy and antisocial individual who is smashing keys at a coffee shop, coding the next big thing. What computer scientist would go to a music party? And what composer would delight in C++ or JavaScript? We wouldn’t consider much overlap between either of these specialties. In fact, though, there are a multitude of areas where computer science intersects with our composition, production, and consumption of music—too many, even, for this article to cover in full.
To start, a great deal of music composition actually involves a logic-based thinking that also underlies computer science. Sound waves passing through the air and into our ears are like information passing between systems. Logic-based thinking and music composition overlap in the work of Arnold Schoenberg (1874-1951), a composer who chose to organize sounds in a very systematic way with his development of the twelve-tone method. The twelve-tone method assigns each of the twelve chromatic notes in the octave a number instead of the traditional letter name. Then those numbers are put into a matrix with various conditions developed by Schoenberg. This matrix is the reference for which pitch is assigned in music. Essentially, traditional western harmony was abandoned for numerical outputs from a matrix. Milton Babbitt, who took inspiration from Schoenberg, was both a mathematician and composer and used these ostensibly diverse specialties to create music that furthered the efforts of Schoenberg and serialization of music. Babbitt and others would go on to serialize not only pitch, but also dynamics, articulation, rhythm, and other musical parameters. Babbitt and Schoenberg’s music haven’t drawn wide audiences, and some of their music is considered unlistenable. Their efforts, however, showed us the outer limits of logic-based thinking as a way to outsource one’s compositional agency.
Beyond the logic-based thinking that can be involved in composition, another intersection between music and computer science happens at the level of production. If we look broadly at the history of music, we start with the human voice and basic acoustic instruments. During the Renaissance, advances in physical models helped develop more powerful instruments that were louder and more versatile. Most recently, with the invention of recording and synthesizing technology, the spectrum of sound possibilities are now endless. Digital audio workstations (called DAWs), made possible by computer scientists, now provide anyone who wants to create music the platform to do so. Mozart used highly esoteric notation to represent his musical compositions. He also needed live and paid musicians to bring his music to life. But today, anyone with a laptop can seek out their creative potential, thus creating a much-needed mechanism for diverse music creators to produce their music.
While logic-based thinking and technology certainly aid in the composition and production of music by humans, the emergence of music created by artificial intelligence (AI) brings the connection to a new level. This was brought to the public’s attention with a recent Google Doodle, which allowed users to compose a small melody and have it harmonized in the style of J. S. Bach. The algorithm works using machine learning, a type of AI where a computer is fed a large amount of example data (in this case, over 300 of Bach’s famous chorales) and uses those examples to generate its own example or prediction. The Google Doodle, along with similar attempts to write music through computer programs, has polarized the music community. Some, such as those working on the Magenta research project, a research collaborative where programmers publish AI models to augment musicians’ toolbox, saw the Doodle as a creative opportunity to extend musicians’ existing skills. Others, like Bach scholar Christopher Brody, expressed disappointment at the Doodle for failing to capture any important elements of Bach’s style and instead attempting to emulate Bach by finding compromises in his music. Similar disagreements are visible in the music community surrounding new AI algorithms that emulate certain composers or produce original music. Regardless of any argument, one fact remains concrete: no one has been able to generate music entirely automatically. That is, every piece of music “created” by AI so far has been based directly off of pieces written by a real, human composer.
Although much of the conversation about technology in music centers around whether AI should play a role in composition, AI already has a prominent function in our consumption of music. It is heavily present in video-sharing platforms like YouTube and streaming services like Spotify, learning what listeners enjoy best and recommending them new music based on their preferences. This can be beneficial to listeners, who are given accurate recommendations by the AI algorithms, and it can benefit artists by promoting their work to listeners who may not have discovered them otherwise. On the other hand, as much as streaming companies tout their ability to unearth hidden music, their profit is ultimately determined by their quantity of listens. Their AI algorithms sometimes reflect this by recommending music they know viewers will listen to rather than suggesting the music they would enjoy most. Though this drawback may limit AI’s potential to give listeners exciting new recommendations consistently, it doesn’t cancel out the improvement in music accessibility that AI creates.
Computer science majors and music majors may rarely interact on a college campus. In fact, on the Rice University campus, the Shepherd School of Music is about as far as you can get from most computer science classes. As we have explored, however, the interaction between these specialties has yielded many innovations that are transforming the way we create and listen to music. With these advances, accessibility to the composition, production, and consumption of music continues to increase but so do the questions about how much human agency is a necessary component to art. Exploring this intersection has taught us that the push and pull between creativity and logic is the motor for progress, perhaps in any field. Out-of-the-box musicians and coders will be the initiators for breakthroughs in the future of technology and music.
Further Reading:
Topics covered:
Milton Babbitt and System Based Composition (see page 7):
https://getd.libs.uga.edu/pdfs/sullivan_erin_l_200508_ma.pdf
Music Composition, Schoenberg, and Matrices:
https://sites.math.washington.edu/~morrow/336_15/papers/rasika.pdf
Deep Learning and Music Composition:
https://cs224d.stanford.edu/reports/allenh.pdf
Bach Google Doodle:
https://www.google.com/doodles/celebrating-johann-sebastian-bach
Debate on the Bach Doodle:
https://slate.com/technology/2019/03/google-doodle-bach-ai-music-generator.html
YouTube Algorithm:
https://www.youtube.com/watch?v=fHsa9DqmId8
Spotify Algorithm:
https://medium.com/s/story/spotifys-discover-weekly-how-machine-learning-finds-your-new-music-19a41ab76efe
Other topics not explicitly discussed:
MIDI (Musical Instrument Digital Interface) as Computer Instruments:
https://blog.landr.com/what-is-midi/
The Virtual Orchestra and MIDI sound banks that work within with DAWs:
http://motu.com/products/software/bmf-encore-soundbank
Virtual Reality and the Orchestra:
https://www.theguardian.com/music/2016/sep/29/virtual-reality-london-philharmonia-orchestra-esa-pekka-salonen-interview
A Brief History of Sampling, Music Production, and Technology:
https://www.musicradar.com/tuition/tech/a-brief-history-of-sampling-604868
Binaural Audio: Music, Physics, Stereo Sound, and “3D Sound” (has applications in gaming as well):
https://www.youtube.com/watch?v=Yd5i7TlpzCk
Technology and Music Education:
http://solfeg.io/music-education-technology/
Longitude.site welcomes applications from students who are interested to explore other topics related to music and technology. Apply here.