A long journey from chip to score
|
|
Listen to the audio clip. It's the music from Paul Norman's game "Aztec Challenge", released by US software house Cosmi in 1983.
The game is conveying the stress of being an Aztec running a gauntlet of spears to get to a temple. The Audio you hear is the the original soundtrack to the game, as programmed by Paul on the Commodore 64 using its "SID" soundchip, compared to an orchestral arrangement designed to be played by a real orchestra. We hope you will agree that the overall aim of the music is unchanged: conveying pulse-pounding action. But the scale of the pieces couldn't be more different. How do you get from one to the other, and hopefully not lose the feel the composer intended? |
Chipmusic
Chipmusic is most widely meant to mean music that's generated in real-time on a sound device built into a normal game console or home microcomputer. The music is usually not entered into the computer for playback as sheet music, it's entered as a series of numbers each of which is based on a look-up table: if you want to play a note, you look up the value for it. Equally, sounds are described as a series of numbers setting various parameters of the sound chip. This process is simplified somewhat these days with "tracker" interfaces that provide a nice UI, and easy management of patterns and sounds.
Music from different composers and different chips sounded different. This was because the chips varied in the type of sounds they could synthesise (some specialised in blippy sounds, some in scratchy), and because different composers used different hand-written code to perform routine things like varying the pitch of a note over time (vibrato/pitch bend) which would drastically alter the character of the sound. Composers generally had their own unique recognisable sound. Anyone who has listened to 80s music and its overuse of a DX7 electric piano sound knows how important it is to have a unique sound.
But how does the music get into the computer?
Music and sounds are entered as data in a computer program, which then runs very fast to throw data at the sound chip for a real-time performance: like a very fast, very flexible musical box.
Quite often the code is required to be as small and fast as possible to fit into the small amount of memory left when the graphics or game programmers had finished. Being small came naturally when you were only allowed to do three or four things at once (though programmers became adept at working around these limitations).
This lack of memory was a driving force behind chipmusic because it forced clever programming, and it made the use of multiple repeated phrases inevitable: the more times you could repeat a part, the more memory you saved.
So, chipmusic was often forced to be small and repetitive, though this did not stop the best composers from creating pieces up to 25 minutes long (as the C64 version of Tetris was. The level of musical and programming ability to create engaging and melodic pieces under these circumstances was advanced. Rob Hubbard was unusual among composers in that his ideas started as ordinary notation, which he then used as a jumping off point for coding. Most composers, not being familiar with musical script, would go straight from a keyboard riff to the coding, or even experiment with the numbers until something came out.
Chipmusic also tended to lack dynamics. It was an unusual sound-chip that allowed you to change its voices independently of one another, and the code to try and generate notes of varying loudness would have been regarded as a waste of memory.
This means that chipmusic is often loud, with no contrast between quiet and loud sections except where the waveforms were changed, or where instruments dropped out.
Compositionally, the structure varied widely between piece and composer, from a linear section progression (such as Rob Hubbard's "Knucklebusters", a 17 minute long progressive electronica opus) to a traditional pop single verse/chorus structure (like Rob's "Crazy Comets").
The best chipmusic has as much compositional effort and skill put into it as any other kind of music. The difference was the limited hardware on which it was played, and the musical limitations imposed by that. In addition, lack of polyphony meant a lack of definition musically which the composers worked around masterfully. Chords have to be left implied, for instance, an example being Aztec Challenge: which has only a bassline and a melody. but manages to convey its musical message unambiguously.
Music from different composers and different chips sounded different. This was because the chips varied in the type of sounds they could synthesise (some specialised in blippy sounds, some in scratchy), and because different composers used different hand-written code to perform routine things like varying the pitch of a note over time (vibrato/pitch bend) which would drastically alter the character of the sound. Composers generally had their own unique recognisable sound. Anyone who has listened to 80s music and its overuse of a DX7 electric piano sound knows how important it is to have a unique sound.
But how does the music get into the computer?
Music and sounds are entered as data in a computer program, which then runs very fast to throw data at the sound chip for a real-time performance: like a very fast, very flexible musical box.
Quite often the code is required to be as small and fast as possible to fit into the small amount of memory left when the graphics or game programmers had finished. Being small came naturally when you were only allowed to do three or four things at once (though programmers became adept at working around these limitations).
This lack of memory was a driving force behind chipmusic because it forced clever programming, and it made the use of multiple repeated phrases inevitable: the more times you could repeat a part, the more memory you saved.
So, chipmusic was often forced to be small and repetitive, though this did not stop the best composers from creating pieces up to 25 minutes long (as the C64 version of Tetris was. The level of musical and programming ability to create engaging and melodic pieces under these circumstances was advanced. Rob Hubbard was unusual among composers in that his ideas started as ordinary notation, which he then used as a jumping off point for coding. Most composers, not being familiar with musical script, would go straight from a keyboard riff to the coding, or even experiment with the numbers until something came out.
Chipmusic also tended to lack dynamics. It was an unusual sound-chip that allowed you to change its voices independently of one another, and the code to try and generate notes of varying loudness would have been regarded as a waste of memory.
This means that chipmusic is often loud, with no contrast between quiet and loud sections except where the waveforms were changed, or where instruments dropped out.
Compositionally, the structure varied widely between piece and composer, from a linear section progression (such as Rob Hubbard's "Knucklebusters", a 17 minute long progressive electronica opus) to a traditional pop single verse/chorus structure (like Rob's "Crazy Comets").
The best chipmusic has as much compositional effort and skill put into it as any other kind of music. The difference was the limited hardware on which it was played, and the musical limitations imposed by that. In addition, lack of polyphony meant a lack of definition musically which the composers worked around masterfully. Chords have to be left implied, for instance, an example being Aztec Challenge: which has only a bassline and a melody. but manages to convey its musical message unambiguously.
Getting the music out again
If you're going to create a symphonic version of a chiptune, the first thing is to get the notes right.
The problem that any arranger has is that the notes are not accessible in a ready-to-consume form, such as a MIDI file. The notes are compiled into the code of a game or demo. For most games/demos, their music is extracted from the game and put into a library such as the C64's HVSC ("High Voltage SID Collection").
So, it would appear that the choices are to dissassemble the code to find the note data, or listen to the piece and recreate it by ear: often very difficult because chipmusic is very often fast by design, and often contains flurries of notes that sound like chords since the notes are "flipped" very fast to convey the impression of multiple notes being played. In addition, some composers use different parts of the same sound to play different instruments: such as a hi-hat being played at the start of a bass note - again, to give the impression of more notes than the chip can support.
However, utilities such as "SID2MIDI" emerged that read the output registers of a chip as it's playing something and capture that information into a standard MIDI file that can be used by other sequencers: rather like transcribing a newspaper article from watching the lips of someone reading it.
The technically minded can also get text dumps from various players of what the chip is doing.
SID2MIDI also valiantly attempts to clean up the results, such as trying to keep an eye on where the notes are in relation to the barlines. Ideally, they should be dead on it, but they hardly ever are: clean-up is often a tiresome and lengthy process.
Once you've got your data about the original tune from one of these sources and entered into a straight transcription of the notes into a package such as Sibelius or Finale, then the arrangement process begins. And it has a lot more steps than just assigning the various phrases in the chiptune to instruments in the orchestra.
The problem that any arranger has is that the notes are not accessible in a ready-to-consume form, such as a MIDI file. The notes are compiled into the code of a game or demo. For most games/demos, their music is extracted from the game and put into a library such as the C64's HVSC ("High Voltage SID Collection").
So, it would appear that the choices are to dissassemble the code to find the note data, or listen to the piece and recreate it by ear: often very difficult because chipmusic is very often fast by design, and often contains flurries of notes that sound like chords since the notes are "flipped" very fast to convey the impression of multiple notes being played. In addition, some composers use different parts of the same sound to play different instruments: such as a hi-hat being played at the start of a bass note - again, to give the impression of more notes than the chip can support.
However, utilities such as "SID2MIDI" emerged that read the output registers of a chip as it's playing something and capture that information into a standard MIDI file that can be used by other sequencers: rather like transcribing a newspaper article from watching the lips of someone reading it.
The technically minded can also get text dumps from various players of what the chip is doing.
SID2MIDI also valiantly attempts to clean up the results, such as trying to keep an eye on where the notes are in relation to the barlines. Ideally, they should be dead on it, but they hardly ever are: clean-up is often a tiresome and lengthy process.
Once you've got your data about the original tune from one of these sources and entered into a straight transcription of the notes into a package such as Sibelius or Finale, then the arrangement process begins. And it has a lot more steps than just assigning the various phrases in the chiptune to instruments in the orchestra.
Starting the arrangement process
Before starting the arrangement, it's vital to work out what you're arranging for. Everything in the 8-bit symphony project was arranged uncompromisingly for a full symphony orchestra, with no cheating (although we did allow ourselves the odd extra instrument from time to time).
Next, a series of never-ending questions. Who are you arranging for? Why did they love that music? Are you hearing something different they're likely to like? What's the soul of the music? What did the composer mean?
A big question is: "is this piece even suitable for orchestra?".
Video game music is heavy on: repetition, complex repeated phrases, fast music (often too fast for humans to contemplate playing) and (sometimes) reliance on a synthesized sound that doesn't have a natural analog in the orchestra (sometimes a piece is a showcase for a technique). None of this is a disqualifier for becoming an orchestral piece: some of Rob Hubbard's music had all of these features which he had to "fix" for the orchestral version. "Flash Gordon" for example had a repeated (and very distinctive) bassline phrase which jumped octaves all the time: and which would have driven the basses and cellos mad. So they were replaced with more straightforward orchestration while adding new flourishes and texture changes into everything else to make the best of the orchestra so that bass change won't be noticed.
It's rarely a straightforward business of copying/pasting phrases to instruments, because thought has to be paid to the wellbeing of who's playing it!
In addition, some chipmusic isn't compositionally sophisticated, relying a fair amount on well-worn electronic music clichés learned from influences such as Kraftwerk, Jean-Michel Jarre or even other video game music composers. Since the end goal is for real orchestras to play it, appreciate it and enjoy it, it makes sense to avoid porting pieces out of a sense of duty, or just out of nostalgia. There's often a halo effect around people's favourite tunes ("that would sound GREAT with an orchestra!") which protects them from any analysis of its musical qualities that would be immediately spotted by someone to whom the piece was new.
The next bit of the arrangement process is the "-> SUCCESS!" part: the bit where you build up the structure, dynamics and orchestration to convey what story the arrangement is trying to tell. Sometimes that's a different story to the original, and sometimes it's a bigger, more cinematic sounding version of the same story. This stage will result at the end in a version that carries the emotions you want it to. It will also often carry playback information unnecessary to humans but vital to the machine playback you need to preview the arrangement. It's an irony that the closer a score gets to "perfect" for a human, the further away it deviates from "machine playable": since you're replacing machine interpretation you can write down with human interpretation you can't (except in general terms by specifying moods or playback techniques).
Here are some examples of chip -> score from this project showing how the original chiptune compares with the orchestration.
Next, a series of never-ending questions. Who are you arranging for? Why did they love that music? Are you hearing something different they're likely to like? What's the soul of the music? What did the composer mean?
A big question is: "is this piece even suitable for orchestra?".
Video game music is heavy on: repetition, complex repeated phrases, fast music (often too fast for humans to contemplate playing) and (sometimes) reliance on a synthesized sound that doesn't have a natural analog in the orchestra (sometimes a piece is a showcase for a technique). None of this is a disqualifier for becoming an orchestral piece: some of Rob Hubbard's music had all of these features which he had to "fix" for the orchestral version. "Flash Gordon" for example had a repeated (and very distinctive) bassline phrase which jumped octaves all the time: and which would have driven the basses and cellos mad. So they were replaced with more straightforward orchestration while adding new flourishes and texture changes into everything else to make the best of the orchestra so that bass change won't be noticed.
It's rarely a straightforward business of copying/pasting phrases to instruments, because thought has to be paid to the wellbeing of who's playing it!
In addition, some chipmusic isn't compositionally sophisticated, relying a fair amount on well-worn electronic music clichés learned from influences such as Kraftwerk, Jean-Michel Jarre or even other video game music composers. Since the end goal is for real orchestras to play it, appreciate it and enjoy it, it makes sense to avoid porting pieces out of a sense of duty, or just out of nostalgia. There's often a halo effect around people's favourite tunes ("that would sound GREAT with an orchestra!") which protects them from any analysis of its musical qualities that would be immediately spotted by someone to whom the piece was new.
The next bit of the arrangement process is the "-> SUCCESS!" part: the bit where you build up the structure, dynamics and orchestration to convey what story the arrangement is trying to tell. Sometimes that's a different story to the original, and sometimes it's a bigger, more cinematic sounding version of the same story. This stage will result at the end in a version that carries the emotions you want it to. It will also often carry playback information unnecessary to humans but vital to the machine playback you need to preview the arrangement. It's an irony that the closer a score gets to "perfect" for a human, the further away it deviates from "machine playable": since you're replacing machine interpretation you can write down with human interpretation you can't (except in general terms by specifying moods or playback techniques).
Here are some examples of chip -> score from this project showing how the original chiptune compares with the orchestration.
However, that's not the end of the tale. The next part is the "orchestration" part. Although some orchestration went on previously, the goal of this is to make the most of the orchestra at hand by changing it so that it's the best version it can be. Occasionally an arranger is good enough at knowing what a live orchestra can do, or how it sounds, and how live players play that this step is taken care of in the arrangement phase. In the case of 8-Bit Symphony, most of the pieces went through Alisdair J. Pickering, who re-scored parts to make them clearer, easier to play, better sounding, or just tidier. This means that the conductor of the concert gets a reasonably clean base to work from.
Next up, the conductor needs to adapt this master score to fit the exact orchestra he's working with: for instance, are some instruments unavailable? Are some sections going to be expanded? How is the emotional flow or directions to the orchestra going to work? This needs a conductor's score that fits onto e.g. B4 or A3 paper, which contains some of the instrument tracks combined into one stave to save space (for instance, having both oboes on one line).
Then, each player in the orchestra needs their part printing out. This part also needs to be carefully looked at. The process of looking at the scores and making cosmetic changes to enhance readability and playability is "engraving". Things like pagination, and making sure that you put "cues" into a player's part. For instance, an instrument might spend 68 bars doing nothing, and then play something. It's much easier for the person to come in at the right time if there's a clue about what plays before they come in: otherwise they spend their entire time counting bars: and it's easy to miss one!
One that's all printed out, then you're basically there :)
Easy peasy!
Except....
Next up, the conductor needs to adapt this master score to fit the exact orchestra he's working with: for instance, are some instruments unavailable? Are some sections going to be expanded? How is the emotional flow or directions to the orchestra going to work? This needs a conductor's score that fits onto e.g. B4 or A3 paper, which contains some of the instrument tracks combined into one stave to save space (for instance, having both oboes on one line).
Then, each player in the orchestra needs their part printing out. This part also needs to be carefully looked at. The process of looking at the scores and making cosmetic changes to enhance readability and playability is "engraving". Things like pagination, and making sure that you put "cues" into a player's part. For instance, an instrument might spend 68 bars doing nothing, and then play something. It's much easier for the person to come in at the right time if there's a clue about what plays before they come in: otherwise they spend their entire time counting bars: and it's easy to miss one!
One that's all printed out, then you're basically there :)
Easy peasy!
Except....
Making a mock-up
If all we had to do in the 8-Bit Symphony Project was to create live scores for orchestra, that process would be finished at this point.
However, the scores were (partially) funded off the Kickstarter proceeds for a box-set of these orchestrations on CD. Obviously we couldn't afford to get a real orchestra to record the amount of music we planned (an almost impossibly ambitious three albums worth: enough for two concerts and then some). So people bought the CDs with the orchestral mock-ups on.
So, how to make these as indistinguishable from a real orchestra as possible? That too is a completely different lengthy process!
Once the score was finalised, it was up to Chris Abbott to take all of the score files generated so far and (still in Sibelius) make them play back through much more expensive sample libraries than the one we'd been using to test the arrangements (Wallander's "NotePerformer").
Now, NotePerformer is "smart" playback: it reads ahead and guesses how you want something played, so it's trying to add back the humanity and interpretation.
However, we want better playback. And that means getting sample libaries with more playback styles ("articulations", such as "staccato" or "legato"), better recordings, more "round-robins" (i.e. when you play the same note, it plays a different recording), and so on.
There are lots of orchestral libraries on the market, but we fell in love with the ones from Orchestral Tools, who also sponsored the project by giving us libraries to work with. Everything from brass sections to percussion. They really made a difference.
One big job was connecting up these libraries to Sibelius. The libraries are all controlled with MIDI and "keyswitches" (where you press a key out of the natural range of the instrument to change the playing technique), and Sibelius had to know all the rules so that a staccato mark on the page leads to the relevant change at the library.
No integration existed, so we had to do one (it took a month).
After that, it's a painstaking process of marking up each note to add much-needed variation and passion. Luckily, a degree of humanisation is provided by Sibelius itself ("Espressivo").
Once that's done and the playback sounds like an orchestra, the individual instruments are saved out to be mixed in Cubase, where minor timing issues can be fixed. Noted musician Kenneth "Slaygon" Mutka has been part of the process since the beginning, and this is where he shines in producing the final track for CD.
And that really is it!
However, the scores were (partially) funded off the Kickstarter proceeds for a box-set of these orchestrations on CD. Obviously we couldn't afford to get a real orchestra to record the amount of music we planned (an almost impossibly ambitious three albums worth: enough for two concerts and then some). So people bought the CDs with the orchestral mock-ups on.
So, how to make these as indistinguishable from a real orchestra as possible? That too is a completely different lengthy process!
Once the score was finalised, it was up to Chris Abbott to take all of the score files generated so far and (still in Sibelius) make them play back through much more expensive sample libraries than the one we'd been using to test the arrangements (Wallander's "NotePerformer").
Now, NotePerformer is "smart" playback: it reads ahead and guesses how you want something played, so it's trying to add back the humanity and interpretation.
However, we want better playback. And that means getting sample libaries with more playback styles ("articulations", such as "staccato" or "legato"), better recordings, more "round-robins" (i.e. when you play the same note, it plays a different recording), and so on.
There are lots of orchestral libraries on the market, but we fell in love with the ones from Orchestral Tools, who also sponsored the project by giving us libraries to work with. Everything from brass sections to percussion. They really made a difference.
One big job was connecting up these libraries to Sibelius. The libraries are all controlled with MIDI and "keyswitches" (where you press a key out of the natural range of the instrument to change the playing technique), and Sibelius had to know all the rules so that a staccato mark on the page leads to the relevant change at the library.
No integration existed, so we had to do one (it took a month).
After that, it's a painstaking process of marking up each note to add much-needed variation and passion. Luckily, a degree of humanisation is provided by Sibelius itself ("Espressivo").
Once that's done and the playback sounds like an orchestra, the individual instruments are saved out to be mixed in Cubase, where minor timing issues can be fixed. Noted musician Kenneth "Slaygon" Mutka has been part of the process since the beginning, and this is where he shines in producing the final track for CD.
And that really is it!