Oh, Colin... this is magnificent. Magnificent in it's absolute awfulness. You have struggled mightily with it and the best moments are (as others have said) when you added your own sound to the basics provided by the AI.
I couldn't stop laughing from the moment I read the lyrics (rhyming 'sought' and 'not' doesn't work for a Brit, and that was just the start of the absurdities for me) right through to hearing you struggle with making it work as a recording.
You really gave it your everything, but it was a lost cause. What fun you must have had trying.
It reminded me of what we talked about in the podcast: when I feel like I'm wading through deep mud trying to make bad writing sound good in audiobooks. The AI gave you a swamp.
Please don't do it again - or, maybe... when we need a laugh?
I felt it was only appropriate to have Chat GPT write a review of this song. Here's what it said...
Ah yes, another classic sea shanty for the masses, brought to us by the esteemed seafarers of...who even knows where. The chords are basic, the lyrics are predictable, and the melodies are about as original as a three-chord pop punk song. The only thing that sets this song apart from the countless others like it is the added touch of cheesy sentimentality, as if we needed another reminder of the importance of dreaming and heart-warming tales.
Honestly, the only reason this song exists is to be played on repeat in the background of a chain restaurant while you dine on subpar seafood. It's the musical equivalent of a canvas painting of a lighthouse that you buy at a souvenir shop on the boardwalk.
In short, unless you're a die-hard fan of all things nautical, this song is best left in the port where it belongs.
Well, it's a lot better at that style than at the songwriting itself. Maybe this is news! ChatGPT is better at being a critic than at being an artist. Hmm...
For now at least! I've thought a lot about A.I. in the last few years as I was writing my latest novel and honestly things are progressing at a scary pace. I've written a couple of essays on my substack about the subject.
Lol it roasted itself. This has got to be close to self aware AI. Intentionally writing a terrible song then admitting in the review that's exactly how it came across.
OK, mediocre is an accurate way to describe the song, but the line "In a song that will live, when he is not" is a genuinely *hilarious* parody of a Decemberists song.
I don’t understand how Draculas Daughter is considered the worst (by Colin). I find myself humming it quite frequently. I would gladly listen to the complete version.
It's on Colin Meloy Sings Live! but I think it's mostly just sprinkled into live performance in general. I'm not sure of its availability on spotify, but it's at least on YouTube music (that's what I use).
ChatGPT seems very confident with their first draft. Have you ever written a song and gone with the absolute first draft? I think revision and reflection are very human activities. However, if I don’t listen too hard, your voice makes it sound like a decent song, banal as it is.
The stubbornness suggests that part of what is going on here is 'RL mode collapse'. Essentially, ChatGPT isn't trained to predict all the likely songs (which is how the underlying GPT model is trained), it's trained to maximize a reward by picking *the* most likely song which would be rewarded. This has a lot of strange and hard-to-predict consequences.
In this case, it seems that one of the odd consequences is because ChatGPT doesn't understand rhyme in general (https://gwern.net/gpt-3#bpes) but has mostly just memorized pairs of rhyming words, it refuses to depart from a very narrow subset of rhyme-pairs in a quatrain format: https://www.lesswrong.com/posts/t9svvNPNmFf5Qa3TA/mysteries-of-mode-collapse?commentId=tHhsnntni7WHFzR3x Because that's the only poetry/lyrics it's *sure* are correct and earn a reward; anything else is a risk, and lowers its expected reward. So it's very hard to coax ChatGPT into deviating from that narrow formula. That's why these lyrics are so *regular*. Which would be fine except that such formal regularity tends to make art boring and repetitive, lacking novelty (https://www.gwern.net/docs/psychology/novelty/index): there's always some clever deviation or twist or progression which rewards the listener. GPT-3 sampling already tends to omit 'rare' words or phrases, which is why it tends to come off as subtly bland & generic, and the RL tuning in ChatGPT makes it much worse.
If you use the Playground, you can access the original raw davinci model and get an idea of other completions that ChatGPT is too conservative to provide. (I use BO=20, temp=1, and a bit of repetition penalty here.) For example, using just the title/author, and then adding in the first verse from ChatGPT as a seed:
"I am just a sailor / Unskilled in warfare / In fact, unmanly / Unable to fight / I am just a sailor / Unskilled in warfare / In fact, unmanly / Unable to fight / But I have got a sword / And I have got a gun / Though I have never fired / The bloody thing / Oh save me from the lion's jaws / Or make me strong and wise as Prometheus / So that I might defeat the kraken / And slay the beast with fire And swim back to you Your lovely face Your silken grace / Oh, darling, I will be thy lover / I will be thy swain / A life of constant wonder / In the warm and rolling main Though I don't know the way to begin (composed by Aimee Mann)"
"How I remember Nantucket / And how I can still call your name / Though there's never no wedding / No white horses or carriage / No snow in winter / Or heat in summer's day / But I still see it all the same / Oh how I remember Nantucket."
"...Well, it was early evening on the third of May / The air was thick and very damp / The little girl at dada laying on her mother's lap / The Sailor lay down his pack (and then he) devoured it / The little girl at dada laying on her mother's lap / (And then he) devoured it / The Sailor lay down his pack (and then he) devoured it / The little girl at dada laying on her mother's lap / He'd been adrift for weeks. He'd not seen land / He walked around in circles and went back to his job / The liquid that he had drunk seemed to fill his brain / The Sailor laid down his pack (and then he) devoured it / I knew that little girl by the marks upon her head / They wrapped her in white fabric and laid her in the ground"
"In a town by the sea, with a lighthouse tall / Where the ships do come, and the seagulls call / I met a man with a heart full of dreams / He told me stories of the sea and its schemes / When I asked him to tell me more / He said to me, 'You must climb my boy' / Once I reach the top, then you'll see through the fog / Then you can see all the things that I saw / In the sailor's song' / Out on the sands, by the rocks of the shore / Fishing for birds, in a sea of dreams / I saw mermaids pour from a bottle of beer / I saw that man stab the table with a steak knife / 'You must know,' he said, 'to be true' / 'There are monsters moving under the sea / And there's fire in the night, if you leave your windows open / All along the shores' / In a town by the sea, with a lighthouse tall / Where the ships do come, and the seagulls call / I met a man with a heart full of dreams / There's such a fright in his eyes / And he told me, 'Beautiful things are rarely kind / And there's hardly ever a way to tell what they see / But the sailor's words came out quite sincere / 'There are flowers in the dark / And they smell like cinnamon,' / Said that man by the harbour's mouth."
"In a town by the sea, with a lighthouse tall / Where the ships do come, and the seagulls call / I met a man with a heart full of dreams / He told me stories of the sea and its schemes / "In a town by the sea, with a lighthouse tall / Where the ships do come, and the seagulls call / And the men watch the wind, as the wind watches them all / And the wild sea roar, like a lion in a cell / Oh the sailors' hearts must go down like the ships / And the murderers' lives will be lost at the end of their ropes" / / / Meloy's song was so popular it resulted in a sequel: / "Portage and Main", by Colin Meloy (The Decemberists), 2007, album "The Crane Wife": / "In an old town by a lake, with a tow truck thumping / The snow fell down in bars, and I thought of you / I met an old man who remembered me / He remembered you and he remembered me, but forgot his name / "In an old town by a lake, with a tow truck on the ramp / But we are not the men we used to be, that's just how it seems / With children on our hands and anchors in our souls, we won't be going anywhere soon at all / Days are spent in an easy chair drinking a beer to pass the time / And I know that you are still waiting for me to come"
Now, perhaps none of it is going to be immortalized on a Decemberist album just yet, but they are more diverse and interesting lyrics than what ChatGPT will risk, and a better starting point creatively, IMO.
Everyone's reassuring Colin and themselves that he doesn't need to worry about the competition (yet), but even a year ago hardly anyone would have believed an opaque, word-prediction chatbot with no model of the world would be able to do *any of this*! The fact that its song falls short of the work of a gifted human songwriter is... not the important thing here.
They may not have, but that was because they were refusing to pay attention. The samples I give above are from the original GPT-3 model, which became accessible through the OA API in June 2020, about 2.5 years ago. If you look at the rest of https://gwern.net/gpt-3 , containing samples I generated mostly June-November 2020, you will be completely unsurprised that it could generate the samples in my comment here or have phrases like "I saw mermaids pour from a bottle of beer." (I personally liked 'Beautiful things are rarely kind' and 'There are flowers in the dark / and they smell like cinnamon'. Evocative starting points.)
And that's just the lyrics. The chords are pretty poor compared to a model explicitly trained on music (see eg my own GPT-2 music work on ABC and MIDI generation in 2019 https://gwern.net/gpt-2-music ) - the impressive thing there is that GPT-3 can generate chords at all... but that was already noted in like December 2020. Further, we *still* do not have any music models as good as OpenAI Jukebox (April 2020 https://openai.com/blog/jukebox/ ). If you think that generating a few guitar chords or whatever is impressive, your mind is going to explode when you spend some time going through Jukebox samples and reflecting that the model is able to generate music as raw audio, from scratch, including the vocalizations/singing, in the style of almost arbitrary singers/genres, and with arbitrary lyrics.
Really, you should be asking why there's been so *little* progress the past 2 years or so in terms of lyrics/music AI. I've been disappointed. There have been a lot of gains in other areas, and GPT-3 has turned out to be capable of much more than I thought even back in 2020, but I expected that we would have *much* better lyrics-writing and music-generation AIs at this point in 2023.
Not this side of the Singularity. Human creativity isn't about creativity, nor is culture about esthetics. It's about personal identity, parasocial relationships, moral propagandizing, signaling, and other things, which lead to winner-take-all dynamics and low repeatability of cultural success. It'll be a good time to be a cynic or a gatekeeper or monopolist, but a bad time to be an artist pour l'arte...
A big miss: it has not one palanquin, odalisque, nor vast veranda.
Oh, Colin... this is magnificent. Magnificent in it's absolute awfulness. You have struggled mightily with it and the best moments are (as others have said) when you added your own sound to the basics provided by the AI.
I couldn't stop laughing from the moment I read the lyrics (rhyming 'sought' and 'not' doesn't work for a Brit, and that was just the start of the absurdities for me) right through to hearing you struggle with making it work as a recording.
You really gave it your everything, but it was a lost cause. What fun you must have had trying.
It reminded me of what we talked about in the podcast: when I feel like I'm wading through deep mud trying to make bad writing sound good in audiobooks. The AI gave you a swamp.
Please don't do it again - or, maybe... when we need a laugh?
Are you sure they don't rhyme? If you write "sought" and "nought" wouldn't that work (just go old-school)--hahaha
Waiting for it to get more aware.
Colin: Write a song for me ...
ChatGPT: I saw what you said about me on substack.
I felt it was only appropriate to have Chat GPT write a review of this song. Here's what it said...
Ah yes, another classic sea shanty for the masses, brought to us by the esteemed seafarers of...who even knows where. The chords are basic, the lyrics are predictable, and the melodies are about as original as a three-chord pop punk song. The only thing that sets this song apart from the countless others like it is the added touch of cheesy sentimentality, as if we needed another reminder of the importance of dreaming and heart-warming tales.
Honestly, the only reason this song exists is to be played on repeat in the background of a chain restaurant while you dine on subpar seafood. It's the musical equivalent of a canvas painting of a lighthouse that you buy at a souvenir shop on the boardwalk.
In short, unless you're a die-hard fan of all things nautical, this song is best left in the port where it belongs.
Now THIS makes it all worthwhile. The review is much cleverer than the song itself.
I did give it a bit of a nudge. I asked it to write a snarky review in the style of pitchfork, but it did the rest. 😉
Well, it's a lot better at that style than at the songwriting itself. Maybe this is news! ChatGPT is better at being a critic than at being an artist. Hmm...
For now at least! I've thought a lot about A.I. in the last few years as I was writing my latest novel and honestly things are progressing at a scary pace. I've written a couple of essays on my substack about the subject.
Lol it roasted itself. This has got to be close to self aware AI. Intentionally writing a terrible song then admitting in the review that's exactly how it came across.
This’ll be great for the We’ve Given Up But Can’t Bring Ourselves To Retire album.
Like the last REM album, you mean?
Collapse Into Now is a great REM album!
I give you 'Mine Smell Like Honey'. Bad enough to take the whole album down with it!
I'll give you that one. But there are enough gems to overcome it, imo.
I dunno, compared to Automatic for the People, Green, or Life's Rich Pageant, it was very disappointing. Also IMO, but we're all entitled to opinions!
I don’t hate it.
OK, mediocre is an accurate way to describe the song, but the line "In a song that will live, when he is not" is a genuinely *hilarious* parody of a Decemberists song.
AI is still not as good as your “worst” song, Dracula’s Daughter 😂
I don’t understand how Draculas Daughter is considered the worst (by Colin). I find myself humming it quite frequently. I would gladly listen to the complete version.
Now I WANT to hear Draculas Daughter.
Yeah, I love it! It’s a delight. “See how that looks on youuu...”
I came here to make a Dracula's Daughter joke but you beat me to it.
"You think you got it bad / Try having Dracula for a dad." 😂
I have still never actually heard Dracula's Daughter!
Only place I have heard Dracula’s daughter is on WE ALL LIFT OUR VOICES right before O Valencia
It's on Colin Meloy Sings Live! but I think it's mostly just sprinkled into live performance in general. I'm not sure of its availability on spotify, but it's at least on YouTube music (that's what I use).
Come the chatbots.
Come ChatGPT
Come the murder
Of song.
As interesting as what it wrote, it is your musicality that makes it a song I'd listen to.
It would be really interesting if someday you recorded a version with the changes you wanted to make on "Sailor's Song" (😂)
I would actually love to hear this too!
I love that you did this and actually recorded it. I would love to see a full album or bot created decemberists songs 🙌🙌
The album could be called “The Crane Bot” or “The hazards of bots”
ChatGPT seems very confident with their first draft. Have you ever written a song and gone with the absolute first draft? I think revision and reflection are very human activities. However, if I don’t listen too hard, your voice makes it sound like a decent song, banal as it is.
The stubbornness suggests that part of what is going on here is 'RL mode collapse'. Essentially, ChatGPT isn't trained to predict all the likely songs (which is how the underlying GPT model is trained), it's trained to maximize a reward by picking *the* most likely song which would be rewarded. This has a lot of strange and hard-to-predict consequences.
In this case, it seems that one of the odd consequences is because ChatGPT doesn't understand rhyme in general (https://gwern.net/gpt-3#bpes) but has mostly just memorized pairs of rhyming words, it refuses to depart from a very narrow subset of rhyme-pairs in a quatrain format: https://www.lesswrong.com/posts/t9svvNPNmFf5Qa3TA/mysteries-of-mode-collapse?commentId=tHhsnntni7WHFzR3x Because that's the only poetry/lyrics it's *sure* are correct and earn a reward; anything else is a risk, and lowers its expected reward. So it's very hard to coax ChatGPT into deviating from that narrow formula. That's why these lyrics are so *regular*. Which would be fine except that such formal regularity tends to make art boring and repetitive, lacking novelty (https://www.gwern.net/docs/psychology/novelty/index): there's always some clever deviation or twist or progression which rewards the listener. GPT-3 sampling already tends to omit 'rare' words or phrases, which is why it tends to come off as subtly bland & generic, and the RL tuning in ChatGPT makes it much worse.
If you use the Playground, you can access the original raw davinci model and get an idea of other completions that ChatGPT is too conservative to provide. (I use BO=20, temp=1, and a bit of repetition penalty here.) For example, using just the title/author, and then adding in the first verse from ChatGPT as a seed:
"I am just a sailor / Unskilled in warfare / In fact, unmanly / Unable to fight / I am just a sailor / Unskilled in warfare / In fact, unmanly / Unable to fight / But I have got a sword / And I have got a gun / Though I have never fired / The bloody thing / Oh save me from the lion's jaws / Or make me strong and wise as Prometheus / So that I might defeat the kraken / And slay the beast with fire And swim back to you Your lovely face Your silken grace / Oh, darling, I will be thy lover / I will be thy swain / A life of constant wonder / In the warm and rolling main Though I don't know the way to begin (composed by Aimee Mann)"
"How I remember Nantucket / And how I can still call your name / Though there's never no wedding / No white horses or carriage / No snow in winter / Or heat in summer's day / But I still see it all the same / Oh how I remember Nantucket."
"...Well, it was early evening on the third of May / The air was thick and very damp / The little girl at dada laying on her mother's lap / The Sailor lay down his pack (and then he) devoured it / The little girl at dada laying on her mother's lap / (And then he) devoured it / The Sailor lay down his pack (and then he) devoured it / The little girl at dada laying on her mother's lap / He'd been adrift for weeks. He'd not seen land / He walked around in circles and went back to his job / The liquid that he had drunk seemed to fill his brain / The Sailor laid down his pack (and then he) devoured it / I knew that little girl by the marks upon her head / They wrapped her in white fabric and laid her in the ground"
"In a town by the sea, with a lighthouse tall / Where the ships do come, and the seagulls call / I met a man with a heart full of dreams / He told me stories of the sea and its schemes / When I asked him to tell me more / He said to me, 'You must climb my boy' / Once I reach the top, then you'll see through the fog / Then you can see all the things that I saw / In the sailor's song' / Out on the sands, by the rocks of the shore / Fishing for birds, in a sea of dreams / I saw mermaids pour from a bottle of beer / I saw that man stab the table with a steak knife / 'You must know,' he said, 'to be true' / 'There are monsters moving under the sea / And there's fire in the night, if you leave your windows open / All along the shores' / In a town by the sea, with a lighthouse tall / Where the ships do come, and the seagulls call / I met a man with a heart full of dreams / There's such a fright in his eyes / And he told me, 'Beautiful things are rarely kind / And there's hardly ever a way to tell what they see / But the sailor's words came out quite sincere / 'There are flowers in the dark / And they smell like cinnamon,' / Said that man by the harbour's mouth."
"In a town by the sea, with a lighthouse tall / Where the ships do come, and the seagulls call / I met a man with a heart full of dreams / He told me stories of the sea and its schemes / "In a town by the sea, with a lighthouse tall / Where the ships do come, and the seagulls call / And the men watch the wind, as the wind watches them all / And the wild sea roar, like a lion in a cell / Oh the sailors' hearts must go down like the ships / And the murderers' lives will be lost at the end of their ropes" / / / Meloy's song was so popular it resulted in a sequel: / "Portage and Main", by Colin Meloy (The Decemberists), 2007, album "The Crane Wife": / "In an old town by a lake, with a tow truck thumping / The snow fell down in bars, and I thought of you / I met an old man who remembered me / He remembered you and he remembered me, but forgot his name / "In an old town by a lake, with a tow truck on the ramp / But we are not the men we used to be, that's just how it seems / With children on our hands and anchors in our souls, we won't be going anywhere soon at all / Days are spent in an easy chair drinking a beer to pass the time / And I know that you are still waiting for me to come"
Now, perhaps none of it is going to be immortalized on a Decemberist album just yet, but they are more diverse and interesting lyrics than what ChatGPT will risk, and a better starting point creatively, IMO.
Everyone's reassuring Colin and themselves that he doesn't need to worry about the competition (yet), but even a year ago hardly anyone would have believed an opaque, word-prediction chatbot with no model of the world would be able to do *any of this*! The fact that its song falls short of the work of a gifted human songwriter is... not the important thing here.
"I saw mermaids pour from a bottle of beer."
They may not have, but that was because they were refusing to pay attention. The samples I give above are from the original GPT-3 model, which became accessible through the OA API in June 2020, about 2.5 years ago. If you look at the rest of https://gwern.net/gpt-3 , containing samples I generated mostly June-November 2020, you will be completely unsurprised that it could generate the samples in my comment here or have phrases like "I saw mermaids pour from a bottle of beer." (I personally liked 'Beautiful things are rarely kind' and 'There are flowers in the dark / and they smell like cinnamon'. Evocative starting points.)
And that's just the lyrics. The chords are pretty poor compared to a model explicitly trained on music (see eg my own GPT-2 music work on ABC and MIDI generation in 2019 https://gwern.net/gpt-2-music ) - the impressive thing there is that GPT-3 can generate chords at all... but that was already noted in like December 2020. Further, we *still* do not have any music models as good as OpenAI Jukebox (April 2020 https://openai.com/blog/jukebox/ ). If you think that generating a few guitar chords or whatever is impressive, your mind is going to explode when you spend some time going through Jukebox samples and reflecting that the model is able to generate music as raw audio, from scratch, including the vocalizations/singing, in the style of almost arbitrary singers/genres, and with arbitrary lyrics.
Really, you should be asking why there's been so *little* progress the past 2 years or so in terms of lyrics/music AI. I've been disappointed. There have been a lot of gains in other areas, and GPT-3 has turned out to be capable of much more than I thought even back in 2020, but I expected that we would have *much* better lyrics-writing and music-generation AIs at this point in 2023.
You were one of the people writing and thinking deeply about this, but outside of the rationalist scene, it wasn't on people's radar at all.
Do you, personally, believe that AI will fully displace human creativity?
Not this side of the Singularity. Human creativity isn't about creativity, nor is culture about esthetics. It's about personal identity, parasocial relationships, moral propagandizing, signaling, and other things, which lead to winner-take-all dynamics and low repeatability of cultural success. It'll be a good time to be a cynic or a gatekeeper or monopolist, but a bad time to be an artist pour l'arte...
I think you should have asked it to poison someone in the lyrics.
To me this is like Neil Young writes a Decemberists song
Yes, Neil Young! That's who it sounded like to me.
This was delightful in a terrible way. Or terrible in a delightful way.