Actually, I don't think Paul played any drums on the Together Alone title track. All session players!
This is always going to be a gray area that can be endlessly debated. I accept that Neil and Nick started recording Time On Earth together with sessions players and at some point decided that what they were creating sounded more to them like a Crowded House record than a Neil Finn solo album.
But Paul H's comment that it is intention that matters and not who is on the record means that the very first Crowded House album may not be a Crowded House album. Similar to TOE, when Neil set out to record Crowded House, he was not aiming to make a band album. That's why it was deemed acceptable to use a different drummer and bass player on "Now We're Getting Somewhere". Only after the album was recorded did Neil and company decide that it sounded like a band album and should be released under a band name. Then they just took the name of the album and made it the band name.
I think at this point it's clear that the only real Crowded House album is Temple of Low Men. Neil, Nick and Paul road tested the songs, played on all of them in the studio and then played them on tour. The rest of the albums are a mix of solo and Finn Brothers projects or different bands using the Crowded House name.