Become ruined.

When hippies took to smoking cannabis in the 1960/70s, one of the more common of the myriad terms for the drug was 'pot'. The derivation of that term is disputed. It was stated in American Speech, as early as 1936, that the word derives from the Spanish word for marijuana - potiguaya. Others have questioned that, saying that there is no such word in Spanish. That aside, journalists of the day couldn't resist the 'gone to pot' label. For example, this piece by John Brewer, in the California newspaper The Independent, June 1970:

"I can get marijuana faster than I can get a text at the bookstore," says a UCLA graduate student. College students, it seems, have gone to pot.

Of course, people had been going to pot long before then. That was in the sense of 'going to (the) pot', i.e. being chopped up and boiled for food. That usage dates back at least as far as the 16th century. For example, in William Tindale's An answere vnto Sir Thomas Mores dialoge, 1530:

Then goeth a part of ye little flocke to pot, and the rest scatter.

The colloquial, metaphorical usage of 'ruin; destruction' is fairly old too and was in common use by the 17th century. Edmund Hickeringill used the term in The History of Whiggism, 1682:

Poor Thorp, Lord Chief Justice, went to Pot, in plain English, he was Hang'd.

That meaning alludes to the fact that the journey of an animal or ingredient to the pot was a one-way trip, with a very short future ahead.