The European Court of Human Rights held unanimously that the controversial sentences, which Ken Clarke as Justice Secretary announced he was scrapping last year, breached prisoners' rights to liberty and security.
More than 6,500 offenders have been sent to jail without any fixed date for their release using indeterminate sentences for the protection of the public (IPPs) since they were brought in by Labour in 2005.
Such a large number of people were sentenced to IPPs that "it soon became clear that existing resources were insufficient" and "IPP prisoners swamped the system in place for dealing with those serving indeterminate sentences", the human rights judges ruled.
The case concerned three Britons - Brett James, Nicholas Wells and Jeffrey Lee - who were given automatic IPP sentences in 2005 and claimed that a failure to ensure they had access to courses in prison harmed their ability to show they were rehabilitated and able to be released.
They and others will now be in line for compensation.
The judges said: "It is clear that the delays were the result of a lack of resources."
While resource implications were relevant, the inadequate resources "appeared to be the consequence of the introduction of draconian measures for indeterminate detention without the necessary planning and without realistic consideration of the impact of the measures", they added.
"Further, the length of the delays in the applicants' cases was considerable: for around two and a half years, they were simply left in local prisons where there were few, if any, offending behaviour programmes.
"The stark consequence of the failure to make available the necessary resources was that the applicants had no realistic chance of making objective progress towards a real reduction or elimination of the risk they posed by the time their tariff periods expired.
"Further, once the applicants' tariffs had expired, their detention was justified solely on the grounds of the risk they posed to the public and the need for access to rehabilitative treatment at that stage became all the more pressing.
"In these circumstances, the court considers that following the expiry of the applicants' tariff periods and until steps were taken to progress them through the prison system with a view to providing them with access to appropriate rehabilitative courses, their detention was arbitrary and therefore unlawful within the meaning of Article 5:1 of the convention."
James was jailed for two years for unlawful wounding with intent and had a series of convictions for violent offences. Wells and Lee were jailed for serious violent offences.
The Parole Board ruled he could not be released because he had not completed assessments to make a judgement of his character. However he was unable to get places on anger management and substance abuse programmes.
All three cases went to the Law Lords, which ruled against the men.
