Indeterminate Sentences for Public Protection, known as IPPs, were a Labour policy that allowed criminals to be locked up indefinitely until they were deemed to pose a threat no longer.
Since they were introduced more than 6,500 offenders have been jailed without a fixed date for release, and according to the Strasbourg ruling some of them have little chance of getting out.
Judges said "IPP prisoners swamped the system in place for dealing with those serving indeterminate sentences", and that "existing resources were insufficient" for them to undergo behaviour courses before they could be considered for parole.
They agreed with the three Britons who brought the case - Brett James, Nicholas Wells and Jeffrey Lee - that there were "delays" in accessing the prison courses, caused by "a lack of resources".
This was "the consequence of the introduction of draconian measures for indeterminate detention without the necessary planning and without realistic consideration of the impact of the measures".
"Further, the length of the delays in the applicants' cases was considerable: for around two and a half years, they were simply left in local prisons where there were few, if any, offending behaviour programmes."
The judges said it was a breach of Article 5:1 of the European Convention on Human Rights - which bans arbitrary detention - for them to have been kept locked up beyond the minimum term specified when they were jailed, without having access to rehabilitation courses.
The three men, who were jailed in 2005 for violent offences, were awarded almost £14,000 in damages and close to £30,000 in costs.
If all 3,500 remaining IPP prisoners claimed compensation on the same grounds, it could cost the Government as much as £16m.
Ministers may also have to make more rehabilitation courses available to those still serving IPPs, or make release tests less reliant on completion of the programmes.
The Ministry of Justice had already announced it was scrapping IPPs last year, but it is still appealing against the ECHR ruling.
The Director of Campaigns for the Howard League for Penal Reform, Andrew Neilson, said: "By appealing, the Justice Secretary is delaying the inevitable. He knows as well as anyone else that is ridiculous and unfair to say to people, many of whom have completed their sentence, that they will only be released if they complete certain rehabilitation courses - and then not provide those courses.
"This makes a mockery of the basic principles of the English justice system, which has always locked people up for what they have done rather than what someone thinks they might do. He should accept this common sense judgment and if he wants to avoid compensation claims focus now on working out a way to safely release those who have already served their time but are still in prison."
