Jack Straw is reviewing the new open-ended jail terms amid fears that they are clogging up the prison system.
The tough "public protection" sentences - known as IPPs - were introduced two years ago for violent and sexual offenders and thousands have been handed down.
Although minimum "tariffs" are set, the decision to release an inmate is taken by the Parole Board.
It means many prisoners are staying in jail for longer than used to be the case - worsening the overcrowding already caused by a record number of inmates.
The Justice Secretary, on a visit to Belmarsh prison, east London, said he recognised the concerns about IPPs.
"What I am going to do immediately is to look very carefully at the operation of IPPs," he said. "I am aware of concern from the judiciary. And I am obviously aware of issues from prison staff, officials and now from prisoners as well."
The sentence prevents dangerous and violent criminals being released until the authorities are satisfied they are no longer a threat to the public.
A number of legal challenges are pending, brought by inmates who say this is in breach of their human rights.
Overcrowding prevents many IPP prisoners gaining access to the programmes they need to complete to persuade the Parole Board they are no longer a danger.
Officials have admitted that ministers seriously underestimated the huge numbers -around 4,000 offenders so far - who would be affected.
No new funds were made available because it was wrongly assumed that resources would be released by a reduction in some other types of prisoners. Fixing the problem will now cost well over ?10 million.
Several Belmarsh inmates told Mr Straw they had been kept inside longer than expected under the tariff imposed by their trial judge.
They complained they had been unable to complete rehabilitation courses because of the overcrowding. The prison population is at a record level of more than 81,000.
Mr Straw said he wanted courts to make greater use of non-custodial community sentences, repeating a plea issued by successive ministers in the past.
Although the Government has promised to provide another 9,500 prison places by 2012, Mr Straw said the answer was not to build more prisons but to keep people out of them.
They have become so crowded that governors have been ordered to release thousands of inmates 18 days before they would normally be freed.
Around 500 extra places are to be provided at existing jails this summer to ease the pressures further, using temporary or rapid-build units.
Overcrowding was also blamed by penal reformers yesterday for a big rise in prison suicides.
Some 50 prisoners have taken their own lives so far this year, compared with 67 in the whole of last year.