Intellectuals who see themselves as the progressive elite claim that Britain is the jail capital of Europe with more prisoners per head of population than any other European Union member, something they attribute to judicial savagery and popular vindictiveness.

Others complain about judicial leniency. The Government has been getting increasingly annoyed with the British public for having an "exaggerated fear" of crime. Others point to increasing violence and disorder.

How much crime do we suffer, compared with other countries and with our own history? In 1950 there were over 461,000 crimes recorded by the police. In 2004/05 there were over 10 times more, at 5.6 million.

Police records of crime throughout Europe reveal that England and Wales had the fourth highest crime rate out of the 39 countries in the 2003 European Sourcebook of Crime, the latest figures available from the Council of Europe.

After standardisation, our figure of 9,817 crimes per 100,000 population, was more than double the average of 4,333. However, the Government wants us to use the British Crime Survey (BCS), which found about 11 million crimes in 2004/05, down from its 1995 peak of over 19 million.

But the trouble with the BCS is that it only covers about half the crime recorded by the police. It misses out murder, rape, drug crime, fraud, all crimes against under-16s, and all commercial crime, including the biggest of all, shoplifting.

To sum up, we can say that crime is down from a peak in the mid-1990s and has now reached a plateau of about 10 times the rate in the 1950s, but violent crime is increasing steadily. We are a high-crime society with a complacent government.

Many Labour MPs remain in thrall to utopian theories of human nature and believe that criminals are driven to commit offences by social exclusion. They are not responsible - it's society that should change.

But this attitude does not go down too well on council estates, where the majority think that crime is all about knowing right from wrong. Jack Straw, Labour's first home secretary, published a White Paper entitled "No More Excuses" to ram home the message that Labour had changed.

But, as the recent battle over the Education Bill showed, many in the Labour Party never really accepted the new realism of Tony Blair. Despite the "tough" rhetoric, the influences have remained a dislike of prison, a naive view about how hardened offenders can be rehabilitated, and a lack of respect for the mass of people, who stand accused of being unduly fearful of crime.

Prison policy has also been ambiguous. Labour has increased the prison population by about 16,000 since 1997, but simultaneously it is letting hardened offenders out early under home detention curfew and urging judges to send fewer criminals to jail. Moreover, huge sums have been poured into "tough" alternatives to prison that are supposed to rehabilitate offenders. None have worked.

WHERE DID IT ALL GO WRONG?

The failure of 'tough' community sentences

Lord Phillips, the Lord Chief Justice, in a classic triumph of faith over fact said last month: "I would contend that a community sentence is more likely to prevent re-offending than a prison sentence." Where is the evidence?

The Intensive Supervision and Surveillance Programme (ISSP) is supposed to be the toughest of the tough alternatives to prison. At a cost of about ?45 million, it initially targeted 2,500 of the most serious young offenders.

It lasts for six months and, in addition to tagging, for the first three months offenders must take part in constructive activities for 25 hours a week, followed by five hours a week. An independent evaluation by Oxford University found that it was less effective than normal community sentences.

Not only did the latter cost much less, the reoffending rate was 84 per cent after 12 months, whereas under ISSP the figure actually went up to 88 per cent. Over half of offenders did not even complete ISSP, with the result that criminals, selected precisely because they were the most serious in their age group, were subject to only the most peremptory supervision.

An even more rigorous scheme was implemented for 18 to 20-year-olds, who were forced into training and education, and tagged each night. The study found that 52 per cent did not even complete the regime. Again, persistent and serious offenders, with an average of 27 previous convictions, who should have been locked up were left free to commit crimes. Anyone can read these studies on the Home Office website, an opportunity apparently not so far seized by the Lord Chief Justice.

Drug Treatment and Testing Orders in the community that require drug takers to submit to regular tests have also failed. About 70 per cent of offenders did not even complete their order and 80 per cent were reconvicted of a crime within two years.

Offending behaviour programmes based on cognitive-behavioural therapy have failed. Offenders are taught to "face up to their crimes and change their ways". In 2004, over ?27 million was spent on schemes, but a recent Home Office study found that the reconviction rate was 75 per cent, identical to that of the control group of similar offenders who did not take the courses.

The causes of crime

Most people do not commit crimes, because they have been brought up with a conscience that tells them it's wrong to steal or hurt other people. No one is a born criminal, but there is now a lot of evidence that their genetic endowment makes some children more likely to become criminals.

Most parents know that you can be faced with a difficult child, however good your parenting skills, and this is why having two parents matters.

A lone parent faced with an undemanding child may be able to manage, but when the child needs constant attention to prevent him from going off the rails, a single mother or father will struggle. Often such children end up in care. The Home Office has found that 27 per cent of prisoners had been in care and 47 per cent had run away from home.

But having two parents is no guarantee against crime if one of the parents is a criminal or a drug user. One Home Office study found that 43 per cent of prisoners had a family member with a conviction, and 35 per cent had other family members who had been in jail. Schools can reinforce the efforts of parents, but if the school is disorderly and fails to control bullying, crime is encouraged.

The criminal justice system also plays a part in moral education. Picture parents struggling to raise their children to be honest citizens on a run-down council estate. They urge them to respect other people and reinforce their appeals to youthful better nature with warnings about what happens if the police catch them committing crimes. But, if other local kids are obviously getting away with it, then the good parents may seem like "muppets" who don't "get it".

HOW DO WE SOLVE THE PROBLEM?

If these are the causes, what are the solutions? We need a better strategy for two groups: we need to control the existing criminals, and we need to discourage youngsters in danger of becoming the criminals of the future.

The police

We should increase police numbers and switch police effort to New York-style "broken windows" policing. In England and Wales in 2000, we had 237 police officers for every 100,000 population; the French had 396, or 67 per cent more.

But they had a much lower crime rate, 6,405 crimes for every 100,000 people, 35 per cent lower than ours. In 2005 we had 143,000 police officers, up from 127,000 in 1997, but there is still a long way to go.

Prison

One deliberate misunderstanding should be squashed. Punishment and rehabilitation are not opposites; they can go together. Excepting criminals whose offences are so heinous that they can never safely be released, the aim of policy should be to welcome offenders back into the community of law-abiding citizens.

But we need to combine our wish to rei
ntegrate criminals into the law-abiding mainstream with a bit of realism. Despite decades of trying, we have not yet discovered very effective ways of rehabilitating offenders, and we need to protect the public while the search continues for better methods.

We need to be clear about what prison is for. Its main value is that it protects the public: while offenders are in jail they can't break into your house. But it's not a matter of throwing away the key, and time inside should be spent preparing to lead a law-abiding life on release.

• First, we should increase prison capacity. The Government thinks there are 100,000 offenders who commit half of all crime, and that only 20,000 are in jail. We need a crash programme to lock up the other 80,000. The Home Office's own projections suggest that over 91,000 prison places could be needed by 2011, but provision is not being made. Instead, the Government is letting prisoners out early under Home Detention Curfew and it has a policy of not allowing the prison population to exceed 80,000.
• Second, we should improve prison regimes by getting prisoners off alcohol and drugs. All should be subject to mandatory drug testing on admission to jail and treatment for all who need it should start immediately. At present there are only random tests - meanwhile drug-taking remains common. A Home Office survey of prisoners in 2001/02 found that 39 per cent of prisoners interviewed had taken drugs in their current prison, about one third cannabis and 21 per cent heroin. Controlling addiction is not easy, but prison-based therapeutic communities, combined with intensive aftercare, have proved to be effective.
• Third, we should provide educational skills for all prisoners. US studies have found that inmates who had acquired vocational qualifications reoffended 33 per cent less than others. There is already a large educational programme but many prisoners leave without benefiting. The extra cost could be met by ending wasteful expenditure on failed offending behaviour programmes.

Impose the full sentence and abolish parole

We should abolish parole and introduce what has been called "truth in sentencing". At present, release at the half-way stage is automatic for nearly all prisoners, and can be earlier. The normal rule should be that the whole sentence is served unless offenders earn up to 20 per cent off for good behaviour.

Discouraging recruitment of the next generation of criminals

We need to increase investment in early socialisation and combating disorder in schools. Most people do not commit crimes because they have been brought up to share the community's standards. No crime policy will alter beliefs and attitudes if institutions for encouraging social cohesion, especially the family and schools, are in a weakened state. Home Office studies have found a link between family breakdown and crime, but the Government has done nothing to strengthen the family based on marriage.

For many young offenders their primary socialisation has failed and steps should be taken to initiate them into the community's sense of right and wrong. If their family is a bad influence, they need to be removed to a law-abiding environment. A graduated scheme is necessary, beginning with a "wel-fare" approach. However, if offences continue to be committed, the level of intervention by the authorities should escalate.

The more recalcitrant the offender, the more determined the response should be. Once offenders have been con-victed three times for an indictable offence, there is such overwhelming evidence that they are likely to spend the next several years committing offences that they should be sent to secure institutions for a significant period, with a minimum of 12 months and a maximum of three years, followed by intensive supervision.

Why can't young people be assisted solely in the community? Because, frequently, their home conditions are the cause of their offending. Sometimes their parents are drug addicts or criminals. In other cases, an older brother or a dysfunctional neighbourhood are a bad influence. Once a youth has reached 14 or so, only a sustained effort away from crime-friendly home influences can be expected to help.

TIME FOR A NEW APPROACH

The consensus that emerged in the 1960s and 70s has been breaking down for some time. We now have an intellectual elite out of step with public opinion. Mr Blair seems to agree, and wants to "rebalance" the system in favour of victims, by which he means reducing many of our traditional safeguards for the innocent.

But there are problems we can fix before we go that far. Increasing the number of police, building prisons, scrapping failed rehabilitation schemes, and humanising prison regimes to focus on education don't require "rebalancing". They do need a resolute government.

Perhaps we need a kind of truth and reconciliation commission to let the self-defined progressives who resist change off the hook. Now is the right time for a Royal Commission, made up of independent-minded people who will stand back, look afresh at the evidence and recommend how best to make our criminal justice system effective without abandoning safeguards for the innocent.

(Dr David Green is the director of the think tank Civitas, The Institute for the Study of Civil Society.)