The 78 year-old was head of the Richardson Gang, a group of criminals who were fierce rivals to the Kray twins in the 1950s and 1960s.
The group, which also included brother Eddie, were also known as the "Torture Gang", who became notorious for using horrific tactics as punishment during "mock trials".
His south London empire was said to have embraced fraud, gambling and protection rackets, claims which he later denied, and also had interests in mining in South Africa.
Victims were reportedly subjected to electric shock, whipping, cigarette burning and pinning victims to the floor with 6in nails and removing their toes with bolt cutters.
They were tortured at the brother's scrap metal yard in Camberwell before being told to clean up their own blood.
If victims survived their "trial", they were then handed a clean shirt, which reportedly led to the torture sessions becoming known as "taking a shirt from Charlie".
Richardson, a father of six whose "business" interests including gaming machines, died yesterday in hospital after collapsing with peritonitis.
His sister Elaine, wife Ronnie and his five children were by his bedside at Pembury Hospital in Kent, it was reported. Friends and family confirmed his death after battling poor health in recent times.
While he was a heavy smoker and had suffered from emphysema, it was claimed he died from blood poisoning following a gall bladder complaint.
Eddie Richardson, who also denies much of the claims against his family, said: "I haven't spoken to him in years so I am not the best person to speak to about his death.
"I can't say he was a good father, but he was a father. He leaves a big family behind him."
Charlie Richardson was arrested on July 30, 1966, the day of the World Cup Final and charged with assault, robbery and demanding money with menaces.
Richardson denied using torture despite being jailed for 25 years the following year after an Old Bailey trial.
The court heard lurid evidence from men who fell out with the brothers and were summoned to their scrap metal yard.
A key prosecution witness was travel agent Lucian Harris, who said he was tortured for information after his partner disappeared owing the family money.
Richardson later said a key witness lied and the judge was biased because he knew one of his fellow defendants and the jury foreman. Eddie Richardson was also sent to jail along with Fraser.
Their downfall came after a series of violent clashes with the Kray gang, who were based in east London, during a bitter turf war.
Richardson attempted several times to be released from jail and in 1980 escaped from an open prison before going on the run for almost a year in France.
He later returned to jail and upon his release in 1984, became a campaigner for young offenders. He consistently claimed the torture stories were a myth.
Richardson's life story was made into a movie in 2004 starring actor Luke Goss.