If victims survived their "trial", they were then handed a clean shirt, which reportedly led to the torture sessions becoming known as "taking a shirt from Charlie".
Richardson, a father of six whose "business" interests including gaming machines, died yesterday in hospital after collapsing with peritonitis.
His sister Elaine, wife Ronnie and his five children were by his bedside at Pembury Hospital in Kent, it was reported. Friends and family confirmed his death after battling poor health in recent times.
While he was a heavy smoker and had suffered from emphysema, it was claimed he died from blood poisoning following a gall bladder complaint.
Eddie Richardson, who also denies much of the claims against his family, said: "I haven't spoken to him in years so I am not the best person to speak to about his death.
"I can't say he was a good father, but he was a father. He leaves a big family behind him."
Charlie Richardson was arrested on July 30, 1966, the day of the World Cup Final and charged with assault, robbery and demanding money with menaces.
Richardson denied using torture despite being jailed for 25 years the following year after an Old Bailey trial.
The court heard lurid evidence from men who fell out with the brothers and were summoned to their scrap metal yard.
A key prosecution witness was travel agent Lucian Harris, who said he was tortured for information after his partner disappeared owing the family money.
Richardson later said a key witness lied and the judge was biased because he knew one of his fellow defendants and the jury foreman. Eddie Richardson was also sent to jail along with Fraser.
Their downfall came after a series of violent clashes with the Kray gang, who were based in east London, during a bitter turf war.
Richardson attempted several times to be released from jail and in 1980 escaped from an open prison before going on the run for almost a year in France.
He later returned to jail and upon his release in 1984, became a campaigner for young offenders. He consistently claimed the torture stories were a myth.
Richardson's life story was made into a movie in 2004 starring actor Luke Goss.