Mr Grayling warned he no longer wanted to see prisoners "sitting in cells watching the Sunday afternoon match of Sky Sports" or viewing jail as a place to "enjoy".
Pledging action on the "completely unacceptable" use of mobile phones in prison, he said: "I want to be a tough Justice Secretary."
Mr Grayling, who replaced Kenneth Clarke at the Ministry of Justice in the cabinet reshuffle earlier this month, said he also had no intention of cutting prisoner numbers.
He said he did not want inmates to enjoy prison, telling a newspaper the criminal justice system needed to be one in which the public could have confidence.
He said: ''I'm bringing a fresh pair of eyes to the job. I'm very mindful of the need to have a criminal justice system in which people have confidence. I think they very often don't have confidence in it.''
Mr Grayling told the Daily Mail: ''Prison is not meant to be a place that people enjoy being in. I don't (want to) see prisoners in this country sitting in cells watching the Sunday afternoon match on Sky Sports.
''Am I planning to reduce the number of prison places? No I'm not. I do not want to set a target to reduce the prison population.
''What I do want to do is bring down the cost of prison. The whole philosophy I will bring to the department is getting more for less.''
Mr Clarke had been a thorn in the side of the Tory right, who saw him as a soft justice secretary, who oversaw the scrapping of indeterminate sentences for the most serious offenders, while at the same time apparently embarking on policies aimed at reducing prison numbers.
The decision by Prime Minister David Cameron to appoint Mr Grayling to the brief was seen as a nod to the Conservative right, who have long seen a tough justice system as central plank of any Tory agenda.
But his tough stance on prison numbers, Mr Grayling said he was supportive of Mr Clarke's policies to rehabilitate inmates.
Private firms could be paid according to results, he said, in a nod to a policy introduced by Mr Clarke.
And he also promised to bring an end to the stalking of victims by inmates using social media after they have successfully smuggled mobile phones in to prison.
''It's completely unacceptable,'' he said. ''I've talked to victims of crime who are effectively being stalked by the person who attacked them.
'It is the case that mobile phones are smuggled into prisons. We have powers to take tough action on that front, blocking signals, other ways of clamping down on it.''