Ministers are facing calls to rethink cuts to legal aid because they will hit vulnerable groups, a parliamentary committee of peers and MPs says.