The babies and toddlers of mothers at Pollsmoor Prison in Cape Town will no longer be raised behind bars - as is the norm across the country.
The facility can house 15 inmates and it has a creche and a clinic.
But Correctional Services Minister Nosiviwe Mapisa-Nqakula warned that it was not a "guesthouse" and that there would be security.
Children under the age of two are allowed to stay with their mothers who are serving a sentence in South Africa's prisons.
Older children are either cared for by a family member or put into foster care.
'Children should not be punished'
Ms Mapisa-Nqakula said the facility at Pollsmoor Prison had been specifically designed to give children a good start in life.
She said she wanted all children living there to not only enjoy quality time with their mothers, but to be able to play with sand, touch grass and see the sky.
"The mothers have wronged society and given birth to their children here, [but] the children should not take the responsibility or be punished," South Africa's Cape Times newspaper quotes Ms Mapisa-Nqakula as saying.
"So this is in their best interest - not to make prisons more comfortable," she said.
Similar units will be opened at prisons across the country in the coming weeks.
According to the Department of Correctional Services, there were 143 children below the age of two living in South African jails last year.