The man, who cannot be named for legal reasons, faced the first day of a jury trial at the Victorian Supreme Court on Monday.
He has pleaded not guilty to 12 charges, including seven of causing another person to enter into and remain in servitude and three of causing injury intentionally.
Prosecutors allege the man forced his wife and children to perform farm and house work over five years, in fear of being beaten by him if they did not follow his instructions.
"This case is about a family living under tyranny," prosecutor Patrick Doyle SC told the jury of 14.
"The accused dominated their lives through physical violence, through threats and through intimidation.
"He not only abused his physical power over them, but he was psychologically and emotionally abusive as well."
Mr Doyle said the abuse began years earlier, in NSW before they moved to a small Victorian country town in 2015.
He alleged the man, who married his wife in 1998, was abusing her long before their children were born and the violence continued in front of them.
One child saw her being beaten in the yard with a garden hose, another saw him throw a cooking pot at her when he complained a meal was not ready and on the table, Mr Doyle alleged.
Their eldest son recalled seeing him punch his mother in the face and stomach until she was "knocked out" when he was six years old.
Mr Doyle said the earlier pattern of abuse had conditioned the man's family into fearing him.
By the time the family moved to a farm and had to participate in the first harvest "they knew what the consequences would be of defying him", he told the jury.
The man allegedly used intimidation, threats and physical violence, including driving at them and hitting them with objects like golf clubs, to force them into working.
Work included digging irrigation channels, harvesting and packing grapes, pruning and clearing vines, tending to animals and building fences.
It also included house work and cleaning the man's truck.
"They did all these jobs in fear of being beaten if they didn't," Mr Doyle said.
He also alleged the man stopped his wife and children from forming relationships outside their immediate family, which further isolated them.
"His children weren't allowed to have friends at school. If the accused learnt they had made friends, or shared personal information about themselves, they'd be physically punished for doing so," he said.
One of his daughters went to police in August 2021, which Mr Doyle said ended the alleged abuse and servitude.
The trial continues on Tuesday, when the man's lawyers will address the jury.
1800 RESPECT (1800 737 732)
Lifeline 13 11 14
15°C