Neville Neville, 64, of Larkfield Close, Bury, denies sexually assaulting a woman by penetration on 23 March.
Bolton Crown Court heard he cried as he told his family he had shared a "passionate" kiss with the woman.
The woman alleges Mr Neville "forced" himself on her. Mr Neville claims the encounter was consensual.
He told the court that after being interviewed by police, he had driven his wife Jill to their house, where his son Phil and daughter Tracey, a former England netball international, were waiting.
'Pull yourself together'
He said his son had "erupted, basically".
"He said 'You have got the most loveliest wife ever' [and] 'She loved you to death'.
"I was crying at this stage. Tracey had obviously broken down."
He said his son had "turned round to me and said 'I'm not going to ask if you sexually assaulted her, I know you would not do that'".
"I said 'Philip, I did not sexually assault that woman.'
"He said 'Stop crying then [and] pull yourself together, you have got some explaining to do with mum'."
Both of Mr Neville's sons were in the public gallery to hear their father give his evidence.
In his interview with police, which was played to the jury, Mr Neville admitted he "should not be kissing other women" as he was married, but said he and his accuser had "kissed passionately" and there were "no problems".
'Not consensual'
Mr Neville also told the police he was diabetic, with bladder cancer and a bad heart, so "could not have had sex with her if [he had] wanted" and claimed the woman probably wanted media attention.
Previously, the woman told the court she "froze" after Mr Neville "pounced" on her.
She said she had not said "no" to him, but she had gone "stiff", which was like "my body was saying it was not consensual".
Mr Neville is the former commercial director and commercial manager of Bury Football Club.
He was involved in the fans' group Save Our Shakers, which was formed when the League Two side faced bankruptcy in 2002.
The case continues.