Survivors told the Red Crescent the tragedy unfolded after some 75 people who had left Zuwara on the northwestern Libyan coast late Thursday on a large boat were transferred to a smaller one that sank off Tunisia.
"The migrants were transferred into a smaller inflatable boat which was overloaded, and 10 minutes later it sank," Mongi Slim, a Red Crescent official in the southern Tunisian town of Zarzis, told AFP.
Tunisian fishermen rescued 16 people and brought them to shore in Zarzis.
The survivors said they spent eight hours trapped in the cold sea before they were spotted by the fishermen who alerted the Tunisian coastguard, Slim said.
The bodies of three people were plucked out of the waters on Friday, the Tunisian defense ministry said.
Survivors said the boat was heading to Italy and had on board only men, 51 from Bangladesh, as well as three Egyptians, several Moroccans, Chadians and other Africans.
Fourteen Bangladeshi nationals, including a minor, were among the survivors, said the Red Crescent.
"If the Tunisian fishermen hadn't seen them (migrants), there wouldn't have been any survivors and we would have never known about this" boat sinking, said Slim.
Charity ships have plied the Mediterranean Sea to rescue migrants in large numbers but the number of rescue operations have dwindled as these vessels have come under fire, namely from the populist Italian government, over their action.
Italy's far-right Interior Minister Matteo Salvini has imposed a "closed ports" policy, refusing to allow migrants rescued at sea to enter his country.
On Friday, however, more than 60 migrants disembarked in Italy after two boats which had left Libya faced difficulties at sea and needed assistance.
The UN agency for refugees UNHCR called for stepped up search and rescue operations to avoid future tragedies in the Mediterranean, which it calls the "world's deadliest sea crossing".
"Across the region we need to strengthen the capacity of search and rescue operations," said Vincent Cochetel, the agency's special envoy for the Mediterranean.
"If we don't act now, we're almost certain to see more tragic events in the coming weeks and months," he warned.
According to the UNHCR, the journey across the Mediterranean "is becoming increasingly fatal for those who risk it".
"In the first four months of this year, one person has died (crossing the Mediterranean) for every three that have reached European shores, after departing from Libya," it said.
Libya, which has been wracked by chaos since the 2011 uprising that killed veteran dictator Moamer Kadhafi, has long been a major transit route for migrants desperate to reach Europe.