UNHCR said in a statement on Friday that 16 survivors were pulled from the water after the vessel sank "in one of the worst incidents on the Mediterranean in months".
"This is a tragic and terrible reminder of the risks still faced by those who attempt to cross the Mediterranean," said Vincent Cochetel, UNHCR's special envoy for the Mediterranean.
Only four bodies have been recovered so far and search operations were ongoing, the International Organization for Migration (IOM) said in a Twitter post.
According to reports by Tunisia's state-run TAP news agency, the boat carrying as many as 70 refugees and migrants sank 40 nautical miles off the coast of Sfax, some 270km south of the capital, Tunis.
Fishing boats rescued the survivors, TAP reported.
IOM, which described the incident as "another tragedy in the Mediterranean", said the vessel took to the sea from neighbouring Libya, where renewed warfare between rival factions has gripped the capital, Tripoli, in recent weeks.
An IOM official told The Associated Press news agency those on board included Bangladeshi and Moroccan nationals, among others.
'Deadliest sea crossing'
Libya's western coast is the main departure point for refugees and migrants hoping to reach Europe by sea.
The number of people attempting the journey has dropped due to an Italian-led effort to disrupt smuggling networks and support the Libyan coastguard with funds, ships and training, in return for carrying out interceptions and rescues.
UNHCR's Cochetel said action needed to be taken "across the region" in order to "strengthen the capacity of search and rescue operations".
"If we don’t act now, we’re almost certain to see more tragic events in the coming weeks and months," he added.
The Mediterranean Sea is the "world's deadliest sea crossing", according to UNHCR, with one in 14 people dying on the route from Libya to Europe last year.
A UN report published in January said six refugees and migrants died every day in 2018 while attempting to cross the Mediterranean.