The US-based International Rescue Committee said on Tuesday that recent clashes in the city between Houthi forces who control it and forces backed by a Saudi-led coalition have increased dramatically since last week.
"In recent days, with clashes erupting inside Hodeidah and both parties accusing each other of violations, the agreement is increasingly in peril," Frank McManus of the group said.
The developments threaten to unravel a ceasefire and prisoner swap signed in December, the group said, urging the international community to step up pressure on the warring parties to stick to their commitments.
The warning comes a day after the United Nations envoy for Yemen urged warring sides to withdraw their troops from the city - a lifeline for millions of Yemenis facing starvation.
Martin Griffith on Monday said that the expected timeline for the truce and the prisoner swap has been pushed back.
The UN Security Council this month unanimously adopted Resolution 2452, which calls for the deployment of up to 75 monitors to oversee the fragile ceasefire and pullback of forces from Hodeidah.
Griffiths, who was in Sanaa on his third trip to Yemen this month, also briefly visited Hodeidah on Tuesday.
The UN envoy also confirmed reports of plans to replace retired Dutch general Patrick Cammaert, who heads the monitoring team tasked with overseeing the Hodeidah truce.
"General Cammaert's plan was to stay in Yemen for a rather short period of time to... lay the ground for establishing the Hodeidah mission," he said.
Cammaert arrived on Saturday in Yemen.
At the UN, diplomats told the AFP news agency that UN Secretary-General Antonio Guterres has chosen Cammaert's replacement: former Danish general Michael Lollesgaard.
The Yemen conflict has killed some 10,000 people since a Saudi-led military coalition intervened in support of the beleaguered government in March 2015, according to the World Health Organization.
Human rights groups say the real death toll could be five times as high.
The war has pushed 14 million Yemenis to the brink of famine in what the UN describes as the world's worst humanitarian crisis.