Chinese minister had accused Japan of a "gross violation" of China's territorial integrity.
The foreign ministers of China and Japan held tense talks over their countries rival claims to a bitterly disputed island chain on Tuesday but made no breakthrough, diplomats said.
"Although the atmosphere was a bit severe, the Chinese side laid out its own case and the ministers agreed to continue the dialogue," a Japanese official told AFP after the encounter on the sidelines at the UN General Assembly.
China s Foreign Minister Yang Jiechi and Japanese counterpart Koichiro Gemba were in New York at the annual UN summit and took the opportunity to discuss a crisis that has plunged Sino-Japanese relations to their lowest level in years.
Their meeting came on the same day as Japanese coast guards fought a high seas water cannon duel with ships from Taiwan, the third claimant to the archipelago, known as Senkaku in Japanese and Diaoyu in Chinese.
All three nations claim sovereignty over the uninhabited chain, but it has been administered by Japan for decades. The latest row blew up when the Japanese government bought the land from a private citizen.
Japan insists it took the move to prevent nationalists from staging stunts on the island but, if the purchase was intended to calm tempers, it failed.
Chinese mobs gathered to protest the alleged land grab and vandals targeted Japanese firms. Taiwan dispatched a fleet of fishing vessels escorted by coast guard cutters to assert their claim to the area.
Against this backdrop, the ministers met on neutral ground at the United Nations for what both sides described as a tense preliminary meeting.
Chinese state news agency Xinhua said its minister had accused Japan of a "gross violation" of China s territorial integrity.
A Japanese official said the ministers had agreed not to release many details of the talks "because of the diplomatic sensitivities" but had agreed to keep a communication channel open at an official level.
"First the Chinese foreign minister made his own case about the Senkaku Islands. He explained his position in detail," the senior official said.
"In response the Japanese foreign minister requested the maximum careful restraint from the Chinese side because of the recent damage to Japanese businesses and affiliates in China. He insisted violence cannot be tolerated."
Japan s Gemba had requested the meeting, diplomats said.
Xinhua quoted Yang as saying that Tokyo had bought the islands in the East China Sea "regardless of China s stern representations and strong opposition."
He added that the move was "an outright denial" of the defeat of Japan in World War II and "a grave challenge to the post-war international order."
The Japanese side did not publicly comment on the talks, but its envoys had earlier briefed journalists on Tokyo s position.
"We do not believe that there is a dispute to be resolved," said Naoko Saiki, deputy director general for press and public diplomacy at the Japanese foreign ministry, as officials briefed reporters in New York.
"We do not believe there are disputes to be resolved in terms of sovereignty or territory, because in light of historical fact and of international law the Senkaku Islands are an equal part of Japan s territory," she said.
Earlier Tuesday, coast guard vessels from Japan and Taiwan clashed with water cannon off the islands after dozens of Taiwanese boats escorted by patrol ships sailed into waters claimed by Tokyo around the island.
"We do not want to have any war or battles or use of force, so we have to stabilize the situation though dialogue in a peaceful manner in accordance with international law. That s the basic position maintained by Japan," Saiki said.
The Japanese delegation provided reporters with copies of documents that it said supported Tokyo s claim to the islands, including copies of Chinese maps from 1932 and 1960 that mark the islands as Japanese territory.