In 1830, barons Ungern-Sternberg established a broadcloth factory in Kärdla. Today's factory square formed the core of the factory complex. The four-storey broadcloth factory was located on the south side of the courtyard of the factory on the shores of the Nuutri river. On the western side of the courtyard, there was the residence of the factory director - known as the Long House, today home to the Hiiumaa Museum is located.

On the northern side of the square, 4 single-storey wooden houses or shop master's houses for senior officials of the factory were erected and have preserved well until the present time. The houses had large gardens.

Until the end of 1970s, a house with pillars or the Summer Manor was located on the eastern side of the square. In the early years of the broadcloth factory, it was the residence of the factory director.