Desperate to save their heritage, the city's shrinking Jewish community decided to act. Defying the possibility of prison, deportation or execution, they buried the gravestones and kept them from Nazi hands.
Some 70 years later, Jewish leaders in the Austrian capital say the long-lost stones have been rediscovered. It is a find they say could transform a small obscure graveyard into one that rivals the significance of Prague's Jewish cemetery, the oldest known burial ground of its kind.
The cemetery has no name and is hard to find, with the only entrance through a city home for the elderly in Vienna's 9th district. Weathered but restored gravestones poke through weedy, uncut grass, with faded Hebrew inscriptions.
But beneath the ground, Jewish leaders say, are other tombstones piled layer upon layer.
The cemetery dates back to the 16th century and had about 900 tombstones until 1938, when the Nazis came to power and gave vandals free reign to deface and destroy them.
Vienna's Jewish leaders say it is not clear exactly how many were buried by the small group of Viennese Jews determined to save their heritage from the Nazi bulldozers.
After workers scored the ground with radar as part of restoration work, they say they are sure there are hundreds beneath the grass. The 20 unearthed in the past few weeks have convinced officials they have a historically significant find, they said.
Raimund Fastenbauer, a senior official with Vienna's Jewish community, told The Associated Press on Wednesday that he believes many of the up to 600 missing stones are still below ground and partially or fully recoverable.
If so, he said, the find, "is totally comparable to the significance of the Jewish cemetery in Prague," once the stones are restored and set up again.
Community leader Oskar Deutsch said the stones are up to 500 years old. While finding and restoring them is expected to last for years, he said more will likely be dug up in the next few weeks.
More than 185,000 Jews lived in Vienna before Hitler's Germany annexed Austria in 1938.
Of the more than 65,000 deported to Nazi death camps, only about 2,000 survived. Most of the rest emigrated, with only about 25,000 remaining by 1946, most of whom then left the country.
Today, there are an estimated 15,000 Jews living in Austria, largely in Vienna. After decades of officially sanctioned depictions of the country as Hitler's victims, Austrian governments have growingly acknowledged that their nation played a significant role in perpetrating Nazi crimes, and the city of Vienna missed few opportunities in attempts to make amends.
Restoration of the cemetery will serve as "another piece of the mosaic in the total picture" of Vienna's attempts to reconstruct its past, said a statement from Andreas Mailath-Pokorny, a city councilman in charge of restoration work at the cemetery.