Whitefish Bay’s History of Segregated Housing

Whitefish Bay. Photo by Graham Kilmer.

It was sometime in the 1960s or ‘70s that some local wag gave Whitefish Bay the nickname of “White Folks Bay.” Like many jokes the jibe contained a hard kernel of truth: The 1960 Census reported that 4,440 homes in the village were owner-occupied – and not one had a non-white owner.

That said, it was certainly not the only suburb like this in Milwaukee County, and since then there has been some progress for Whitefish Bay: the most recent Census data shows the village is 87% white, 6% Hispanic or Latino, 3.3% Asian and 2.6% Black or African American.

But the recent decision by the village architectural board to reject an apartment building with affordable housing subsidized by Milwaukee County raises questions about its receptivity to lower income or non-white residents. The style of the housing seems a very good fit for Whitefish Bay, and as Urban Milwaukee reported, the proposed location is at a “dead corner,” as one village board member described it, that has been vacant for years, with a “dilapidated” building sitting there, as a developer noted.

While Whitefish Bay was incorporated in 1892, it had a small population going into the 1930s. The boom in new residents that began the mid-1930s was the direct result of federal programs that were restricted to whites and reinforced the restrictive covenants created by communities like Whitefish Bay, part of a national pattern documented by Richard Rothstein’s 2017 book, The Color of Law. It was not a coincidence that only white people were homeowners in the village by 1960: it was part of a plan.

“The Federal Housing Administration and Veterans Administration mortgage programs largely shaped the ‘character’ of Whitefish Bay – both physically and demographically” says Lois Quinn, who researched the history of racial covenants in Milwaukee County for the Metropolitan Integration Research Center in the 1970s. “For decades the FHA made housing construction, home purchase, and modernization financing affordable for the white middle and upper classes, while denying opportunities for African Americans, the laboring classes, and recent immigrants. After World War II, the VA provided new housing opportunities for war veterans, while excluding veterans of color.”

Quinn’s 1979 study found that, “By the 1940’s at least sixteen of the eighteen Milwaukee County suburbs were using racially restrictive covenants to exclude black families from residential areas. For example, subdivisions established in 1927 in Cudahy, Shorewood, West Milwaukee, Whitefish Bay, and Wauwatosa excluded all non-Caucasian families.” Many of the racially restrictive covenants “extended into the late 1960’s and mid-1970’s.”

Those covenants were welcomed by the FHA, which was created in 1934 as part of the New Deal with the goal of encouraging homeownership: it insured bank mortgages that covered 80% of purchase prices, had terms of twenty-years, and were fully amortized. The FHA Underwriting Manual specified that neighborhoods with racially restricted covenants, designed to prevent “the occupancy of properties except by the race for which they are intended,” were given preference for FHA-approved loans. The manual emphasized the importance of having schools that “should not be attended in large numbers by inharmonious racial groups.” It noted the importance of natural or “artificially established barriers” to protect a community from “the infiltration” of “lower class occupancy, and inharmonious racial groups.”

The FHA manual also instructed appraisers to investigate the neighborhood of loan applicants “to determine whether or not incompatible racial and social groups are present,” so that “an intelligent prediction may be made regarding the possibility or probability of the location being invaded by such groups.”

“Because the FHA’s appraisal standards included a whites-only requirement, racial segregation now became an official requirement of the federal mortgage insurance program,” Rothstein’s book explained. “The FHA judged that properties would probably be too risky for insurance if they were in racially mixed neighborhoods or even in white neighborhoods near black ones.”

In 1936 Whitefish Bay provided space in its village hall for an FHA branch office so that local residents could easily apply for the mortgage insurance program, as the Shorewood Herald reported that year. The office was open 51 hours a week (including Saturdays and Tuesday and Friday evenings), and a second FHA branch office was opened nearby in the basement of Atwater School in Shorewood. The FHA told the Shorewood Herald that these North Shore branches were opened “so as to make all trips to the downtown [Milwaukee] offices unnecessary.”

Appraisal maps prepared by the federal Home Owners Loan Corporation in 1938 show how it determined which Milwaukee County residents were preferred for federal housing support. All of Whitefish Bay was located in the top two lending classifications, given its many residents with “business, professional, and white collar clerical” occupations, few families receiving relief payments, few foreign-born families, and “no Negroes.” By contrast the nearby Riverwest neighborhood in Milwaukee was considered an undesirable area for federal lending support, given its population of “laborers,” with 80% of people being Polish, Russian and German “foreign-born” families, and “many” relief families. Listed as a “detrimental influence” in Riverwest was “Polish infiltration.”

“After World War II, the newly established VA also began to guarantee mortgages for returning servicemen,” Rothstein writes. “It adopted FHA housing policies, and VA appraisers relied on the FHA’s Underwriting Manual. By 1950, the FHA and VA together were insuring half of all mortgages nationwide.”

These programs created a national boom in homeownership, which rose from 44% of Americans in 1940 to 62% in 1960. Much of that growth was in suburbs like Whitefish Bay, which saw its population grow from  5,362 residents in 1930, at the beginning of the Great Depression, to 18,390 in 1960. That included just 16 Black residents, none of them homeowners.

These policies didn’t just institutionalize segregation nationally and in Milwaukee County, but made it harder for non-white Americans to become homeowners, which has been one of the principle ways the nation’s middle class has accumulated wealth.

Shorewood, too, benefitted from how the FHA and VA targeted only middle and upper class whites for homeownership. And like Whitefish Bay it has little affordable housing. But unlike Whitefish Bay its leadership voted unanimously to create an affordable housing program in 2021, as Urban Milwaukee reported. “This is a moment in time where we can reverse historical segregation that has plagued not only Shorewood, but this region for decades,” said Ellen Gilligan, a village resident and head of the Greater Milwaukee Foundation, at a village board hearing.

As Whitefish Bay board member Raisa Koltun told Urban Milwaukee, adding some affordable housing to the village should be seen as a plus for the community. “Whitefish Bay is obviously not a cheap place to live,” she said. “So having the ability for people, like teachers and firefighters and people who work here be able to afford to live here is very appealing, and then, ideally, send their kids to school here.”

It is also a chance for Whitefish Bay to live up to the racial equity statement it created in 2020, which states that “We are committed to continuing to make Whitefish Bay an inclusive, accepting village, where all are welcomed regardless of race, ethnicity, gender, religion, sexual orientation, mental/physical ability or socio-economic status.”