Social infrastructure are physical structures that draw people out of their homes and encourage interaction with others. They have been described as physical places and organizations that shape the way people interact and spaces that facilitate social connection. We investigate the role of social infrastructure, encompassing elements like parks, libraries, and cafes, in fostering tolerance and mitigating white flight behavior – the tendency of majority groups to relocate in response to an influx of minority group members. We argue that accessible social infrastructure facilitates positive intergroup interactions, reducing the likelihood of white flight. Leveraging OpenStreetMap data and Swedish population registers, we empirically test this hypothesis in Stockholm. Our findings suggest that social infrastructure can indeed mitigate white flight behavior, but only if it is in close proximity to a majority members’ home. Increasing amenity density on the other hand can increase the probability for white flight behavior. This research contributes to the literature by linking the built environment, social interactions, and urban composition, offering insights for city planners and policymakers.