Buildings are widely regarded as potential sources for demand flexibility. The flexibility of thermal and electric load in buildings is a result of their interactive nature and its impact on the building’s performance. In this paper, the interaction of a building with the three interaction counterparts of the physical environment, civil infrastructure networks and other buildings is investigated. The literature review presents a wide variety of pathways of interaction and their associated potential impacts on building performance metrics such as net energy use, emissions, occupant comfort and operational cost. It is demonstrated that all of these counterparts of interaction should be considered to harness the flexibility potential of the buildings while maintaining other buildings performance metrics at a desired level. Juxtaposed with the upside potential for providing demand flexibility, numerous implementation challenges are identified that are associated with the evaluation and financial valuation of the capacity for demand flexibility, the aggregated flexibility potential, as well as the control and communication to facilitate the interactions.