The Annex is bounded by Dupont Street to the north, Bloor Street to the south, Bathurst Street to the west and Avenue Road to the east. Bordering the University of Toronto, the Annex has long been a student quarter and is also home to many fraternity housing and members of the university's faculty.

The Annex is mainly residential, with tree-lined one-way streets lined with Victorian and Edwardian homes and mansions, most of them built between 1880 and the early 1900s. The architectural detail is among the finest in the city, ranging from pyramidal roofs and turrets to recessed grand archways and wooden spindled porches. The 1950s and 1960s saw the replacement of some homes and mansions with mid-rise and a handful of high-rise apartment buildings.