This paper shows an interplay of music and mathematics which strongly differs from the usual scheme reducing mathematics to a toolbox of formal models for music. Using the topos of directed graphs as a common base category, we develop a comprising framework for mathematical music theory, which ramifies into an algebraic and a topological branch. Whereas the algebraic component comprises the universe of formulas, transformations, and functional constraints as they are described by functorial diagrammatic limits, the topological branch covers the continuous aspects of the creative dynamics of musical gestures and their multilayered articulation. These two branches unfold in a surprisingly parallel manner, although the concrete structures (homotopy vs. representation theory) are fairly heterogeneous. However, the unity of the underlying musical substance suggests that these two apparently divergent strategies should find a common point of unification, an idea that we describe in terms of a conjectural diamond of categories which suggests a number of unification points. In particular, the passage from the topological to the algebraic branch is achieved by the idea of the gestoid, an ``algebraic'' category associated with the fundamental groupoid of a gesture.