How Spotify models its internal software architecture Computerworld

The Spotify music streaming service relies on a sophisticated hair of nested software. So that its teams of developers can communicate with a common language on this software architecture, the Swedish company has modeled it in several diagrams, also using the C4 model.

To deliver music, podcasts and videos, Spotify relies on “an incredibly complex network of thousands of interconnected software systems owned by hundreds of teams”. This is how the Swedish streaming service describes its application heritage developed for the 188 million premium subscribers it has worldwide (figures for the 2nd quarter of 2022). In order to communicate around this sophisticated software architecture and the associated catalog, the IT team has developed a common language and concept definitions, in other words a system model. The latter presents a set of basic entities and abstractions that can be used to synthesize data about the health of these applications, their properties and their dependencies. Last month, the European supplier described this model in a ticket.

Spotify’s software is modeled using three main entities: APIs (the links between different components, defining the interface between them), the components themselves which represent different parts of software (for example, a back-end service, a website, a data pipeline, a library) and the resources, i.e. the infrastructure required to operate a component that runs (for example, databases, virtual machines, storage spaces referred to in English as buckets). “The ability to map and be able to track software components in a catalog has been extremely valuable to us,” Renato Kaman, Senior Software Engineer, and Johan Wallin, Software Engineer at Spotify explain in the post. “It gave us a better understanding of the scale and development of our software. But as our catalog of individual components has grown, these components have become increasingly difficult to understand, examine, and relate to each other. So the IT team introduced a few more abstractions, Systems and Domains, to make it easier to understand the software ecosystem as a whole. Systems are entities that work together to perform certain functions, Domains are entities and systems related to parts of the business. “By expressing this model as metadata, we were able to create a software catalog that tracks components, their owners, their dependencies, and the lifecycles of our ecosystem,” the post’s authors state.

Build on the C4 model

Like a lot of other IT teams, those of Spotify use physical whiteboards to schematize their developments, filling them with felt-tip moult rectangles and arrows. But these objects are attached to a location (a desk) and, from one team to another, the meaning of the symbols used sometimes requires explanation. The objective was therefore to develop a simple way to visualize the software architecture, optimizing this method to facilitate communication between users. “We wanted diagrams that could allow us to understand our software with minimal context,” the post says. “But the solution was right in front of us: the C4 model that our software developers were using.”

It is therefore the C4 model that was chosen to create the Spotify diagrams. This model itself has a collection of software abstractions that the streaming service has combined with the abstraction layer of its own modeling system. It was then necessary to redefine all the basic diagrams to document the architecture and design of the system. This gives three diagrams: the “System landscape”, the “System context” and the “System components”. The first describes a set of related systems, how they are connected, and the external systems on which they depend. The second describes how a system fits into a larger context of dependencies, dependents, and users. The third describes how a system is built from unitary components, which corresponds to the 2nd diagram (or level) of the C4 model which includes 4 of them: 1/Context, 2/Container, 3/Component, 4/Code.

Backstage for auto-updating diagrams

Using its catalog of software and components and the associated metadata, Spotify was able to transform its ecosystem into diagrams on which it also benefits from automatic rendering and interactive navigation. “This allows us to communicate and visualize our entire architecture using a common language and model,” explains the music streaming service. This automation of architectural diagrams ensures that they are always up-to-date with the design as expressed in the metadata. There is therefore no need to update them as the system evolves, nor to worry about whether the proposed visualization is outdated, explain the software engineers. Automation of architecture diagrams is done in Backstage, an open source platform for building developer portals. Spotify explains that it is very useful for him to have the model of his system in Backstage to explore it and discover the components, to understand the life cycles, to find the owners of the software components and the relationships between the components. And ultimately, to improve collaboration between teams.

In conclusion, if it becomes too difficult to understand a software architecture, visualizing it in the form of diagrams can be very profitable. Companies that want to try it can rely on models such as C4, Spotify System Model or create their own. “Having a standard notation is essential to make software diagrams understandable,” insists Spotify. “Drawing on best practices from the C4 model, we were able to use our System Model to create easy-to-understand software visualizations with minimal context. This combination has made it possible to create a common language used throughout the organization to facilitate communication”, conclude the two engineers, taking advantage of the post to launch a call to recruitment.

We would love to thank the writer of this short article for this outstanding material

How Spotify models its internal software architecture Computerworld

Our social media profiles here and other pages related to them here.