Asset Pipeline 2022 Update

In the past few weeks the Blender Studio team spent some time redesigning their Asset Pipeline. This article is a comprehensive summary of the outcome of the process.

---

The core design for the Asset Pipeline is the following: take advantage of how Blender stores data and thus enable artists to work in parallel, without having to wait for the output of another task. This is done by merging the different tasks outputs in a publish file. On top of that, use Subversion to provide a safety net and a way to granularly log changes.

The foundations of the Asset Pipeline were established during the Settlers project, and were further iterated on during the production of Sprite Fright.

In this picture you get an overview of what is going to change after the last round of iterations.

The original system was quite simple. We would have one Asset Publish file, that gets constantly updated and linked in a shot. When Artists perform a publish they would actually be editing the publish file and use the Character Pipeline Assistant to pull the changes from the task files.

In the new system we introduce asset versions, requiring a shot to link to a specific version of an asset. That link can be updated to point to another version. When artists perform a publish, they are now in the task file and push their changes to the asset versions.

Note: A publish can actually push to multiple versions at the same time.

On top of Asset Versions, we rely on SVN revision to track changes more granularly.

Design Challenges

• Backwards Compatibility: Because we had only one asset publish file that would get linked in all the shots, an animation that was done months ago could break if the rig received changes in the meantime. This would make it difficult to render or re-render a shot.

• Open Heart Surgery: Changing the one asset version always means a risk of breaking things in other files that link to it.

• Poor logging: When things break it can be hard to find out when and why, as artists sometimes neglect good logging.

• Naming Conventions: A lot of tools were based on very specific and rigid naming conventions, which is something we want to move away from.

• Scattered Tools: Finally, we use this project as a chance to reorganize our tools and create a document that explains how everything comes together in a bigger picture.

Improvements

To address these challenges and make the pipeline more scalable in the future we will introduce some new concepts.

• Asset Versions: Partial asset versioning system, backwards compatibility

• Asset States: Determines if an asset can be used in production

• Asset Builder Pipeline 2.0: More customizable and new features

• Asset Collections: Asset IDs, properties for studio sssets

• Kitsu Integration: Centralized entity definition for easy access, share IDs

• Automated Test Pipeline: Adds safety layer before commit

• Tracking & Logging: Metadata files, which asset version is used where

• Asset Updater: Track changes to assets linked in a shot

Task Layers

Task Layers are a new concept introduced in this update.

When setting up a new production, we need to create a configuration file we define how an asset is assembled, through Task Layers. Each Task Layer owns a domain of data and all of them together describe the entire data of an asset.

Task Layer can depend on each other and overlap. In this case the shading Task Layer needs the data of the rigging Task Layer, because without objects there can be no material assignments.

Task Layers can be pulled in from different sources by following a set of merge instructions that describe exactly how all layers are puzzled together.

And last but not least, Task Layers can be locked or live. A locked Task Layer cannot be changed anymore, meaning the data it owns stays the same. A live Task Layer, on the other hand, can be changed.

The new System

With the Task Layer theory out of the way, an asset directory might look something like what is picture in this illustration.

We have a couple of Asset Task files and some Asset Publish files. Tasks can publish the selected Task Layers to the Asset Publish files. Notice that a change will be published to all versions in which the selected Task Layer is in the live state.

This enables the following scenario: if the rigging department does a change that would break animation in a number of shots, they would create a new Asset Publish, which locks all rigging Task Layers of the previous asset publishes. That way the animation of those shots would not be affected, as it references the "legacy" rig. The shading Task Layer is still live in the previous versions, which means they will still receive regular updates on every shading publish. Even tough the rig is considered outdated, we can still reliably render that shot because the asset still has the latest shading state.

This partial versioning feature gives us great flexibility and fixes the backwards compatibility problem we had earlier. In the previous pipeline, such a breaking rigging change could have broken the animation.

A task file can also push multiple Task Layers at the same time. However, it makes sense to create one task file for each Task Layer so artists can work in parallel as much as possible.

One important aspect in this scenario is that artists need to keep the other Task Layers up to date. The rex.shading.blend task for instance is responsible for material assignments, shaders, uvs and other things. Whenever rex.rigging.blend publishes new rigging changes, the rex.shading.blend should make sure to get these changes as well. Otherwise, it might miss an object or shade an object that does not even exist anymore. For this reason it's important to perform a pull before an update push.

When artists pull the other Task Layers --on which they are not working on--, they always pull it from the latest asset publish.

This is all automated by the pipeline and makes sure that the task files are always as close as possible to latest asset publish.

Versioning

The goal is to create as few asset versions as possible and only create a new version when a department does a breaking change. A change is considered breaking if it breaks shot tasks or if it breaks other Task Layers of the same asset.

A new version of an asset will always be in the 'Review' state and needs to be first approved by the team before its available in the production.

Asset Builder

The Asset Builder is responsible for pushing a Task Layer from an Asset Task to an Asset Publish or pulling a Task Layer from the latest Asset Publish to the current Asset Task.

The goal of the Asset Builder is to transfer a set of selected Task Layers from a source collection to a target collection. The Asset Builder runs through different phases.

1. Process all the Task Layers that are defined in a Production level config file
2. Move on to custom tasks that can be defined per Asset.
3. Run post merge hooks. These are usually repetitive steps that the artist has to do to prepare data for publishing (and often revert it again for working).

If you want to learn more about this, check out the Developer Guide for the asset-pipeline.

Asset States and Tests

Two new concepts in the Asset Pipeline will be Asset States and Tests.

While Asset States are pretty self-explanatory, let's have a quick look at Tests. We often push changes to multiple asset versions at the same time. For example, all asset versions receive the same shading updates (because the shading Task Layer is live) but the underlying rig and objects might be different (as the rigging Task Layer is locked in an older asset version). It's really important to test if everything went well in the transfer.

The plan is to build an automated test pipeline that can run an Asset Version throug a set of technical tests and possibly also display a rendered picture to visually confirm that everything went well.

SVN Integration

To simplify the SVN commit user experience, the plan is to create an SVN client as a Blender add-on. In order to reduce errors (misnamed, missing files, poor logging) the goal is to provide as much help as possible to the artist, for example by detecting all external files of the current .blend file and commit them if they were modified.

File Structure

The Asset Pipeline relies on metadata to establish states, versioning and other concepts. The metadata will be saved in XML format next to the file that the metadata belongs to. That way, we define a clear and self-contained syste which does not rely on an external (online) database to track and sync changes. An XML file is also human readable and easily editable.

We still have a connection to Kitsu, and use it for initializing the Assets by fetching project information and IDs.

With that, we end up with an asset directory that looks like this:

Asset Updater

The Asset Versions that are produced by the Asset Builder are used in shots or set files. Once there is a new version of an Asset, an artist working on a shot should have an easy way to update them. That's the purpose of the Asset Updater. It is very light weight compared to the Asset Builder and is mostly a notification system.

The Asset Update generates notifications by:

1. Scanning scene for found Assets
2. For each asset check the asset publish directory for all versions (Ignore asset publishes in review state)

The Glossary

As a little bonus, here is our Glossary.

Current State

This project is actively in development. It is part of the Blender Studio Tools and you can clone it from here to get the latest changes.

There is also a zip file of the project uploaded to the Pipeline and Tools section.