Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready 3D Characters (2024)

2.1 Modeling Dynamics in 3D Vision and Graphics

Modeling dynamics is a fundamental task in computer vision and computer graphics.It involves both the spatial representation of 3D models and the formulation of their temporal behaviors.Recent approaches in this area can be categorized into mesh-based, mesh-free, reduced-order, and proxy-based methods.

Mesh-based elasticity models have been a cornerstone in modeling dynamics within computer graphics[terzopoulos1987elastically].One of the most commonly employed variants is the linear tetrahedral finite element method[Sifakis2012FEM, cutler2002procedural, bargteil2007finite, bouaziz2014projective],which requires a robust surface-to-tetrahedron algorithm like TetWild[hu2018tetrahedral].

However, both the linear finite element method and tetrahedral generation are sensitive to the geometry of the model, necessitating special handling for intricate shapes, such as thin features[Baraff1998cloth] and rod-like geometries[Bergou2008rods].For complex geometries, including non-manifold shapes, mesh-free methods, first introduced by[Desbrun1995implicit], provide an effective solution for addressing the challenges of mesh-based dynamic modeling.For example, smoothed-particle hydrodynamics (SPH)[monaghan2005smoothed, peer2018implicit] and material point method (MPM)[stomakhin2013material, Jiang2016MPM, muller2004point], which use particles as the main spatial representation, overcomes the difficulties of modeling different materials and complex dynamics in mesh-based methods, yielding high-quality results.

One major drawback of particle-based methods is their high computational complexity, resulting from the dense sampling required. To address this, researchers introduced reduced-order dynamics[barbivc2011real, barbivc2005real, barbic2007real], which decrease the degrees of freedom by projecting motions onto a limited set of deformation bases. This approach significantly accelerates computations and is applicable to both mesh-based and mesh-free methods. Recently, methods such as CROM[chen2022crom] and LiCROM[chang2023licrom] have advanced continuous reduced-order modeling using neural networks.

Reduced-order dynamics simplify temporal behaviors by reducing their dimensionality. Similarly, spatial representations can be simplified through proxy-based methods, which employ low-dimensional proxies to efficiently compute dynamics and subsequently apply deformations to the original geometry. Notable proxy-based techniques include Free-Form Deformation (FFD)[sederberg1986free] and cage-based methods[stroter2024survey, ju2008reusable, coros2012deformable, kim2014interactive]. These approaches utilize simplified spatial representations to expedite dynamic calculations, which are then used to deform the detailed geometry.

The most widely used proxy-based method is Linear Blend Skinning (LBS).It deforms a mesh by blending transformations from an underlying skeletal structure, allowing for efficient and intuitive animation of complex models. This technique was introduced by Magnenat-Thalmann et al.[magnenat1988joint] and further developed by Lewis et al.[Lewis2000pose].Recent advancements with neural networks have further enhanced LBS, making it suitable for geometry-agnostic simulation[Modi2024simplicits] and enabling more natural deformations through neural blend shapes[li2021learning].

2.2 Authoring Animation-Ready 3D Models

In the 3D animation and gaming industry, Linear Blend Skinning (LBS) is the standard for character animation. Creating animation-ready 3D models with LBS involves two key processes: constructing rigs and assigning skinning weights to the input models. Traditionally, these tasks are performed manually in 3D modeling software like Autodesk Maya, where artists place rigs and paint skinning weights to achieve the desired deformations.

Pinocchio[baran2007automatic] is a pioneering system for automatic rigging and skinning of 3D characters.It automates the embedding of a skeleton into a 3D mesh and assigns skinning weights based on vertex proximity to bones, enabling smooth and natural deformations during animation.Recent advancements in automatic rigging have leveraged deep learning techniques to enhance flexibility and accuracy.RigNet[xu2020rignet] utilizes neural networks trained on extensive datasets of animated characters to predict rigging parameters, enabling the generation of custom rigs suitable for diverse character models.Similarly, TARig[ma2023tarig] employs a template-aware approach, combining a humanoid skeleton with a shared graph neural network backbone to ensure precise joint positioning and skin weight estimation, thereby improving rigging efficiency.To enhance skinning quality, Li et al.[li2021learning] proposed neural blend shapes in addition to predicting rigs, achieving better deformation compared to static skinning weights.Another advancement in automatic rigging involves relaxing the requirement for the input pose to be in a standard position, such as A-pose or T-pose; Theisel et al.[Theisel2021bipedal] introduced a method that generates rigs for characters in arbitrary poses.Nonetheless, the inadequate quality, the limited robustness against complex inputs, and the unsatisfying time cost, all hinder the practical application of these auto-rigging methods.

An alternative approach to creating animation-ready models involves generating 3D models that are pre-bound to a rigged template. For instance, TADA[liao2024tada] produces textured human meshes by deforming a SMPL-X[pavlakos2019smplx] template through score distillation sampling (SDS) optimization[poole2022dreamfusion].Similarly, HumanGaussian[liu2024humangaussian] generates human models in Gaussian splats from a SMPL[loper2015smpl] mesh using a structure-aware SDS algorithm.Additionally, DreamWaltz-G[huang2024dreamwaltzg] employs a skeleton-guided distillation method combined with a hybrid 3D Gaussian avatar representation to achieve realistic generation and expressive animation.Although the aforementioned works have introduced various strategies to improve the shape diversity of generation, they remain constrained by the preset body ratio of SMPL mesh and are limited to avatars resembling realistic humans.Moreover, the fixed template skeleton topology also prevents their extension to non-standard bone structures.

3.1 Preliminaries

Intuitively, modeling the dynamics of an object requires a per-timestamp deformation of all the particles (i.e., vertices for meshes or splats for 3D Gaussians) that make up its geometry.Suppose that we have an object composed of $N$ particles and a desired dynamic sequence of $T$ timestamps. The temporal deformations of all the particles can be denoted by a matrix $\bm{D}\in\mathbb{R}^{T\times N\times d}$, where $d$ is the degrees of freedom (e.g., $d=6$ for rigid transformations).Generally, $\bm{D}$ has a lot of redundancy when representing real-world dynamics, so we would like to seek a low-rank approximation expressed by$\bm{D}^{T\times N\times d}\approx\bm{B}_{t}^{T\times K\times d}\bm{W}_{s}^{K$,where $K$ is the desired rank, $\bm{B}_{t}$ and $\bm{W}_{s}$ are temporal basis and spatial weight matrices, respectively.The practical applications of this low-rank form of dynamics include the sparse control paradigms[huang2024scgs, lei2024mosca] and the linear blend skinning (LBS)[loper2015smpl, pavlakos2019smplx] algorithm, where $\bm{B}_{t}$ is interpreted as the transformations/poses of control nodes/body joints, and $\bm{W}_{s}$ is the blend weights of the nodes or joints. Since the motions under the same joint setting can be reused, solving $\bm{B}_{t}$ for humanoid characters is usually converted to finding the transformations from all poses to a predefined rest pose.

Based on this formulation, given a shape representation of a character composed of $N$ particles, all the assets necessary for animating it (named as animation assets) are:1) $K$ jointsrepresented by bone head and tail positions, $\bm{J}\in\mathbb{R}^{K\times 6}$, which are connected based on any predefined skeleton topology;2) a 6-DoF transformation, $\bm{P}\in\mathbb{R}^{K\times 6}$, which transforms the input posed joints to a common rest pose;3) the blend weights $\bm{W}\in\mathbb{R}^{K\times N}$ between any joint-particle pair. The former two are also referred to as bone attributes since they are orthogonal to the input particles.

In the following subsections, we will elaborate on how these animation assets are obtained with our proposed unified framework, which is illustrated in Fig.2.For the sake of clarity, we will first introduce the core part of this framework — a shape autoencoder that describes the input geometry as compact neural latent representations.

3.2 Particle-based Shape Autoencoder

Geometry-aware attention.One of the benefits of our particle-based autoencoder is the support for various shape representations (as long as point sampling is possible on them). Nevertheless, for surface representations like mesh, much geometric information about the shape is lost during the sampling.This can lead to blend weight corruption problems that semantically distant but spatially close points may incorrectly share similar weight values.To address this issue, a non-intrusive way of injecting geometric awareness into the shape representation is proposed.Specifically, we extract the per-point normal values from the input mesh, as they carry rich geometry information.Then a lightweight fusing module is implemented using a simple attention layer as a branch way to the shape encoder, whose parameters are initialized so that the normal branch produces zero impact on the encoding results at the beginning.We randomly corrupt the normal values in training to make it more robust to low-quality meshes and also compatible with inputs like 3DGS in inference.Eventually, the attention mechanism adaptively decides what regions should care about the normals and whether the given normal values carry useful cues.

3.3 Coarse-to-Fine Shape Representation

With the shape autoencoder, our model can produce promising blend weight predictions, but joint outputs, particularly for fine-grained regions like the hands, still face convergence challenges.We attribute these issues to the ambiguity of the input points and view the aforementioned pipeline as a coarse stage that provides rough but valuable localization information about the input character.To enhance the shape representation, we adopt two strategies: canonical transformation and hierarchical sampling.The canonical transformation resolves pose ambiguity by aligning the input shape to a common orientation, while the hierarchical sampling method ensures higher sampling density in key regions like the hands to improve accuracy without increasing computational cost.Further implementation details are provided in the supplementary material.

3.4 Structure-Aware Modeling of Bones

With a learnable query assigned to each bone, we can effectively predict the bone-wise attributes. Although those queries adaptively attend to different regions of the input shape, they lack awareness of the skeleton topology, leading to imperfections in predictions, especially for deep-level bones in a kinematic tree.Therefore, we propose to model the bones in a structure-aware way, including additional designs for both network architecture and loss functions.

Next-child-bone prediction via causal attention.Without structure awareness of the skeleton, the model tends to produce independent homogeneous predictions.For example, the pose of a finger bone largely depends on the poses of its ancestral bones like hand and arm.If the pose of the direct parent is known, predicting the finger’s pose becomes easier, as it only requires accounting for the subtle transformation relative to its parent.To this end, inspired by the decoder-only next-token prediction paradigm prevalently adopted in LLMs[vaswani2017attention, brown2020language], we develop a structure-aware transformer based on a novel next-child-bone prediction architecture, as depicted in Fig.3.

Let us start from the bone-wise shape-aware embeddings, $\bm{F}_{b}=\{\bm{f}_{1},\bm{f}_{2},\dots,\bm{f}_{K}\in\mathbb{R}^{C}\}$, produced by the shape decoder (Sec.3.2).Instead of directly feeding $\bm{F}_{b}$ into a bone attribute decoder, we add several masked causal attention layers to enhance them. Each bone embedding, $\bm{f}_{i}$,$i\in[1,K]$, is regarded as an individual query token.Meanwhile, we use a bone attribute encoder to map the actual attribute values (joint positions or poses) into some same-shaped bone-wise latents$\bm{\mathcal{L}}_{b}=\{\bm{l}_{1},\bm{l}_{2},\dots,\bm{l}_{K}\in\mathbb{R}^{C}\}$.Then given a kinematic tree denoted by $\mathcal{K}$, to let $\bm{f}_{i}$ learn about the context information from its parent bone, we add the corresponding parent-bone latent $\bm{l}_{j}$,$j=\operatorname{Anc}(i,\mathcal{K})(1)$ to it, where $\operatorname{Anc}(i,\mathcal{K})$ returns a tuple of ancestral bones for the bone $i$ recursively from the nearest up to the root node in $\mathcal{K}$, and here we only select the first element (i.e., the direct parent bone).

We then feed the summation of $\bm{f}_{i}$ and $\bm{l}_{j}$ into a masked causal attention module that fuses that per-bone latent with those of other bones.We build an ancestral mask derived from the kinematic tree, ensuring that each bone learns from its ancestors and treats the states of its children as “future” information. The ancestral mask is defined by$\mathcal{M}\in\mathbb{R}^{K\times K}$ where $\mathcal{M}_{i,j}=1~{}\text{iff}~{}j\in\operatorname{Anc}(i,\mathcal{K})$. Equipped with $\mathcal{M}$, the cross-attention operation outputs per-bone embeddings that have both shape and structure awareness.Eventually, they are sent into a bone attribute decoder to be translated back to the predicted attributes.

Similar to the causal attention in language models, our structure-aware transformer follows an autoregressive process at the inference stage. The ground-truth parent-bone attributes are replaced by the predictions produced by the decoder. It takes several progressive iterations to complete the decoding and one level in the kinematic tree is predicted in an iteration, hence the name “next-child-bone prediction”.

Body prior loss.Our predictions of bone attributes are directly supervised by the ground-truth values. Since a character can have multiple possible solutions in practice, we further constrain the optimizing process using prior losses, so that the predictions follow some essential body patterns.Specifically, three kinds of prior knowledge are leveraged as follows.1) Bone connectivity: most bone heads have to be connected to the tails of their parent bones.2) Bone symmetry: typically in the predefined rest pose, the left and right parts of the skeleton have to be symmetric about the spine.3) Bone parallelism: in the rest pose, multiple bones belonging to the same limb have to share the same direction.

3.5 Training and Inference

The proposed framework is trained in an end-to-end data-driven manner, supervised by the $L_{1}$ losses with the ground-truth blend weights, bone positions, and pose-to-rest transformations, as well as the extra body prior losses (Sec.3.4).As introduced in Sec.3.3, a coarse-to-fine training strategy is adopted.In the coarse stage, we uniformly sample the input shape and predict only the bone positions, applying random 3D rotations for data augmentation to enhance generalization. Then in the fine stage, we transform the input character to canonical coordinates and apply the hierarchical sampling.

Once the training is finished, the model can take any particle-based character (in any pose with any global transformation) and infer its animation assets.With a fixed number of neural shape latents and a learnable discrete querying strategy, our framework achieves efficient inference, with speed less affected by input particle number.Generally, the whole feed-forwarding inference takes less than 1 second.

Although our framework requires a fixed definition of kinematic tree, and we typically choose a standard human skeleton to train it on, it can actually be extended to any predefined skeleton topology.If the framework has already been trained with the standard skeleton, adapting it to include extra bones is as simple as fine-tuning the final layer of the weight decoder and the extra learnable queries.In Sec.4.2, we will present some animating results of characters with additional accessories like long ears or tails.

4.1 Experimental Settings

Dataset.We utilize a collection of artist-designed 3D models from Mixamo[mixamo], consisting of 95 high-quality characters and 2,453 diverse motion sequences. Each character is preprocessed to conform to a standard skeleton structure with $K=52$ bones, and the motion sequences contain an average of 200 frames. We allocate 95% of the data for training and the remaining 5% for validation. During each training iteration, a character-motion pair is randomly selected, resulting in an effective dataset size equivalent to over 40 million frames.

Furthermore, our training framework accommodates additional skeleton topologies. We used Vroid Studio[vroid] to manually create 35 distinct anime characters, each featuring accessories like animal ears and tails. These characters were preprocessed to match the Mixamo skeleton definition, with the addition of extra bones for these accessories.

We select bipedal humanoid characters from the ‘ModelsResource-RigNetv1’ dataset[xu2019predicting] to construct a test set for quantitative comparison with existing automatic rigging algorithms. For qualitative evaluation, we gather a diverse collection of in-the-wild 3D characters from various sources, including the Objaverse dataset[deitke2023objaverse], generative tools[meshy, tripo], and other artist-designed models.

Baselines.We choose two lines of existing methods that can produce animatable characters as our baselines.Auto-rigging methods. These methods share the same workflow as ours, i.e., taking a 3D character as the input and outputting a rigged one with bones and blend weights.a) Meshy[meshy]: a commercial software that generates general 3D meshes from text/image prompts, and optionally rigs the meshes of bipedal humanoid inputs. Since it does not support direct 3D inputs, we feed an image of the test 3D character into it.b) Tripo[tripo]: another commercial software that basically has the same functions as Meshy, but differs in quality.c) RigNet[xu2020rignet]: a data-driven model that can rig any mesh-based geometry. We feed the same test characters to both RigNet and our model.Template-based avatar generating methods.By leveraging the SMPL template[loper2015smpl, pavlakos2019smplx] as a strong prior, this line of methods directly generates animatable humans.a) TADA[liao2024tada]: mesh-based avatar generation. b) HumanGaussian[liu2024humangaussian] (HG): Gaussian-splatting-based avatar generation.The generated avatars of these two works are rigged with their corresponding template skeletons and blend weights.

Metrics.In the ablations conducted on the test split of Mixamo dataset, we report the percentage error between the predicted and ground-truth animation assets.As for the quantitative comparison with RigNet[xu2020rignet], we employ several metrics to evaluate the quality of skeleton predictions, i.e., the IoU, Precision and Recall of bone matching, and the CD-J2J, CD-J2B, and CD-B2B (Chamfer distances between joints and bone line segments)[xu2019predicting].For other in-the-wild test cases without ground truth, we provide extensive qualitative comparison by visualizing the skeletons, blend weights, and animating results. Due to the different skeleton topologies, the same motion sequence is usually not applicable to other baselines. For fairness, we manually set the bones from different methods to similar poses in 3D modeling software and render the animations.

4.2 Comparison Results

In Fig.4, we compare the proposed method with two commercial software, Meshy[meshy] and Tripo[tripo].Meshy tends to enlarge the influence regions of skinning weights, leading to incorrect deformations of clothes or muscles. It also fails to extract the rest pose of some complex input shapes and animates the characters in a weird way.Tripo does better in blend weights, but often produces inaccurate and unmatched bones. Plus, it also has difficulty determining the correct rest state for unusual input poses.Moreover, both of the two cannot predict finger bones, resulting in coarse hand poses in the running scenario.In contrast, our method produces complete and accurate bones as well as suitable blend weights, so the animations are also natural and fluid.

In Fig.5, we evaluate the auto-rigging method RigNet[xu2020rignet] with given test meshes.It can be observed that RigNet often generates unnecessary bone structures, e.g., a cycle graph (the old man case) and a star topology (the bear case). These bone errors are also manifested in Tab.2.Besides, its weights usually fail to maintain spatial smoothness, resulting in fragmented animating results.

In Fig.6, we further compare two generative works that perform text-to-3D-avatar generation. TADA[liao2024tada] produces textured meshes and HumanGaussian[liu2024humangaussian] chooses to use Gaussian splats. Our framework is directly compatible with both their representations. Since these two works generate shapes based on the well-defined SMPL[loper2015smpl] template, their joints’ locations and weights are good enough for animating. Nevertheless, our method can achieve comparable, if not superior, quality using a template-free pipeline, without any prior knowledge of the character’s shape or pose parameters. Furthermore, although these methods exhibit some deviations from the template mesh, they remain constrained by the preset body ratio of SMPL and are limited to shapes resembling realistic humans. In contrast, our method can be applied to a much wider range of character shapes.

We show more results produced by our method, focusing on the details and some tricky cases. Each sub-figure of Fig.7 proves some advantages of our methods, which is interpreted one by one as follows. (a)Fine-grained control of fingers. (b)Capacity of abnormal shapes. (c)Complex input poses. (d)Efficiency for high polygon models. (e)Support for asymmetric inputs. (f)Adaptation to non-existing bones. (g),(h)Extension to extra bones.

4.3 Ablation Study

We conduct ablative experiments to validate the effectiveness of the proposed components and strategies, as shown in Tab.3 and Fig.8.It can be observed that the canonical transformation is vital for the proper convergence of our model, as it greatly reduces the distribution dispersion of inputs.The large improvement brought by hierarchical sampling also indicates the importance of the proposed coarse-to-fine shape representation. As illustrated in Fig.8 (a), sampling more in the fine-grained hand regions leads to a more accurate estimation of finger bones.Besides, the injection of additional normal information via geometry-aware attention boosts the performance, especially in weight prediction. We visualize the attention score of the input point clouds in Fig.8 (b), where brighter color indicates more attention to normals rather than coordinates. The model adaptively learns to rely more on normals in regions like inner thigh and between fingers since coordinates become less discriminative there.Furthermore, accurately modeling pose transformations presents a complex challenge. When poses are predicted as local rotations rather than global dual quaternions, the model is limited to handling simpler inputs. Without a body prior loss, predicted limb and spine bones often exhibit slight but critical offsets, disrupting connectivity and symmetry.Additionally, in the absence of a structure-aware transformer, deeper-level bones, such as those in the fingers and feet, are susceptible to inaccuracies in pose-to-rest transformations. Their poses are heavily influenced by ancestral bones, where even minor inconsistencies can result in significant deformities.

S2.1 Coarse-to-Fine Shape Representation

To address the limitations of the proposed framework and boost the performance, we introduce an additional design at the input side of the autoencoder, i.e., the coarse-to-fine shape representation (Sec.3.3).In addition to the overall framework presented in Fig.2, we include two separate pipelines in Fig.S1 here for a clearer illustration of the coarse and fine training stages.In the following content, we will provide more details of this part, particularly regarding the motivations and implementation choices.

With the particle-based shape autoencoder, our method can already produce fairly good results of blend weights. However, the joint outputs are still unsatisfying in fine-grained regions like the hands. Meanwhile, the pose prediction can hardly converge.We attribute these issues to the ambiguity of the input points and treat the lite training process (taking uniformly sampled points as input and only predicting bone positions, as shown in Fig.S1 upper) as a coarse stage that provides rough but valuable localization information about the input character.To be specific, we exploit some of the coarse joint locations to gain a finer shape representation by applying two different strategies, i.e., the canonical transformation and the hierarchical sampling.

S2.2 Networks and Hyperparameters

For the shape autoencoder, we use a point cloud of size $N=32768$ as the input. All the points are sampled on the surface of the input mesh. In the fine training or the inference stage equipped with the hierarchical sampling, 50% of the points are uniformly sampled and the other 50% are sampled near the hand joints.The hyperparameters of the shape autoencoder are consistent with the original setting of 3DShape2VecSet[zhang20233dshape2vecset], which internally uses $M=512$ latents (each of them is a $512$-D vector) for the neural field.Our framework is trained on 8 NVIDIA A100 GPUs.The learning rate is linearly increased to $1e-4$ within the first 1% iterations (warm-up), and then gradually decreased using the cosine decay schedule until reaching the minimum value of $1e-5$.

S3.1 Mixamo Dataset

To obtain sufficient training data of 3D characters with high-quality geometry and animation assets, we collect the artist-designed 3D models from Mixamo[mixamo] to form a dataset, which comprises texture meshes of 95 exquisite characters (each has an average of 15000 vertices), along with 2453 diverse motion sequences (each has an average of 200 frames).All the characters are preprocessed to share a standard skeleton structure with $K=52$ bones (the leaf bones are removed).For characters with non-standard skeletal structures originally, the blend weights of those bones are transferred to their topologically nearest ancestral bones within the standard skeleton.If some standard bones are missing in a character (e.g., armless person), we mask their corresponding values (weight channels and bone-wise attributes) in loss computing.We use 95% of the data for training and the remaining 5% for for validation.During each training iteration, we randomly choose a character-motion pair to get the corresponding animation assets, resulting in an effective dataset size equivalent to over 40 million frames.In Fig.S2, we show some example characters and motions from the Mixamo dataset.

S3.2 Vroid Dataset

Our training framework can also be extended to different skeleton topologies.To demonstrate this capacity, we use Vroid Studio[vroid] to manually create 35 different anime characters (30 for training and 5 for validation) with additional accessories including two rabbit-like ears and a fox-like tail, as shown in Fig.S3.These characters differ in body shape, clothes, hair style, and the shape of ears and tails.They are all preprocessed to share the same skeleton definition of Mixamo[mixamo] but with extra bones of the accessories. Therefore, they can also be animated with any of the Mixamo motion sequences during the extra-bone training (i.e., fine-tuning the final layer of the weight decoder and the extra learnable queries, based on the standard-skeleton model pretrained on the Mixamo dataset).The experiments show that with our framework, 30 training characters are sufficient to obtain a good model that produces promising predictions for extra bones.

S4.1 Additional Comparison Results

S4.2 Additional Visualizations