I Tried Rokoko Video - Free Ai Motion Capture

Over Xmas, I had a play around with Rokoko Video, a free service that was released a month or 2 earlier. I think it’s important to keep on top of new tech that’s coming out and I wanted to see if I could include it in my animation pipelines to get stuff done faster and more efficiently.

Check out their website HERE, if you want to learn more about it and if you want to give it a go, here is their official how to get started video. You can either upload videos or record directly from your webcam.

I chose to upload videos. Below you can what my play-around turned out like (Straight from Rokoko into Unreal Engine 5. This whole process took me maybe 30 min, with a bit of time wasted as I was working out how to use stuff.

My Pipeline

Here is the pipeline I ended up using for the whole process from sign-up to importing an animation into Unreal Engine 5:

Part 1 - Rokoko website

1. Signed up via their website - Link to sign-up

2. Filmed some short video clips on my iPhone, then sent those videos to my email (to get them on my PC more easily)

3. Headed over to the Rokoko Video Dashboard to begin processing - Link to the dashboard

4. Followed the steps to create a new “Scene”

5. Chose Select Local Video and then upload one of my videos (300MB max)

6. Previewed the video and used the timeline and white handles to trim to the range I wanted

7. Hit the “Turn Into Animation” button in the top-right

8. Wait a few min for the footage to process. A relatively quick process. For me a 5-second clip took less than 1 and a half minutes

Part 2 - Rokoko Studio (desktop APP)

1. While waiting for the processing I download Rokoko studio (needed to process and export the animation into something usable). Make sure to download the Rokoko Studio Beta For Windows (or Mac) - Link to Download

2. Installed Rokoko studio, opened it up and logged into the app

3. On the homepage click the buttons under Recently Added Scenes, this loads your processed animations into a scene

4. Click on the Animation Clips rollout on the left-hand side.

5. This brings up your clip, double-click on it and it will load it onto the character

6. It sets foot locks automatically, but you can adjust these by adjusting the start-stop ends of the sliders. You can also create a new one by double-clicking in a blank area on that tracks layer

7. To export look to the right side of the screen for an export tab, clicking this brings up a bunch of export options, there are a bunch of settings you can play with but for my tests, I used the Skeleton Present: Unreal Engine

8. Hit Export Clips and boom, exported

Part 3 - Unreal Engine 5

1. Hit the open folder next to it to find its location in your file explorer

2. Drag those FBX files into Unreal Engine (in my case 5), pick the correct skeleton and there you have it all in the engine

Unreal Engine animation to skeleton Problem

I came across an interesting problem when importing to Unreal and wanted to write a little more about it for other people that may run into it:

I had issues importing onto the UE5 mannequin skeleton, but it was fine on the UE4 mannequin skeleton

The Left GIF contains the problem the Right GIF is with it fixed

The default retargeting settings are different between the older UE4 mannequin and the newer UE5 ones which is creating the difference. I guess in the future Rokoko might add in a skeleton preset made for these new UE5 skeleton scales. You would also be able to fix this kind of stuff if you imported and tweaked in say Blender first then into Unreal. In the meantime, you can fix this by adjusting some of the retargeting settings on the UE5 character skeleton.

Here is a link to the Unreal engine documentation explaining retargeting options and below are the settings I tweaked on the skeleton:

Overall experience using Rokoko Video

The Good

Overall it was a good experience, simple and enjoyable. I would probably think about using it (combined with Blender) if I found myself needing to make a large amount of background character animation loop variations in a short period of time. Outside of that some of the pros were:

• Free

• Only need one video from one angle (or webcam view)

• Good animation base to start with (can capture some natural human motion moments that motion capture is useful for)

• Much quicker pipeline and processing than some of the other cheaper mocap solutions I’ve used in the past (e.g. iPi soft)

• Devs say that this is the first version of their tool and they want to do a lot more

The Bad

There were definitely some problems with the output animation. The main issues I had with it were:

• Minimal hand/wrist tracking and no fingers

• Larger rotations seemed to be quite off (makes sense as is from one point of view)

• The foot lock didn’t seem to work very well during the processing and needed to be manually set a lot

• Center of mass is all off, see the video below when I rotate the camera 90 degrees (makes sense as is from one point of view)

Expanding my pipeline:

If I was going to put more time into building out a pipeline with Rokoko Video I would definitely look into taking the animation and cleaning it up before importing it into Unreal Engine. Rokoko supports Blender, so that would be something I would look into. There are a bunch of tutorials on Youtube of people doing this for example this video by Grant Abbitt.

I’ve also been wanting to try out/look into Cascadeur, an AI-Assisted Keyframe Animation Software, which can help with posing, physics and secondary motion. I’m thinking potentially Rokoko Video and Cascadeur could complement each other well and would help in reducing some of the problems I talked about earlier (specifically the center of mass being so off).

Conclusion:

It’s crazy to think that this is coming from a single camera angle and it seems like a great step forward in technology, which a lot of people are going to benefit from. It’s going to make animation and more specifically motion capture more accessible to smaller teams and solo indie devs, which is great. Also big kudos to the team at Rokok for having a FREE version of it. I’m super excited to see where they take it in the future and if it continues to grow/expand I would jump on the paid versions of it, for sure.