All About Rotoscoping: Concept, History and Application

When you ask animation artists and VFX technicians what is rotoscoping, you might get entirely different answers. Depending the aim of the work, people may refer to different techniques when applying the so-called rotoscoping process.

For instance, when you search rotoscoping tutorials for After Effects, most YouTubers will teach you how to separate characters from the background, or how to remove unwanted objects without affecting the surroundings.

Alternatively, if you are curious about what is rotoscope in animation, everyone will be talking about tracing the picture and transcribing the film frame by frame.

Aren't they the same thing? What is rotoscoping in the general sense? Why do people from different professional backgrounds talk about rotoscoping in a relatively different meaning? The article explains everything you need to know about rotoscoping in various contexts.

1. What Is Rotoscoping

When it was initially invented by Max Fleischer, rotoscoping (or roto) refers to an animation technique to trace over each frame of a live-action film, and draw it frame by frame onto animation papers. In that way, rotoscoping turns a live-action film into a cartoon, with characters replicating all the actions of the real people, thus avoiding jerky and clunky motion in previous animations.

It is done by projecting the live-action film stock onto glass with light and manually tracing over the image on each frame. This kind of rotoscoping still exists in the digital era, only that the glass and projector are replaced by digital drawing boards and computers.

The idea of adjusting scenes frame by frame was then involved in making mattes, and the main aim was to create scenes that are otherwise non-existed in the real world. Later the concept of rotoscoping was upgraded by VFX technicians and artists, and the technique plays an important role in live-action films. Rotoscoping is used to extract object and replace the background, create fantasy objects such as the lightsaber in Star Wars.

To better understand how rotoscoping come all the way to what it is today, here is a quick glimpse of the history of rotoscoping.

2. Rotoscoping History

2.1 Max Fleischer and the Invention of Rotoscoping

Koko the Clown is a character from Max Fleischer's short animation Out of the Inkwell. When it was first released in 1914, people were amazed by how fluid and lifelike the way the clown moved and danced.

That's when the technique rotoscoping was first used in animation. For reference, Max Fleischer filmed his brother Dave Fleischer, who dressed in the clown's costume and walked and jumped in a way a clown did. Then Max Fleischer hand-drawn and transcribed the film frame by frame, thus turning it into animation. The rotoscoping technique was patented in 1915.

Following the huge success of Koko the Clown, Max created a series of Out of the Inkwell and later established his own animation company, Fleischer Studios. Some much-loved characters such as Betty Boop and Popeye are the brainchildren of this studio.

Another notable example of rotoscoping is Minnie the Moocher from the Betty Boop series in the 1930s. The cartoon took its name from Cab Calloway's popular song "Minnie the Moocher". In the toon, the idiosyncratic dance moves of Cab Calloway was rotoscroped into the performance of a walrus ghost, even the singer himself was amused when he saw the creature dancing in his style.

Max Fleischer animated Gulliver's travels in 1939 and it was the last time Max used the rotoscoping technique for Gulliver, Glory and David.

2.2 Rotoscoping in Walt Disney's Works

Although rotoscoping is the invention of Max Fleischer, the technique was put into wide usage in Disney's studio and became even more influential and inspiring.

Before the 1930s, the rotoscoping technique was used exclusively by Max Fleischer's studio due to the patent. After the expiration of the patent in 1934, other studios started to adopt the technique in their creations.

Notable examples are Snow White and the Seven Dwarfs, Pinocchio and Alice in Wonderland from the Disney Studio. Instead of tracing over the footage frame by frame, the animation artists would use the footage as a reference for the hand drawing. In effect, the artists will not trace every movement onto the animation paper, but take notice of the general spirit of the movements. The idea is, by observing the footage, the cartoon artists can get some idea of the movements that are unconsciously suggested by the performer.

The dances we saw in Snow White are rotoscoped from the dancer, Marjorie Belcher. When Disney's Studio was working on Alice in Wonderland, besides body movement, the face and mouth movements of an actor are also rotoscoped. Thanks to rotoscoping, the animators no longer need to imagine how the arms and legs are moved.

The trace of the action soon got adopted elsewhere and speed up the production with more life-like movements.

2.3 Adaptation of Rotoscoping in Visual Effects

From the 1940s and onwards, Rotoscoping was adopted not only as a technique to trace action for animation, but also as way to create otherworldly backgrounds, scenes, or remove objects for special visual effects in live-action movies.

Ubbe Ert Iwwerks (known as Ub Iwerks) plays an important part in the improvement of rotoscoping. Ub is a good friend of Walt Disney's and he joined the studio in the 1940s. The two created Mickey Mouse and many memorable characters.

Ub uses a type of rotoscoping that creates matte to separate the actor from the background, so that the team can replace the background and add animated characters that are otherwise impossible in the real world.

Mary Poppins (1964) is a notable example of the visual effects created by this kind of rotoscoping. It uses the sodium vapor screens to create a real-time matte for the tricks to come true. You can read more about the sodium process in the history of chroma keying.

Ub Iwerks and his team also collaborated with Alfred Hitchcock for the special effects in The Birds (1963). In the most unforgettable scene of the birds attacking, they used rotoscoping to create the scene with hundreds of birds flying and attacking the town.

The 1970s witnessed the creative power of rotoscoping in visual effects. It is adapted to create exciting elements and scenes in the action film. Enter the lightsaber in the Star War trilogy. With rotoscoping, the VFX technicians were able to add a colored glow to the sword, while in reality, the actors were merely fighting each other with sticks. The technique was adapted to remove unnecessary objects, composite shots for explosion and alien attacking.

These all happened in an age without green screen, blue screen and chroma keying technique aided by modern-day computers. It is until the early 1990s did rotoscoping switch over to digital, thanks to animator and computer scientist, Bob Sabiston.

2.4 Interpolated Rotoscoping

Previously, animators need to hand-draw each frame, which is a laborious and time-consuming process. Bob Sabiston, an animator and computer scientist from MIT Media Lab, invented a process called interpolated rotoscoping, and developed it into a computer-based application Rotoshop in1997, with the naming being inspired by Photoshop.

Rotoshop eases the work for animators to a large extent. The artist only needs to draw keyframes for the start and end of the action, and the program will automatically generate the in-betweens. The interpolated shapes and lines drawn by the computer are smooth and fluid. Rotoshop is a proprietary software owned by Bob Sabiston's company Flat Black Films.

Interpolated rotoscoping was initially used in Sabiston's award-winning short movie Snack and Drink, and the technique received much more attention in the making of the feature movies Waking Life (2001) and A Scanner Darkly (2004) directed by Richard Linklater. The two experimental films were shot digitally and rotoscoped into animation with a distinctive look. The unearthly style created by the rotoscoping animation matched the psychological sci-fi story perfectly.

Sabiston modified Rotoshop substantially during the making of A scanner Darkly. Into the new era, there are many visual effects software that offers tools and features for rotoscoping, both in terms of transcribing and removing objects and replacing backgrounds. The VFX animators, or the "roto artists" in every VFX team are in charge of chroma keying, motion tracking, effects painting, object and rigs removing, and more.

Although the practice of transcribing frame by frame dates back decades, the basic principles are still in use in modern days. For instance, in the Guardians of the Galaxy, Rocket is rotoscoped from a real raccoon by the VFX animators.

3. Rotoscoping Technique

For people familiar with the history of rotoscoping, it is obvious that rotoscoping is a cover term for several techniques that are distinctive from each other, or, involved as per the need of the production.

In its earliest application, rotoscoping refers to the technique of transcribing live action footage onto animations papers, in an aim to capture lifelike movements. Then the technique is used by Disney as a reference for the animation artists. In the digital age, drawing from live-action footage is conducted on computer-aided software, for cartoons as well as feature films.

In terms of visual effects, rotoscoping is used alongside other techniques to create mattes, to separate characters from the background, add virtual elements, or removed unwanted objects. In pre-digital times, it is done with clever tricks, such as using traveling mattes or sodium lamp yellow screen. Later in the digital era, it is much easier with computing algorithms.

3.1 Rotoscoping as a Tracing Process

When Max Fleischer worked on rotoscoping, he repurposed a rostrum camera to be a projector, which can project the image of the film frame by frame onto the back of a glass easel. Then Max would trace the image and draw it onto the animation paper, one after another.

That drawing light table is referred to as a rotoscope, and the resulting picture is a rotograph. Although modern-day roto takes place on computers, the process and the technique are still referred to as rotoscoping.

For DIY lovers to reproduce the good old transcribing, there are tabletop drawing lightboxes with LED, or a simple windows and tape will do. One can place the still photos underneath a paper, and tape everything onto the window. With the daylight, the image can be see-through. Then use a pen to trace the details of the image.

3.2 Rotoscoping as Manual Matte Painting

In pre-digital films, full manual matte painting is involved to rotoscope the characters and their environment. In Alfred Hitchcock's The Birds (1963), the animation technician created special mattes to separate the actors from the birds. Two artists spent three months rotoscoping 500 frames just for that one shot.

Another example is the Return of the Jedi (1983). The roto artists created a holdout matte by hand, and rotoscoped the trees in the forest. Then the AT-Ats walking vehicles appear to walk between the woods and trees, as if the virtual elements were really there.

Although it still involves hand painting, the aim here is for compositing, instead of recreating lifelike movements like Snow White dancing and Koko the Clown walking.

3.3 Computer-Aided Rotoscoping

As mentioned before, with the introduction of Rotoshop into the rotoscoping process, Sabiston has upgraded the meaning of rotoscoping in the digital age. With the interpolated rotoscoping, the computer can smooth out the motion with intermediate frames, and relieve the burden of animation artists, who now only need to create the keyframes. With modern-day VFX software, you can roto with the aim to trace the movement or roto to keyout the character or remove unwanted elements.

When it comes to compositing, the chroma keying technique is widely acknowledged, and almost every video editing hobbyists know more or less about the green screen. However, there are times when one cannot simply pull the key to remove the background and that's when modern day rotoscoping comes into aid.

Rotoscoping in VFX software mainly works for these purposes:

1. To create mattes to separate characters, foregrounds and backgrounds;
2. To remove unwanted objects, wires and rigs;
3. To use as a great aid to strengthen the keying.

Using a matte helps to composite fantasy scenes and elements that one can dream of, or simply separate the subjects from their environment to add interesting effects. For instance, one can place clouds between a traveler and the landscape. With VFX software, creators can use masking tools, rotobrush, or dedicated tools such as Mocha and Silhouette for the roto.

These tools can also be used to remove objects as modern-day rotoscoping is not all about creating mattes. In the composition, creators can remove rigs, pylons, or boom mics that accidentally stray into the shots.

Rotoscoping can also be used in juxtaposition with chroma keying. Sometimes, the green screen is almost perfect, except for some parts. In that case, the artists can rotoscope that part specifically to save the keying, instead of discarding everything.

4. Rotoscoping Software Modern Day

Rotoscoping is one of the key features of visual effects software. Other fundamentals include keying, tracking, compositing, rigging, and 3D stuffs such as texturing, lighting, particle effects, modeling and so on. For anyone trying to embark on rotoscoping, here are industry-standard tools to begin with.

4.1 After Effects

Adobe After Effects is an industry-standard rotoscoping software, with a dazzling array of pro features for visual effects and animation. It is widely used in post-production for feature films, animations, and motion graphics. For rotoscoping process, one can utilize its powerful rotobrush for quick selection, or go more precisely with masking tools.

After Effects also allows users to add shadows to make the roto more realistic, with powerful tools to imitate the direction of the light and render out the corresponding shadows.

4.2 Fusion

Fusion has been around for more than three decades and is one of the most trustworthy VFX software for rotoscoping. Developed by Blackmagic Design, the same company that created the much-loved color grading tool DaVinci Resolve, Fusion offers robust tools for 3D compositing, fast motion tracking and sophisticated keying.

The vector paint and rotoscoping tools in Fusion allow users to create masks and mattes to control a specific part of the object in the clips. Its deep pixel compositing feature can blend 2D images with CGI lights and shadows, all in an interactive manner.

4.3 Blender

Blender is fantastic open-source software, making it the best free rotoscoping software on this list. When it comes to 3D animation and rotoscoping, blender wins the hearts of beginners and designers. This tool allows users to create stylish anime, sculpting and special effects for rotoscoping.

It is also widely used to print 3D models, video games, and interactive applications.

5. How to Rotoscope in VFX Software

Since the underlying principles are the same for each VFX software in terms of rotoscoping, we use After Effects as an example for the three popular methods involved in rotoscoping: masks, rotobrush, and Mocha

Method 1: Masks for Rotoscoping

Masks involve a meticulous frame-by-frame approach, though you can use the tracking feature to create in-betweens of masks to save time. To rotoscope the action, the VFX artists need to draw the mask around the targeted object and make adjustments frame by frame.

A pro tip to save time is to draw masks on key actions, and let the software generate masks between the actions. The key actions here refer to the start and end point of an action, for instance, when the character starts to rise the arm and when the arm is in position.

Also, it is good practice to separate the masks into several groups with colors, instead of creating a large mark for the entire object.

Method 2. Rotobrush

Using rotobrush is a faster roto solution if you value speed over the degree of control. The rotobrush in AE uses artificial intelligence to detect the edges of the target object. One can quickly scrub through the object using the rotobrush, and the software will automatically select intended areas.

There are additional brushes to fine tune the selection, such as adding, subtracting and refining.

Method 3. Mocha

Mocha can be used for planar tracking and replacement, making it an ideal plug-in for rotoscoping. It allows users to change the number of points in the selection, and won't affect the shape dramatically. You can watch the advanced rotoscoping with mocha tutorial below.

6. Takeaways: Notable Rotoscoped Films & Animations

Below are animations and films that use rotoscoping technique in the sense of rotoscoping animation, which involves tracing over the live-action film to achieve vivid effects, or drawing out imaginary elements.

The following list of films use rotoscoping to create stunning visual effects with mattes and VFX tricks.