MARS: Motion-Augmented RGB Stream for Action Recognition

Code

Mars cartwheel gif

The test code and models are released under the MIT license.

Both the code and models are on github.

Publication and blog

The CVPR 2019 publication related to the codes and models is MARS: Motion-Augmented RGB Stream for Action Recognition [PDF]

Authors: Nieves Crasto1, Philippe Weinzaepfel1, Karteek Alahari2, Cordelia Schmid2 [1NAVER LABS Europe  2Inria]

Look at it differently – there’s also a quick read blog article on MARS

Information related to code and models

Most state-of-the-art methods for action recognition consist of a two-stream architecture with 3D convolutions: an appearance stream for RGB frames and a motion stream for  optical flow frames. Although combining flow with RGB improves the performance, the cost of computing accurate optical flow is high, and increases action recognition latency. This limits the usage of two-stream approaches in real-world applications requiring low latency. In this paper, we  introduce two learning approaches to train a standard 3D CNN, operating on RGB frames, that mimics the motion stream, and as a result avoids flow computation at test time. First, by minimizing a feature-based loss compared to the Flow stream, we show that the network reproduces the motion stream with high fidelity. Second, to leverage both appearance and motion information effectively, we train with a linear combination of the feature-based loss and the standard cross-entropy loss for action recognition. We denote the stream trained using this combined loss as Motion-Augmented  RGB Stream (MARS). As a single stream, MARS performs better than RGB or Flow alone, for instance with 72.7% accuracy on Kinetics compared to 72.0% and 65.6% with RGB and Flow streams respectively.

blank

Citation

@inproceedings{crasto2019mars,
 title={{MARS: Motion-Augmented RGB Stream for Action Recognition}},
 author={Crasto, Nieves and Weinzaepfel, Philippe and Alahari, Karteek and Schmid, Cordelia},
 booktitle={CVPR},
 year={2019}
}

This web site uses cookies for the site search, to display videos and for aggregate site analytics.

Learn more about these cookies in our privacy notice.

blank

Cookie settings

You may choose which kind of cookies you allow when visiting this website. Click on "Save cookie settings" to apply your choice.

FunctionalThis website uses functional cookies which are required for the search function to work and to apply for jobs and internships.

AnalyticalOur website uses analytical cookies to make it possible to analyse our website and optimize its usability.

Social mediaOur website places social media cookies to show YouTube and Vimeo videos. Cookies placed by these sites may track your personal data.

blank