Learning 3D Human Dynamics from Video
From an image of a person in action, we can easily guess the 3D motion of the person in the immediate past and future. This is because we have a mental model of 3D human dynamics that we have acquired from observing visual sequences of humans in motion. We present a framework that can similarly learn a representation of 3D dynamics of humans from video via a simple but effective temporal encoding of image features. At test time, from video, the learned temporal representation can recover smooth 3D mesh predictions. From a single image, our model can recover the current 3D mesh as well as its 3D past and future motion. Our approach is designed so it can learn from videos with 2D pose annotations in a semi-supervised manner. However, annotated data is always limited. On the other hand, there are millions of videos uploaded daily on the Internet. In this work, we harvest this Internet-scale source of unlabeled data by training our model on them with pseudo-ground truth 2D pose obtained from an off-the-shelf 2D pose detector. Our experiments show that adding more videos with pseudo-ground truth 2D pose monotonically improves 3D prediction performance. We evaluate our model on the recent challenging dataset of 3D Poses in the Wild and obtain state-of-the-art performance on the 3D prediction task without any fine-tuning. The project website with video can be found at https://akanazawa.github.io/human_dynamics/.
Authors

Are you an author of this paper? Check the Twitter handle we have for you is correct.

Angjoo Kanazawa (edit)
Jason Zhang (edit)
Panna Felsen (add twitter)
Jitendra Malik (add twitter)
Ask The Authors

Ask the authors of this paper a question or leave a comment.

Read it. Rate it.
#1. Which part of the paper did you read?

#2. The paper contains new data or analyses that is openly accessible?
#3. The conclusion is supported by the data and analyses?
#4. The conclusion is of scientific interest?
#5. The result is likely to lead to future research?

Github
User:
None (add)
Repo:
None (add)
Stargazers:
0
Forks:
0
Open Issues:
0
Network:
0
Subscribers:
0
Language:
None
Youtube
Link:
None (add)
Views:
0
Likes:
0
Dislikes:
0
Favorites:
0
Comments:
0
Other
Sample Sizes (N=):
Inserted:
Words Total:
Words Unique:
Source:
Abstract:
None
12/04/18 06:01PM
7,848
2,247
Tweets
arxiv_pop: 2018/12/04 投稿 3位 CV(Computer Vision and Pattern Recognition) Learning 3D Human Dynamics from Video https://t.co/fZb7ZnU3Yn 4 Tweets 35 Retweets 97 Favorites
TheKiranPrakash: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/biqkeSuCVu #computervision https://t.co/BxT47xe2W7
aneomatrix: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
CFrank42: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
JudeLaw_zhao: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
mnrmja007: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
irhumshafkat: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
chengoldberg: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
GNUcgraph: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
JaredHeinly: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
DCasBol: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
chiefaiofficers: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
montrealdotai: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
BarrySlaff: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
dksdc: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
IntuitMachine: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
Quebec_AI: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
Montreal_AI: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
PerthMLGroup: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
ceobillionaire: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
AssistedEvolve: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
Koundinya33: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
ComputerPapers: Learning 3D Human Dynamics from Video. https://t.co/RqouqK2zDP
jp_axs4ll: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
p_kot1: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
darkproger: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
cghosh_: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
sir_goe: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
sansuiso: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
q_tarou: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
robotic_hands: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
indy9000: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
ayirpelle: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
swaroopkpal: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
gabeibagon: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
CSProfKGD: RT @quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
quantombone: Learning 3D Human Dynamics from Video. https://t.co/vkKunHPQGl #computervision https://t.co/gXMFnGJNlX
BrundageBot: Learning 3D Human Dynamics from Video. Angjoo Kanazawa, Jason Zhang, Panna Felsen, and Jitendra Malik https://t.co/BbEpjJlYcW
Images
Related