D2
Computer Vision and Machine Learning

Jan Eric Lenssen (Postdoc)

Dr. Jan Eric Lenssen

Address
Max-Planck-Institut für Informatik
Saarland Informatics Campus
Campus E1 4
66123 Saarbrücken
Location
E1 4 - 618
Phone
+49 681 9325 2141
Fax
+49 681 9325 2099

Personal Information

My interest lies in designing differentiable algorithms, architectures and representations for percieving and representing the 3D world. Past research includes works in the areas of graph neural networks, equivariant operators, implicit neural fields and differentiable eigendecomposition. Currently, I am interested in sparse correspondence representations between neural fields.

If you want to collaborate in these areas or looking to write a Master's Thesis, reach out to me.

For more information and publication list, please visit my personal page​​​​​​​​​​​​​​.

Publications

Zhou, K., Lal Bhatnagar, B., Lenssen, J. E., & Pons-Moll, G. (2022). TOCH: Spatio-Temporal Object Correspondence to Hand for Motion Refinement. Retrieved from https://arxiv.org/abs/2205.07982
(arXiv: 2205.07982)
Abstract
We present TOCH, a method for refining incorrect 3D hand-object interaction sequences using a data prior. Existing hand trackers, especially those that rely on very few cameras, often produce visually unrealistic results with hand-object intersection or missing contacts. Although correcting such errors requires reasoning about temporal aspects of interaction, most previous work focus on static grasps and contacts. The core of our method are TOCH fields, a novel spatio-temporal representation for modeling correspondences between hands and objects during interaction. The key component is a point-wise object-centric representation which encodes the hand position relative to the object. Leveraging this novel representation, we learn a latent manifold of plausible TOCH fields with a temporal denoising auto-encoder. Experiments demonstrate that TOCH outperforms state-of-the-art (SOTA) 3D hand-object interaction models, which are limited to static grasps and contacts. More importantly, our method produces smooth interactions even before and after contact. Using a single trained TOCH model, we quantitatively and qualitatively demonstrate its usefulness for 1) correcting erroneous reconstruction results from off-the-shelf RGB/RGB-D hand-object reconstruction methods, 2) de-noising, and 3) grasp transfer across objects. We will release our code and trained model on our project page at http://virtualhumans.mpi-inf.mpg.de/toch/
Export
BibTeX
@online{Zhou_2205.07982, TITLE = {{TOCH}: Spatio-Temporal Object Correspondence to Hand for Motion Refinement}, AUTHOR = {Zhou, Keyang and Lal Bhatnagar, Bharat and Lenssen, Jan Eric and Pons-Moll, Gerard}, LANGUAGE = {eng}, URL = {https://arxiv.org/abs/2205.07982}, EPRINT = {2205.07982}, EPRINTTYPE = {arXiv}, YEAR = {2022}, ABSTRACT = {We present TOCH, a method for refining incorrect 3D hand-object interaction sequences using a data prior. Existing hand trackers, especially those that rely on very few cameras, often produce visually unrealistic results with hand-object intersection or missing contacts. Although correcting such errors requires reasoning about temporal aspects of interaction, most previous work focus on static grasps and contacts. The core of our method are TOCH fields, a novel spatio-temporal representation for modeling correspondences between hands and objects during interaction. The key component is a point-wise object-centric representation which encodes the hand position relative to the object. Leveraging this novel representation, we learn a latent manifold of plausible TOCH fields with a temporal denoising auto-encoder. Experiments demonstrate that TOCH outperforms state-of-the-art (SOTA) 3D hand-object interaction models, which are limited to static grasps and contacts. More importantly, our method produces smooth interactions even before and after contact. Using a single trained TOCH model, we quantitatively and qualitatively demonstrate its usefulness for 1) correcting erroneous reconstruction results from off-the-shelf RGB/RGB-D hand-object reconstruction methods, 2) de-noising, and 3) grasp transfer across objects. We will release our code and trained model on our project page at http://virtualhumans.mpi-inf.mpg.de/toch/}, }
Endnote
%0 Report %A Zhou, Keyang %A Lal Bhatnagar, Bharat %A Lenssen, Jan Eric %A Pons-Moll, Gerard %+ Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society %T TOCH: Spatio-Temporal Object Correspondence to Hand for Motion Refinement : %G eng %U http://hdl.handle.net/21.11116/0000-000A-ACF3-2 %U https://arxiv.org/abs/2205.07982 %D 2022 %X We present TOCH, a method for refining incorrect 3D hand-object interaction sequences using a data prior. Existing hand trackers, especially those that rely on very few cameras, often produce visually unrealistic results with hand-object intersection or missing contacts. Although correcting such errors requires reasoning about temporal aspects of interaction, most previous work focus on static grasps and contacts. The core of our method are TOCH fields, a novel spatio-temporal representation for modeling correspondences between hands and objects during interaction. The key component is a point-wise object-centric representation which encodes the hand position relative to the object. Leveraging this novel representation, we learn a latent manifold of plausible TOCH fields with a temporal denoising auto-encoder. Experiments demonstrate that TOCH outperforms state-of-the-art (SOTA) 3D hand-object interaction models, which are limited to static grasps and contacts. More importantly, our method produces smooth interactions even before and after contact. Using a single trained TOCH model, we quantitatively and qualitatively demonstrate its usefulness for 1) correcting erroneous reconstruction results from off-the-shelf RGB/RGB-D hand-object reconstruction methods, 2) de-noising, and 3) grasp transfer across objects. We will release our code and trained model on our project page at http://virtualhumans.mpi-inf.mpg.de/toch/ %K Computer Science, Computer Vision and Pattern Recognition, cs.CV
Tiwari, G., Antic, D., Lenssen, J. E., Sarafianos, N., Tung, T., & Pons-Moll, G. (n.d.). Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields. In European Conference on Computer Vision (ECCV 2022). Tel Aviv, Israel: Springer.
(Accepted/in press)
Export
BibTeX
@inproceedings{Tiwari_ECCV22, TITLE = {Pose-{NDF}: {M}odeling Human Pose Manifolds with Neural Distance Fields}, AUTHOR = {Tiwari, Garvita and Antic, Dimitrije and Lenssen, Jan Eric and Sarafianos, Nikolaos and Tung, Tony and Pons-Moll, Gerard}, LANGUAGE = {eng}, PUBLISHER = {Springer}, YEAR = {2022}, PUBLREMARK = {Accepted}, BOOKTITLE = {European Conference on Computer Vision (ECCV 2022)}, SERIES = {Lecture Notes in Computer Science}, ADDRESS = {Tel Aviv, Israel}, }
Endnote
%0 Conference Proceedings %A Tiwari, Garvita %A Antic, Dimitrije %A Lenssen, Jan Eric %A Sarafianos, Nikolaos %A Tung, Tony %A Pons-Moll, Gerard %+ Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society External Organizations Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society External Organizations External Organizations Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society %T Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields : %G eng %U http://hdl.handle.net/21.11116/0000-000A-B582-6 %D 2022 %B European Conference on Computer Vision %Z date of event: 2022-10-23 - 2022-10-27 %C Tel Aviv, Israel %B European Conference on Computer Vision %I Springer %B Lecture Notes in Computer Science
Zhou, K., Bhatnagar, B. L., Lenssen, J. E., & Pons-Moll, G. (n.d.). TOCH: Spatio-Temporal Object Correspondence to Hand for Motion Refinement. In European Conference on Computer Vision (ECCV 2022). Tel Aviv, Israel: Springer.
(Accepted/in press)
Abstract
We present TOCH, a method for refining incorrect 3D hand-object interaction sequences using a data prior. Existing hand trackers, especially those that rely on very few cameras, often produce visually unrealistic results with hand-object intersection or missing contacts. Although correcting such errors requires reasoning about temporal aspects of interaction, most previous work focus on static grasps and contacts. The core of our method are TOCH fields, a novel spatio-temporal representation for modeling correspondences between hands and objects during interaction. The key component is a point-wise object-centric representation which encodes the hand position relative to the object. Leveraging this novel representation, we learn a latent manifold of plausible TOCH fields with a temporal denoising auto-encoder. Experiments demonstrate that TOCH outperforms state-of-the-art (SOTA) 3D hand-object interaction models, which are limited to static grasps and contacts. More importantly, our method produces smooth interactions even before and after contact. Using a single trained TOCH model, we quantitatively and qualitatively demonstrate its usefulness for 1) correcting erroneous reconstruction results from off-the-shelf RGB/RGB-D hand-object reconstruction methods, 2) de-noising, and 3) grasp transfer across objects. We will release our code and trained model on our project page at http://virtualhumans.mpi-inf.mpg.de/toch/
Export
BibTeX
@inproceedings{Zhou_ECCV2022, TITLE = {{TOCH}: {S}patio-Temporal Object Correspondence to Hand for Motion Refinement}, AUTHOR = {Zhou, Keyang and Bhatnagar, Bharat Lal and Lenssen, Jan Eric and Pons-Moll, Gerard}, LANGUAGE = {eng}, PUBLISHER = {Springer}, YEAR = {2022}, PUBLREMARK = {Accepted}, ABSTRACT = {We present TOCH, a method for refining incorrect 3D hand-object interaction sequences using a data prior. Existing hand trackers, especially those that rely on very few cameras, often produce visually unrealistic results with hand-object intersection or missing contacts. Although correcting such errors requires reasoning about temporal aspects of interaction, most previous work focus on static grasps and contacts. The core of our method are TOCH fields, a novel spatio-temporal representation for modeling correspondences between hands and objects during interaction. The key component is a point-wise object-centric representation which encodes the hand position relative to the object. Leveraging this novel representation, we learn a latent manifold of plausible TOCH fields with a temporal denoising auto-encoder. Experiments demonstrate that TOCH outperforms state-of-the-art (SOTA) 3D hand-object interaction models, which are limited to static grasps and contacts. More importantly, our method produces smooth interactions even before and after contact. Using a single trained TOCH model, we quantitatively and qualitatively demonstrate its usefulness for 1) correcting erroneous reconstruction results from off-the-shelf RGB/RGB-D hand-object reconstruction methods, 2) de-noising, and 3) grasp transfer across objects. We will release our code and trained model on our project page at http://virtualhumans.mpi-inf.mpg.de/toch/}, BOOKTITLE = {European Conference on Computer Vision (ECCV 2022)}, SERIES = {Lecture Notes in Computer Science}, ADDRESS = {Tel Aviv, Israel}, }
Endnote
%0 Conference Proceedings %A Zhou, Keyang %A Bhatnagar, Bharat Lal %A Lenssen, Jan Eric %A Pons-Moll, Gerard %+ Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society %T TOCH: Spatio-Temporal Object Correspondence to Hand for Motion Refinement : %G eng %U http://hdl.handle.net/21.11116/0000-000A-B586-2 %D 2022 %B European Conference on Computer Vision %Z date of event: 2022-10-23 - 2022-10-27 %C Tel Aviv, Israel %X We present TOCH, a method for refining incorrect 3D hand-object interaction sequences using a data prior. Existing hand trackers, especially those that rely on very few cameras, often produce visually unrealistic results with hand-object intersection or missing contacts. Although correcting such errors requires reasoning about temporal aspects of interaction, most previous work focus on static grasps and contacts. The core of our method are TOCH fields, a novel spatio-temporal representation for modeling correspondences between hands and objects during interaction. The key component is a point-wise object-centric representation which encodes the hand position relative to the object. Leveraging this novel representation, we learn a latent manifold of plausible TOCH fields with a temporal denoising auto-encoder. Experiments demonstrate that TOCH outperforms state-of-the-art (SOTA) 3D hand-object interaction models, which are limited to static grasps and contacts. More importantly, our method produces smooth interactions even before and after contact. Using a single trained TOCH model, we quantitatively and qualitatively demonstrate its usefulness for 1) correcting erroneous reconstruction results from off-the-shelf RGB/RGB-D hand-object reconstruction methods, 2) de-noising, and 3) grasp transfer across objects. We will release our code and trained model on our project page at http://virtualhumans.mpi-inf.mpg.de/toch/ %K Computer Science, Computer Vision and Pattern Recognition, cs.CV %B European Conference on Computer Vision %I Springer %B Lecture Notes in Computer Science