HapticDrone: An Encountered-Type Kinesthetic Haptic Interface with Controllable Force Feedback

Researcher: Muhammad Abdullah, Minji Kim, Seokhee Jeon


HapticDrone is our new approach to transform a drone into a force-reflecting haptic interface. While our earlier work proved the concept, this paper concretizes it by implementing accurate force control along with effective encountered-type stiffness and weight rendering for 1D interaction as a first step. To this end, generated force is identified with respect to drone’s thrust command, allowing a precise control of force. Force control is combined with tracking of drone and hand, turning the HapticDrone into an ideal end-effector for the encountered-type haptics and making the system ready for more sophisticated physics simulation.

Stiffness and weight of a virtual object is our first target of the simulation. Conventional stiffness and gravity simulation is merged into the encountered-type position control scheme, allowing a user to feel the softness and weight of an object. We further confirmed the feasibility by proving that the system fulfills the physical performance requirements commonly needed for an encountered-type haptic interface, i.e., force rendering bandwidth, accuracy, and tracking performance.

Haptic-Enabled Digital Rectal Examination Simulator Using Pneumatic Bladder

Researcher: Ashiwari Talhan, Seokhee Jeon

Data-Driven Modeling of Haptic Properties

Researchers: Arsen Abdulali, Sunghoon Yim, Seokhee Jeon
Pure data-driven haptic modeling and rendering is one of the emerging techniques in the field of haptics. This approach records necessary signals generated during manual or automated palpation of a target object, e.g., high frequency vibrations through active stroking of a target surface for haptic texture, and uses them in rendering
for approximating the responses of the target surface under given user’s interaction. It
can effectively handle diverse and complex behaviors with less computational load
without knowledge about the object and system. Consequently, it is one of the most
relevant approaches for applications requiring high haptic realism.

In our research, we targeted three haptic properties for the data-driven modeling; stiffness, friction, and haptic texture. The followings are our research outcomes for the three properties.

Data-driven modeling/rendering of stiffness and friction

  • Sunghoon Yim, Seokhee Jeon, and Seungmoon Choi, “Data-Driven Haptic Modeling and Rendering of Viscoelastic and Frictional Responses of Deformable Objects,” IEEE Transaction on Haptics, Accepted.
  • Sunghoon Yim, Seokhee Jeon, and Seungmoon Choi, “Data-Driven Haptic Modeling and Rendering of Deformable Objects Including Sliding Friction,” In Proceedings of the IEEE World Haptics Conference (WHC), pp. 305-312, 2015.

Data-driven modeling/rendering of haptic texture

  • Arsen Abdulali and Seokhee Jeon, “Data-Driven Modeling of Anisotropic Haptic Textures: Data Segmentation and Interpolation,” In Proceedings of EuroHaptics, pp. 228-239, 2016.


Data-driven modeling/rendering of object dynamics

Data-driven haptic modeling is an emerging technique where contact dynamics are simulated and interpolated based on a generic input-output matching model identified by data sensed from interaction with target physical objects. in this research we present a new algorithm for the sample selection where the variances of output are observed for selecting representative input-output samples in order to ensure the quality of output prediction. The main idea is that representative pairs of input-output are chosen so that the ratio of the standard deviation to the mean of the corresponding output group does not exceed an application-dependent threshold.


  • Arsen Abdulali, Waseem Hassan, and Seokhee Jeon, “Stimuli-Magnitude-Adaptive Sample Selection for Data-Driven Haptic Modeling,” MDPI Entropy, vol. 18, no. 222, 2016.

Image Texture/Perception-Based Haptic Model Assignment to 3D Mesh (2012.09-current)

Researchers: Waseem Hassan and Seokhee Jeon 

As the lack of haptic contents is one of the limitations of current haptics technology, this research aims at developing a hardware and software framework for efficient haptic model building based on actual measurement data. In this work, we develop and integrate data-driven haptic contents modeling methods for stiffness, friction, and surface texture, and construct “The Haptic Library.” Then, we seek for new perception and image-based techniques to automatically assign the acquired models into 3D mesh models. This research is a part of Global Frontier project funded by NRF Korea.


  • Waseem Hassan and Seokhee Jeon, “Evaluating Differences Between Bare-handed and Tool-Based Interaction in Perceptual Space”, 2016 IEEE Haptics Symposium (HAPTICS). IEEE, 2016.

Property-Changing Haptic Handle for Encountered-Type Haptic Device (2012.09-current)

Researchers: Noman Akbar, Hongchae Lee, and Seokhee Jeon

Previous approaches on encountered-type haptic interfaces suffer from limited generality. Customized and pre-shaped end effectors can only represent a specific object, limiting the flexibility of the system. Manually or automatically replacing the end effector can be one option, but it is still not feasible when surface properties should be frequently changed. An ideal solution is to introduce a “universal end effector” where its haptic properties, such as shape and stiffness, can be systematically altered to effectively cope with virtual objects with different haptic property.


Research outcomes

  • Noman Akbar and Seokhee Jeon, “Encountered-Type Haptic Interface for Grasping Interaction with Round Variable Sized Objects via Pneumatic Balloon,” in Proceedings of EuroHaptics, 2014 (to be presented).


  • Hongchae Lee, Noman Akbar, and Seokhee Jeon, “Haptic Rendering of Curved Surface by Bending an Encountered-Type Flexible Plate,” in Proceedings of Korea Computer Congress 2014.
  • Seokhee Jeon, “Haptic Rendering of Curved Surface by Bending an Encountered-Type Flexible Plate,” IEICE Information and Systems, Accepted.


Haptic Augmented Reality (2009-current)

Researchers: Sunghoon Yim, Matthias Harders, Seungmoon Choi, and Seokhee Jeon

As augmented reality (AR) enables a real space to be transformed to a semi-virtual space by providing a user with the mixed sensations of real and virtual objects, haptic AR does the same for the sense of touch; a user can touch a real object, a virtual object, or a real object augmented with virtual touch. Visual AR has relatively mature technology and is being applied to diverse practical applications such as surgical training, industrial manufacturing, and entertainment. In contrast, the technology for haptic AR is quite young and poses a great number of new research problems ranging from modeling to rendering in terms of both hardware and software. We have been examining the potential of this haptic AR technology. Our long-term research aims at developing a systematic methodology for modulating the haptic properties of a real object with the aid of a haptic interface, i.e., a “haptic AR toolkit.” Please see below for our major research outcomes.

Research outcomes

    1. Sunghoon Yim, Seokhee Jeon, and Seungmoon Choi, “Normal and Tangential Force Decomposition and Augmentation Based on Contact Centroid,” AsiaHaptics, 2014 (Honorable mention – Final candidate for the best demo award ).
  • Seokhee Jeon, Seungmoon Choi, and Matthias Harders, “Haptic Augmentation in Soft Tissue Interaction,” in Multisensory Softness, edited by M. Di Luca, Springer (in print)

    This book chapter summarizes our major research results with more focus on the stiffness modulation.

  • Seokhee Jeon and Matthias Harders, “Haptic Tumor Augmentation: Exploring Multi-Point Interaction,” IEEE Transactions on Haptics, Accepted, 2014.

    Our second attempt to apply the stiffness modulation algorithms to a breast tumor palpation scenario. This time, we focused on the palpation using two point contacts. Special attentions was on the realistic recreation of mutual effect between the points.

  • Seokhee Jeon, “Haptically Assisting Breast Tumor Detection by Augmenting Abnormal Lump,” IEICE Transactions on Information & Systems, vol. E97-D, no. 2, pp. 361-365, 2014.

    This paper reports the use of haptic augmented reality in breast tumor palpation. In general, lumps in the breast are stiffer than surrounding tissues, allowing us to haptically detect them through self-palpation. The goal of the study is to assist self-palpation of lumps by haptically augmenting stiffness around lumps. The key steps are to estimate non-linear stiffness of normal tissues in the offline preprocessing step, detect areas that show abnormally stiffer responses, and amplify the difference in stiffness through a haptic augmented reality interface. The performance of the system was evaluated in a user-study, demonstrating the potential of the system.

  • Seokhee Jeon and Matthias Harders, “Extending Haptic Augmented Reality: Modulating Stiffness during Two-Point Squeezing,” In Proceedings of the IEEE Haptics Symposium (HS), pp. 141-146, 2012 (Oral presentation; acceptance rate = 26%).

    In this paper we generalize the approach by enabling a user to grasp, lift, and manipulate an object via two interaction points. Modulated stiffness can be explored by squeezing an object. To this end, two haptic interfaces equipped with force sensors are employed to render the additional virtual forces of the augmentation at the two interaction points. We introduce the required extended algorithms and evaluate the performance in a pilot user study.

  • Seokhee Jeon, Jean-Claude Metzger, Seungmoon Choi, and Matthias Harders, “Extensions to Haptic Augmented Reality: Modulating Friction and Weight,” In Proceedings of the World Haptics Conference (WHC), pp. 227-232, 2011 (Oral presentation; acceptance rate = 16.6%).

    In this paper, we extend our framework to cover further haptic properties: friction and weight. Simple but effective algorithms for estimating and altering these properties have been developed. The first approach allows us to change the inherent friction between a tool tip and a surface to a desired one identified in an offline process. The second technique enables a user to perceive an altered weight when lifting an object at two interaction points.

  • Seokhee Jeon and Seungmoon Choi, “Real Stiffness Augmentation for Haptic Augmented Reality,” Presence: Teleoperators and Virtual Environments, vol. 20, no. 4, pp. 337-370, 2011.

    This paper is the extended version of our previous Haptic Symposium paper. In Haptics Symposium 2010, we presented earlier results that included basic rendering algorithms and physical performance evaluation for the ball bearing tool. The present paper is significantly extended in various aspects, especially in the inclusion of a more general solid tool with associated algorithms and physical performance evaluation, the perceptual performance assessment of the whole haptic AR system, and in-depth discussions about further research issues.

  • Seokhee Jeon and Seungmoon Choi, “Stiffness Modulation for Haptic Augmented Reality: Extension to 3D Interaction,” in Proceedings of the IEEE Haptics Symposium (HS), pp. 273-280. 2010 (Recipient of Best Demo Award).

    Our previous work assumed an 1D interaction of tapping for stiffness perception as an initial study. In this paper, we extend the system so that a user can interact with a real object in any 3D exploratory pattern while perceiving its augmented stiffness. A series of algorithms are developed for contact detection, deformation estimation, force rendering, and force control. Their performances are thoroughly evaluated with real samples. A particular focus has been on minimizing the amount of preprocessing such as geometry modeling. 

  • Seokhee Jeon and Seungmoon Choi, “Haptic Augmented Reality: Taxonomy and an Example of Stiffness Modulation,” Presence: Teleoperators and Virtual Environments, vol. 18, no. 5, pp. 387-408, 2009.

    This paper is an extended version of our previous EuroHaptics paper in the following aspects: (1) An extended taxonomy encompassing both visual and haptic AR with a thorough survey of previous studies and associated classification, (2) Evaluation of the force control algorithm using another haptic interface, (3) Perceptual assessment of our haptic AR system, and (4) Presentation of many open research issues in haptic AR. 

  • Seokhee Jeon and Seungmoon Choi, “Modulating Real Object Stiffness for Haptic Augmented Reality,” Lecture Notes on Computer Science (EuroHaptics 2008), vol. 5024, pp. 609-608, 2008 (Acceptance rate = 36%).

    As an initial study, this paper investigates the feasibility of haptically modulating the feel of a real object with the aid of virtual force feedback, with the stiffness as a goal haptic property. All required algorithms for contact detection, stiffness modulation, and force control are developed, and their physical performances are evaluated. 

  • US patent pending, “Apparatus and Method for Providing Haptic Augmented Reality,” US 12/394,032, 2009.02.26.



Previous Researches

BEAMING Project (2010-2012)

Overall goal of this project is to build a framework to make a digital copy of real objects and to “beam” it to a remote site, allowing a collaborator to interact with it. As a participating researcher, constructing and beaming the haptic sensation was the focus. This project was a part of EU Project.





Inconsistencies in Augmented Reality Systems (2006-2008)

One of our current research thrusts is the evaluation of various aspects of VR/AR/MR interfaces. For instance, there are many cases where the interfaces and the corresponding interaction objects do not match in certain aspects such as interaction timing, view/interaction direction, multimodal feedback, etc. This may be due to lack of technology, inappropriate metaphor, missing modality, and so on. In addition to evaluations, we seek computational solutions to improve the these situations. For instance, in the picture, we assess the usability with regards to various placements of cameras (which capture the real environment in a desktop AR setting), and also look for ways to apply image warping or mosaicing techniques to allow the placement of cameras at user-convenient locations.

mismatch inc

Research outcomes

  • Seokhee Jeon, Hyeongseop Shim,Gerard J. Kim, “Viewpoint Usability for Desktop Augmented Reality,” International Journal of Virtual Reality , Vol.5. No.3, pp.33-39 2006.
  • Seokhee Jeon, Gerard J. Kim “Mosaicing a Wide Geometric Field of View for Effective Interaction in Augmented Reality” Proc. of IEEE ISMAR, 2007.  
  • Seokhee Jeon, Gerard J. Kim “Providing a Wide Field of View for Effective Interaction in Desktop Tangible Augmented Reality” Proc. of IEEE VR, 2008. 



Interactions with Large Ubiquitous Displays Using Camera Equipped Mobile Phones (2005-2010)

In the ubiquitous computing environment, people will interact with everyday objects (or computers embedded in them) in ways different from the usual and familiar desktop user interface. One such typical situation is interacting with applications through large displays such as televisions, mirror displays, and public kiosks. With these applications, the use of the usual keyboard and mouse input is not usually viable (for practical reasons). In this setting, the mobile phone has emerged as an excellent device for novel interaction. This research topic introduces user interaction techniques using a camera-equipped hand-held device such as a mobile phone or a PDA for large shared displays. In particular, we consider two specific but typical situations (1) sharing the display from a distance and (2) interacting with a touch screen display at a close distance. Using two basic computer vision techniques, motion flow and marker recognition, we show how a camera-equipped hand-held device can effectively be used to replace a mouse and share, select, and manipulate 2D and 3D objects, and navigate within the environment presented through the large display.

tabletopphone4 tabletopphone5 tabletopphone tabletopphone2 tabletopphone3

Research outcomes

  • Seokhee Jeon, Jane Hwang, Gerard J. Kim, and Mark Billinghurst, “Interaction with Large Ubiquitous Displays Using Camera-Equipped Mobile Phones,” Personal and Ubiquitous Computing, vol. 12, no. 2, pp. 83-94, 2010.
  • Seokhee Jeon, Gerard J. Kim, and Mark Billinghurst, “Interacting with a Tabletop Display Using a Camera Equipped Mobile Phone,” Lecture Notes on Computer Science (HCI International 2007), vol. 4551, pp. 336-343, 2007.
  • Seokhee Jeon, Jane Hwang, Gerard J. Kim, and Mark Billinghurst, “Interaction Techniques in Large Display Environments using Hand-held Devices,” In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pp. 100-103, 2006. 



Cart for Speed (2005-2006)


Research outcomes

  • Yongjin Kim, Jaehoon Jung, Seokhee Jeon, Sangyoon Lee, and Gerard J. Kim, “Telepresnce Racing Game”, SIGCHI ACE 2005.



Run in Place Interface in CAVE System (2003-2004)



MGET (Molecular Geometry Education Tool) in CAVE (2003)