For many industries that range from computer-generated images from Hollywood to product design, 3D modeling tools often use text or image requests to determine different aspects of visual appearance such as color and shape. As much as this makes sense as the first point of contact, these systems are still limited due to their neglect of something central to human experience.
The fundamental of the uniqueness of physical objects is their tactile properties such as roughness, bumpiness or the feeling of materials such as wood or stone. Existing modeling methods often require advanced computer -aided design competence and rarely support tactile feedback that can be of crucial importance for perception and interaction with the physical world.
In this sense, researchers of the MIT Laboratory for Computer Science and Artificial Intelligence (CSAIL) have created a new system for stylizing 3D models using image requests, whereby both the visual appearance and tactile properties are replicated.
With the “tact style” tool of the CSAIL team, the creators can style 3D models based on pictures and at the same time include the expected tactile properties of the textures. Tact style separates visual and geometric stylization and enables replication of both visual and tactical properties from a single image input.
Play video
With the tool with the tactstyle tool, the creators can stylize 3D models based on pictures and at the same time include the expected tactile properties of the textures.
The PhD student Faraz Faruqi, the leading author of a new paper about the project, says that tact style could have far-reaching applications that extend from residential culture and personal accessories to tactile learning tools. With tact style, users can download a basic design – e.g. B. a headphone stand of Thingive verses – and adapt it with the desired styles and textures. In education, learners can explore various textures from all over the world without leaving the classroom, while in product design the fast prototyping becomes easier because designers quickly print several iterations to refine tactile properties.
“You can imagine using this type of system for common objects such as telephone stands and Earbud cases to enable more complex textures and to improve tactile feedback in various ways,” says Faruqi, who has written the paper together with Associate Stefanie Mueller, head of human interaction (HCI) Engineering Group in the CSAIL group. “You can create tactile educational instruments to demonstrate a number of different concepts in areas such as biology, geometry and topography.”
In conventional methods for replication of textures, special tactile sensors such as GelsSsight am with, which physically touch an object in order to grasp its surface microgometry as a “Heightfield”. However, this requires a physical object or its recorded surface for replication. With tact style, users can replicate the surface microgometry by using generative AI to create a Heightfield directly from a picture of the texture.
In addition, it is difficult for platforms such as the 3D print repository Thingiversum to take and adapt individual designs. If a user lacks sufficient technical background, the change in a design manually is the risk of actually “breaking” it so that it can no longer be printed. All of these factors have caused Faruqi to create a tool that enables the adjustment of downloadable models on a high level, but also maintains functionality.
In experiments, Tactstyle showed significant improvements compared to conventional stylization methods by creating precise correlations between the visual image of a texture and its hemefeld. This enables the replication of tactical properties directly from an image. A psychophysical experiment showed that user clock -styles -generated textures perceive both the expected tactile properties from visual inputs and the tactile characteristics of the original texture as similar, which leads to a uniform tactile and visual experience.
Tact style uses an existing method using the name “Style2fab” to change the color channels of the model so that they correspond to the visual style of the input image. Users first indicate a picture of the desired texture, and then a finely coordinated variation car code is used to translate the input picture into a corresponding Heightfield. This Heightfield is then used to change the geometry of the model in order to create the tactile properties.
The color and geometry stylization modules work together and styles both the visual and the tactile properties of the 3D model from a single image input. According to Faruqi, the core innovation is in the geometry stylization module, which uses a finely divided diffusion model to generate height fields from texture images-ETWAS Earlier stylization frameworks do not replicate exactly.
According to Faruqi, the team wants to expand tact style to generate new 3D models with generative AI with embedded textures. This requires exactly the type of pipeline that is necessary to replicate both the shape and the function of the 3D models manufactured. They also plan to examine “visuo-haptic false adjustments” to create new experience with materials that oppose conventional expectations, such as something that seems to be made of marble, but has made it out of wood.
Faruqi and Mueller visited the new paper together with the doctoral students Maxine Perroni-Scharf and Yunyi Zhu and the student of the students, Jaskaran Singh Walia, visited Masters Student Shuyue Feng and assistant professor Donald Degraen from the Human Interface Technology (HIT) Laboratory NZ in New Zealand.