The Unification of Virtual Try-On and Metaverse Avatar Customizations
This is our second article on the topic of NFTs and the metaverse written by Ben Houston - CTO and founder of Threekit. You can find the previous article focusing on NFT's here. The focus of this article is primarily around virtual try-on and what it is, customization within the Metaverse, and where we could potentially see it headed in the future. Please note that this is a thought leadership piece, and the content in this article is not representative of future Threekit features or roadmap.
Written by Ben Houston, CTO/Founder of Threekit
The ecommerce trend of virtual try-on, where you can try on items virtually using the front facing camera of your smart phone, has been gaining adoption in recent years. It has become ever more valuable with the recent pandemic as well as online-first businesses. What companies do not generally yet realize is that by preparing themselves and their product lines for virtual try-on, they are also preparing themselves for the next emerging market - that of metaverse avatar customization. It isn’t yet clear how big the market will be for metaverse avatar customization, but it could be potentially huge and quickly developing if the technology visions of Meta (Facebook) and Apple come to pass.
What is AR Virtual Try-On?
Virtual try-on is when you can use the front facing camera of your smartphone pointed at you and then have virtual items such as glasses, shoes, hair styles or even clothing appear on you and track your body movements. This is a new and developing area of the ecommerce journey for many consumers, especially valuable in the era of Covid where in-person shopping has decreased in frequency.
The way that Virtual Try-On works is that items intended for virtual try-on have additional information added to them that specifies where they will be worn. This could be identifying that glasses are positioned relative to the bridge of the nose and extend to above the ears, or it could be saying that an engagement ring fits before the first and second knuckles on your wedding finger, that shoes are positioned relative to the toes and heel, and even that clothing drapes onto the shoulders and falls down otherwise.
Then to do the try-on session, neutral networks are used to processing the incoming camera feed. These neutral networks identify the components of your body, such as eyes, mouth, ears, as well as any visible joints. Anything that isn’t visible is inferred based on what is visible. Thus even if you are wearing a shirt, it is still possible to estimate the pose of your back and shoulders by looking at the outline of the shirt and combining it with information on where your head and arms are. This pose of your body is then combined with the information where the virtual object should be worn in order to position it. This virtual object is then positioned on a per-frame basis using this extract pose and superimposed on top of the camera feed, thus making it appear to be virtually worn by the user.
Advanced virtual try-ons methods even try to estimate the lighting of the scene from the camera feed and apply that to the virtual object. This aims to better blend in the virtual object into the camera feed to make it seem more realistic, rather than something that is artificially pasted on top. Virtual try-on has become a key component of the Snap smartphone app where they encourage users to try things on, take selfies, and also then purchase those items in real life. It seems to be just an accepted part of culture for those under 25 years old.
What is Metaverse Avatars Customization?
The metaverse is a concept where we will be interacting with others using a fully virtual self. In these scenarios - that virtual self is referred to as an avatar. A computer generated representation of ourselves as we want to be seen, not necessarily reflective of how we look in the real-life.
There are emerging standards for avatars centered around the ability to have a persistent visual representation of oneself across multiple metaverse-like games. Many games already have their own avatar creation tools, like Fortnite or even BeatSaber. Some platforms are in on this using Roblox’s avatar creator for their experiences as well as both Apple’s Memojis and Meta’s Oculus avatar. Although Meta’s Oculus avatar system doesn’t appear to be used by many games on the platform, I do not know of anyone but Apple’s built in tools that currently use Apple’s Memojis.
There is a few attempts at creating truly cross platform avatar systems. The most successful and prominent so far is ReadyPlayer.me. They are using similar technology under the hood to Apple’s Memojis but they are doing it focused on the web first and ensuring it is transferable. ReadyPlayer.me is also pushing hard on getting market adoption for their avatar system.
“We continue to believe the avatar and its associated “backpack” of digital assets, costumes, currency, etc. will be your most valuable asset in the Metaverse.” - Charlie Fink, Forbes, December 30, 2021 (article)
The ReadyPlayer.me avatar systems currently are mostly focused on the visual appearance of the person’s body, face and hair -- such as their height, build, skin color, hair color, hair style, eye color, eye style, lip style and some very rudimentary clothing options. The Roblox avatar system is much less realistic in terms of the body appearance than ReadyPlayer.me, but it makes up for it with a deep richness in its clothing options. There is a lot of branded content and creative options that Roblox players can purchase for virtual in-game currency to customize their avatars.
The future is likely going to be a combination of the high fidelity ReadyPlayer.me avatars combined with the rich and diverse clothing options of Roblox. It is also likely that purchases of these virtual clothing items will be done via NFTs such that they are both authentic as well as transferable across platforms.
The Same Thing Under the Hood
From a technical standpoint, it turns out that putting clothing on a virtual avatar is actually mostly the same data and technology as doing virtual try-on for an ecommerce experience.
In both cases, one needs to have a virtual 3D representation of the item being tried on or worn by the avatar. This virtual 3D representation must have data as to how it is to be worn on the human body. The 3D representation must be realized in a fashion such that it is easily transferable between different rendering engines, including Apple’s ARKit, Google’s Scene Viewer, as well as in Roblox, Fortnite, and all of the many emerging platforms.
This means that if you are a vendor of clothing, you can create one 3D digital representation for both virtual try-on as well as for avatar customization. And then you can use it for both applications. Threekit’s platform for virtual digital visual twins is designed to enable this type of workflow for those selling both physical and virtual goods, or just one or the other.
It is likely that NFTs will be used for both virtual try-on as well as avatar customizations. For avatar customizations, NFTs will be used to track ownership and authenticity. For virtual try-ons, it is likely that NFTs will be used as certificates of authenticity and to ensure that others do not just download the 3D data used for virtual try-on and use it as an NFT themselves illegitimately.
Potential Future Scenarios
Once you realize that these are the same thing, there is a lot of crossover you can exploit between these two technologies.
Virtual Demand Testing for Physical Brands
You could test out whether one should produce new clothing styles by making them first available for avatar customization. If they sell, then it may be that you should offer it in your physical stores.
Explosion in Virtual-First Brands
There will be brands that focus purely on virtual clothing and attire. These brands may become absolutely first. There will not just be a few of these brands, there will be hundreds, then thousands and then tens of thousands. This will be because the start-up costs are low and the tooling and knowledge required to make unique virtual clothing is nearly free on Youtube and via open source tools.
Virtual Brands Making Physical Drops
There will also be brands that focus on virtual first and then once they reach a certain level of popularity they may start to do pop-up stores or physical goods drops. Thus cross from virtual to physical.
Ecommerce Stores that Sell both Virtual and Physical Goods
Ecommerce stores in the future will offer not only physical goods, but the virtual version of the same thing. You can choose which you want to buy. Sometimes both? Why should stores limit themselves to just physical or just virtual when both are possible, especially when one is creating the virtual goods anyhow to enable virtual try-on.
Imagine that when you are checking out buying your daughter her new t-shirt, you will be offered the ability to also purchase the NFT for a virtual t-shirt that is exactly the same as the physical one for a couple extra dollars. Then your daughter can wear that t-shirt both in the real world while also wearing it in Roblox or any other metaverse-like game.
Robust Secondary Markets for Avatar Customizations
There could be robust secondary markets where previously purchased avatar customizations can be resold to others at market rates. This will be possible because avatar customizations will be represented by NFTs.
Wearing Virtual Clothing in the Real World
Once augmented reality technology matures enough, it is likely to be worn at least by parts of society in a relatively persistent fashion. This means that many will have a constantly augmented experience of reality. In this future scenario, it will make sense for one to wear one’s virtual avatar customizations on your physical presence -- of course those avatar customizations, or your virtual clothing, will only be present when you are viewed via augmented reality, but if AR becomes pervasive, this may be the main way you are perceived.
Thus people may be buying virtual clothing to wear out in the real world, rather than just on their virtual avatars in video games.