[Service Development Team Jeon Jeon Jeon]
Facebook AI unveiled the Droidlet platform for robot development that can be used in real and virtual environments on the 28th of last month. Although the published code is not yet fully documented and tutorial, it is open source that is being actively updated from the existing code on Facebook.
A robot operating in a real environment can work well only when it has an awareness of the environment and an accurate recognition of the user's commands. To help with this robot agent programming, Facebook has released Droidlet, and the published code can be used in a real environment. LocoBot or for robot development in a virtual environment. Minecraft We are making it compatible with the game. It consists of the end-2-end code that controls the robot or the agent in the game, but since it is composed of modules, it seems that it is possible to create a new intelligent robot by making a trained ML model and linking it with Droidlet.
https://ai.facebook.com/blog/droidlet-a-one-stop-shop-for-modularly-building-intelligent-agents/
Minecraft is a game that allows you to interact with various objects made up of blocks in an open world world and non-player characters (mobs) such as players and animals. Facebook AI has long used Minecraft games for experimental environments such as NLU Tasks. (Why Build an Assistant in Minecraft?)
CraftAssistproposes a framework for developing intelligent agents in the Minecraft environment. CraftAssist uses structures such as dialogue managers and memories to derive actions to create buildings or objects through interaction with other players and objects. We are helping to develop agents that can be used in Minecraft through structured data. CraftAssist is included in the source so that it can be used organically with other modules of Droidlet.
https://arxiv.org/pdf/1907.08584.pdf
I made an agent for Minecraft using Doridlet. Facebook also provided an NLU model, and when a user types a conversation in chat, it is transmitted to the NLU model through the server, and the model can perform a specific action by outputting an action to control the agent's behavior. As you can see in the video, I was able to basically perform simple conversations and making shapes and objects.
These days, the virtual world “metaverse” is in the spotlight, and in this metaverse environment, an intelligent agent is needed to give users a more realistic experience and create a virtual environment that feels more real. Droidlet seems to be very helpful in providing an environment where you can create and experiment with various intelligent agents.
github: https://github.com/facebookresearch/droidlet
paper: https://arxiv.org/abs/2101.10384
