Download presentation
Platform MMI for the Metaverses
PROJECT PORT-HOLE or «PORT»
Author E. Shubralov
Objective of the project:
To give the world a Mega (Meta) assistant - a tool for interacting with the outside world and the home world in the form of a Port device (with a monitor imitating a porthole) represented by the virtual personality of Umnik (Egg Head), "living and working" in the neighboring (behind the wall) space for the benefit of its user.
The functions of the Port device cover three areas:
І - Mega Assistant, providing interaction in all areas of life (Communications, Health, Finance, SH, Security) and using a flexible system for setting up all its channels / interests / tasks, both independently and through managing a home network of devices;
ІІ – Device Port as a porthole – as a window to the virtual worlds – to the worlds of the Meta universes and to its own MMI as a new user scenario, as an essential addition to VR helmets and AR glasses;
ІІІ – Port as a window (porthole) to the real worlds – to the Universe and to the natural world of the Earth in the mode of movement, in the mode of flight and immersion, in the format of realistic virtuality - RV.
I. Technical part of project
1) Technical field
The scope of this device (hereinafter Porthole or PORT) is to help a person perform various kinds of routine tasks that can be automated - as well as structured control over various kinds of home assistants that already exist in the man-made environment surrounding a person (existing in the form of various kinds of gadgets - from smartphones to smart speakers and assistants in home appliances).
2) Prior Art
To date, mobile voice assistants (Alisa, Google Assistant, Alexa , Siri, Salyut and others) are known - each of which exists in its own closed and isolated ecosystem. At the same time, solutions that can always be online do not exist. Obviously, due to these circumstances, a situation has arisen in which existing solutions are not able to provide the ability to work with a sufficient level of penetration into a person’s life, but solutions that can provide the proper level of interaction (including in the form of control of already existing separately, each isolated in its ecosystem of voice assistants) is not yet available. Obviously, there is a need for a meta-assistant to appear (capable of managing assistants, at least in parallel - given that, due to the development of existing assistants, they can only perform tasks sequentially, and life dictates the need to perform tasks in parallel). And also able to independently (using the settings and management table) assist the User in all his possible personal interests.
3) Disclosure of a technical solution
A technical solution that allows to combine everything a person needs at this stage of the development of human society should include:
- the ability to manage the existing fleet of voice assistants;
- the ability to always be in touch with a person;
- the ability to provide a person with the need to communicate with a machine in a natural language using voice communications, which will ultimately allow him to receive various kinds of assistance in all areas of his interests and home needs;
- the ability to be involved in an exciting process of interaction with a smart and live virtual individual created by the User with his own hands and head, which subsequently will have every reason to become a family member and a trusted executor in all matters that will be entrusted to her.
4) Implementation of the technical solution
The technical solution is implemented using the following possibilities already mastered by the industry, among which the following can be distinguished:
- to play a voice stream from a device to a person - devices that support the UpnP or DLNA protocol (allowing playback of PORT messages on any device in the house that supports the specified protocol);
- for receiving a voice stream from a person - the above protocol for receiving voice messages within a home WiFi network
- to output visual information (3D model of a meta-assistant in an appropriate communication environment) - the above UpnP & DLNA protocols, as well as the MiraCast protocol - to output visual information through television or projection devices located within a home wireless or wired network.
- to search for a person within the house - microphones of home devices available via DLNA / UPnp protocols, which allow determining the positioning of a person inside the house in order to communicate with a person through the input-output device closest to him.
- for visiting third-party meta-universes - the entire existing infrastructure for input-output of sound and video information through a flexibly customizable meta-assistant command apparatus.
5) Implementation of the technical solution
The technical solution is implemented using the following possibilities already mastered by the industry, among which the following can be distinguished:
- to play a voice stream from a device to a person - devices that support the UpnP or DLNA protocol (allowing playback of PORT messages on any device in the house that supports the specified protocol);
- for receiving a voice stream from a person - the above protocol for receiving voice messages within a home WiFi network
- to output visual information (3D model of a meta-assistant in an appropriate communication environment) - the above UpnP & DLNA protocols, as well as the MiraCast protocol - to output visual information through television or projection devices located within a home wireless or wired network.
- to search for a person within the house - microphones of home devices available via DLNA / UPnp protocols, which allow determining the positioning of a person inside the house in order to communicate with a person through the input-output device closest to him.
- for visiting third-party meta-universes - the entire existing infrastructure for input-output of sound and video information through a flexibly customization aware meta-assistant command device.
The terms of reference for infrastructure development include the following components:
- the PORT project monitor is implemented on open source hardware (in the RISC-V architecture), or on platforms derived from this architecture (for example, AllWinner) in the form of a single-board computer that allows running an operating system based on AOSP (Android Open Source) Project), to display the HDMI output using the hardware implementation and the corresponding necessary inputs, while some subsystems can be implemented by separate boards - for subsystems that require increased resource consumption;
- the reference computing node is implemented on open source hardware (in the RISC-V architecture), or on platforms derived from this architecture (for example, AllWinner) in the form of a single-board computer that allows running an Ubuntu-based operating system (Ubuntu Mate) on it serving the necessary hardware (stored data, P2P WiFi Direct networks, etc.) - similarly, based on the computing power of the reference computing node, a smart audio player can run - a digital rights management system that allows the owner of the system to listen in the highest quality by broadcasting sound streams to devices DLNA audio materials, the right to listen to which is provided to subscribers for a monthly subscription fee (a practice used in various music services such as Apple Music, Google Music and others, which differs from the method implemented in the PORT system by low in terms of frequency and, accordingly, the quality of sampling rations by the playback quality value - while the playback device allows the copyright holder to transfer music to network terminal devices in compressed TAC (Time-Amplitude Coding) format - the rights to which belong to the authors of the PORT project;
- quadrocopters from the PORT system are implemented on the basis of existing models of quadrocopters with the integration of additional open source hardware (in the RISC-V architecture), or platforms derived from this architecture (for example, AllWinner) in the form of a single-board computer that allows you to run an operating system on it on based on Ubuntu (Ubuntu Mate), which has feedback with the main quadcopter control logic exclusively in terms of room navigation in order to increase the efficiency of servicing the system owner (audio video control by gestures and voice commands) and timely recharging (in order to be able to stay in the air semi-permanently) - the project monitor in conjunction with the network core nodes provides end-to-end routing of incoming telephone calls to various places in the room where the owner of the system is currently located (by switching the call forwarding modes on the mobile device to virtual numbers landing on various devices where the owner is currently may be located) - taking into account movement through the premises and external spaces within the house (estate) due to the timely switching of virtual numbers (DID) as the goal of the unconditional forwarding mode;
As a result of performing the activities of a mega-assistant, based on a flexible system of configuration and control (interaction), as well as on the implemented ability to control home devices, the possibility of comprehensive support for the user will be achieved, both in all his communications and in all issues in the areas indicated above - in fact, on all matters of life.
6) Visual information
In this section, we present the device hierarchy
Visually, the hierarchy of devices in the project can be seen in Scheme No. 1.
Explanations for the diagram are given below.
- the root server and servers of lower levels are the main core of the MMI meta-universe system and provide routing of calls to different levels of the space and time hierarchy in the coordinate system of the MMI project and its meta-universe
- the PORT system monitor is a multi-functional device, the main task of which is to display visual information on the display, which is able to show both the contents of the meta-universe of the MMI project, and the interiors in which there is a visual virtual person - called Umnik, which makes control possible meta-sets of devices used in the process of human life - with the only difference being that being an electronic device, unlike a person, Umnik can perform several tasks at the same time - for example, ordering tickets for a flight, ordering a taxi in parallel and checking the availability of food in the automatic feeder for pets - to complete such a pool of tasks, a person will need much more time than in parallel with the executable Umnik. To perform such a wide range of duties, the PORT System Monitor is equipped with a modern infrastructure (for example, based on a mobile system based on a MediaTek chip running Android OS, using the Awaya3d 3D library and Unreal Engine to create Umnik models, in whatever form you would like it to see its creator.
Of course, to ensure interactivity, the PORT System Monitor is equipped with sources for input and output of sound commands
- the data storage system is implemented either on mobile processors in a compact design, or in the form factor of NETTOP class devices - which allows you to store system data on a dedicated device for a number of reasons
- the reference node is an intermediate link located in the peer-to-peer network formed by the virtual assistant, created at the start of the PORT system
- TVs and gadgets in the PORT system play the role of a kind of "smart watch" - having the ability to notify the owner of the system about emergency situations, or broadcast communication with Umnik, wherever the owner is
- quadrocopters in the PORT system play the role of remote microphones of the Network Monitor - allowing you to provide audio contact from anywhere in the room, regardless of the distance from the main monitor, while, for energy efficiency, such quadrocopters can be located where necessary (where there is noise, for example, produced by the owner of the system) - while monitoring the situation, simultaneously charge your batteries at the nearest reference device, which also allows you to recharge the copters
- a system for taking medical indicators of the owner (and / or members of his family) allows additional control over the health of family members and the owner
- mapping and space services - extensions available through the PORT System Monitor, allowing you to expand the range of capabilities applicable through the PORT system
II. Graphical interface description
1) Introduction
The interface of the PORT project is a graphical interface that implements a visual environment in which, depending on the circumstances of communication, a meta-helper called Umnik is located and interacts with a person. This interface is implemented by executing an application that implements a 3D model and displays it on the monitor screen (supplied separately as part of the PORT system, which has on board an operating system distributed free of charge, for example, based on the Android OS - on which an application is run that provides the ability to interact with subsystems PORT meta helper interface).
The monitor device may include cameras that connect to the device via the USB Host functionality, as well as microphones that allow two-way data exchange.
Additional dependent monitor devices can be voice-activated quadrocopters standing on charging platforms in the absence of sounds, activated in the presence of sound information, which provides voice communication with a person - in situations where the distance from a person to the monitor exceeds the distance of the sensitivity limit of standard microphones and device cameras.
The purpose of quadrocopters can be as follows: flying around the territory along routes around the house, inside the house, and as a means of communication between the User and the Smart Guy in certain situations, as a means of delivering a smartphone, a glass of coffee from a coffee machine to the User, as a means of finding keys, a remote control and any other item, which the Clever will remember. The command for the quadcopter to fly around on such and such a route:
- Smart guy, send the copter along route 1 and give me a picture on my smartphone.
- I understand, I do.
Additional PORT network nodes are HDMI reference devices that allow you to run the necessary software products that are part of the PORT device core network on HDMI devices and a home WiFi network.
These additional nodes of the PORT network implement, in addition to the functions of the reference node, the functions of content playback (audio content in TAC format, which is received directly from the servers of the copyright owner, and reproduced in the best possible quality through the DLNA network of terminal devices that allow playback of such content At the same time, the player for playing TAC content is protected by a digital distribution system that allows the player to be launched only if permission is received on the server of the copyright holder, without the possibility of dumping such a protected process.
2) Metaverses and Existing Visual Resources
The interface of the PORTHOLE device allows you to view an unlimited number of meta-universes (including the meta-universes of the METAMATE.INTERNATIONAL project, hereinafter referred to as MMI) - without taking part in providing the infrastructure of external projects, providing only a visit to them) - due to which there is no need to spend energy resources on computing power performed in the cloud.
The METAMATE INTERNATIONAL project is paired with the MetaExchange X888 blockchain project, which allows all calculations in the meta-universes to be carried out using ERC20 tokens and a corresponding decentralized trading platform that allows you to exchange these and any other tokens through automatic trading contracts.
Pairing the MMI project with this project provides something that for a long time cannot be realized either by virtual reality glasses or other visualization tools - yes, the virtual world, excellent! Yes, you can be in it, but due to the fact that a person receives up to 90% of information visually, he SHOULD be SEEED - and seeing him by exploiting the inertia of human vision is hardly the best idea.
Whereas the porthole of the PORTHOLE project and its brave helmsman Umnik is able to provide a truly unlimited view in any dimensions - and no means are needed to control the view, except for voice and gestures.
In MMI projects, which at first may seem to someone like projects a la Second Life, there are no boundaries - yes, that's right. Given the unlimited scalability of latitude and longitude coordinates, we can create both macro-worlds (noticeable only with a known level of coordinate scaling) and micro-worlds - again, using coordinate scaling (latitude, longitude, height and time).
The MMI metaverse has been timed since November 2007, the date that a pre-registration describing this project was submitted to the Library of Congress as a pre-registration in order to stake out the time of the idea in its current sense.
During the existence of the growing MMI meta-universe, a huge amount of work has been implemented - including the security of connecting to virtual worlds in the form of proxying requests to MMI objects, as well as the infrastructure for listening to music with a quality that was truly inaccessible before (with a sample rate of 384 kHz without changing the music files in size) . In addition to such revolutionary breakthroughs, within the framework of MMI, the functions of a virtual stylist are implemented - who, by analyzing the appearance of a person, is able to find in his wardrobe items of clothing in which the proportions of the golden section are more clearly manifested in their owner - in the presence of which the person looks much more attractive than without them.
What else I would like to note in terms of the Metaverses is the potential of its Cartographic resource, which, in addition to its extensive functionality, has good prospects of becoming the basis of the future ecosystem of the project, which, with all its participants, services and content, in turn, can become a good basis for creating own Metaverse of the P-H project within our Metamate.international (MMI) platform. Especially if we keep in mind the presence in the project as the main communication system of the WeChat system.
3) Interface customization
To customize the interface capabilities, the appearance of Umnik and other settings that can be changed, a customization table is used that has no restrictions on the amount of parameters (including those indirectly arising from the added parameters).
For voice interface customization, taking into account the data of the tuning table, the TTS (Text To Speech) Android OS interface and the STT (Speech To Text) interface are used, which are optimized for execution on mobile platforms and processors that do not require large computing power.
In addition to voice commands, the interface customization also includes modeling the appearance of Egghead by the user himself. We believe that through technological partnerships with leaders in the visualization market and products for it (we mean, first of all, the respected company Epic Games), we will be able to achieve the best results in creating a unique and comprehensively useful virtual character Umnik, which so far has no analogues of such a presence and applications, as well as with the possibilities of its animation of any complexity.
Of course, you can also simulate the voice - as studies of focus groups have shown, people most often like their own voice (and we have algorithms that allow us to simulate the voice of the owner of the Umnik - making communication with him easy and enjoyable).
4) 3D libraries for displaying worlds
The Away3D Engine, a library optimized to work on mobile devices and processors, is used to implement 3D communication of Umnik.
Of course, one library is not enough - a full-fledged 3D scene needs both a scene and a model - but the models in the project will be generated by the recognized gurus of 3D modeling and display - the Epic Games SDK (and their Unreal Engine).
The displayed worlds, in turn, are developed by maintainers who create their own virtual worlds, taking into account the main concept of the MMI project - scaling space and time and its immutability (remember the film with Bill Murray "Groundhog Day"). Yes, many people like to return in their memories to those days or those places that are dear to their hearts - at the same time, as established by physiologists, at such moments the human body experiences real happiness - at least at the level of dopamine hormones in the blood. And what I like, I want to show everyone and everyone - that's what MMI universes exist for.
5) Interactive interaction with other meta-sources
To interact with the MMI meta-universes and monetize the resources of these universes, the Meta Exchange X888 blockchain platform is used, while any resources of the meta-universes can be purchased and sold for ERC20 standard tokens called X888. To implement meta-universes at the level of 3D objects and avatars, the Unreal Engine library is used, which allows high-quality formation of 3D objects and their interaction with each other. Additionally, the developers of this project, in cooperation with the developers of the MMI project, decided to distribute a certain number of X888 tokens so that any virtual migrant can create their own cozy little world for free, which can be viewed through the PORTHOLE porthole
6) Additional core network nodes
To optimize computing resources, additional nodes of the core network of the PORT device can be used - implemented using NVIDIA's CUDA technologies (in the form of modules separately connected to the home network via uninterruptible power supplies, allowing you to make the necessary calculations for your immediate location, with the possibility of renting computing resources to other members of the MMI network with the possibility of using the corresponding third-party resources of other members in case of a shortage of their own.
III. Programming model of project.
The software part of the project, which runs on the monitor, is implemented on the basis of the Adobe AIR ecosystem (Umnik in the monitor, Umnik in the Android application, Umnik in the iOs application).
Speech recognition and synthesis software (support for 26 languages) - ezTTS/ezSTT for implementation in the Adobe Air ecosystem
3D display software - Away3D for implementation in the Adobe Air ecosystem.
Visual emotion recognition - based on the Phase Only Correlation (2D FFT) algorithm for implementation in the Adobe Air ecosystem (Air Native Extension with OpenCV).
The software part of interaction with MMI is implemented in the Adobe Air ecosystem.
The software part of interaction with DLNA & UpnP is implemented in the Adobe AIR ecosystem (Air Native Extension with Java Native Interface).
Regarding project security, the project operator implements a reverse proxy system that allows you to protect any resource from ddos attacks, as well as cache data available for caching (the closest analogue is the Cloudflare service).
IV. Integration part of the project
The integration part of the project is implemented on the MMI side
V. Project content
VI. Project monetization. Sources of income.
1) Sale of the Port device - monetization of the main line of business.
CALCULATIONS are approximate for three markets.
Let's take the middle class as a guideline when defining the SAM market.
China. The middle class (up to 100 yuan per day) - 12% of the population, or 150 - 160 million people.
If we take as an assessment of the market the number of houses / apartments where the device can be placed, the average figure of the number of people in the family (say 4) and singles from this middle class, then we get the figure 2.5 (4 + 1/2) - then the volume of available market in China, we can get 60 million houses/apartments/families (150/2.5).
If we take a net profit margin of 10% at a price per unit of $2,000 and without taking into account the economics of other components of the project, we will get a market in dollars of 12,000,000,000. And this is the minimum, since a family can have not one device, but two or more. There are figures for China's middle class and 400 million people.
Europe. 200 million is the middle class in Europe, or let's take about 70 million houses/apartments/families. Accordingly, the market volume will be $14 billion.
USA. The middle class is 150-180 million people, or 60 million houses/apartments. And, accordingly, the market will be from $12 billion.
We took only three markets and with minimal indicators.
2) Sales from the project website:
From the Help Center (Health, Finance, Security, AM, Communications):
- to the Health system to offer digital devices for reading indicators;
- offer a number of devices for the Security system: webcams, opening sensors, motion sensors, etc.;
- to the Smart House and IoT system, offer the relevant products of these groups from partner companies or other manufacturers of Smart house systems for their subsequent inclusion in the management of Umnik.
From Additional Devices:
- Tee for communication with the User throughout its territory;
- Quadcopters of different manufacturers and price categories, but applicable in Port/Umnik destinations;
- Mini robot and other suitable products from the IoT category.
3) From the Map Service:
- You can offer a subscription to connect to Web cameras (layer No. 2);
- You can monetize all layers of the Cartographic resource through a subscription
4) Subscription to the Space Engine simulator
5) Advertising:
Advertising is placed and monetized in all components of the project, where it is appropriate and appropriate.
6) MMI and other Metaverses
Income from MMI and other Metaverses is primarily due to the operator fee (in relation to the MMI infrastructure, the project is an operator that coordinates the actions taking place in the infrastructure by routing requests and ensuring that the requested information is received in whatever form it is provided. The infrastructure operator provides attracting institutional players to participate in providing infrastructure (manufacturers of servers and active network equipment will not be able to find advertising better than assessing the benefits of using such equipment in the metaverse - at the same time, the scaling principle works on smaller scales (imagine that the northern hemisphere of the Earth in the metaverse was bought by Intel, and the southern hemisphere of the Earth in the metaverse was bought by AMD.) Within the hemispheres, parts of the continents were bought by large corporations the size of Xerox, IBM, Cisco - if the buyers of the hemispheres were given control and responsibility for global measurements (hemispheres), smaller companies can operate with parts of the continents - and even smaller companies - with sizes of the order of countries and states, as well as spaces between continents. Within countries there are regions, provinces and other divisions of territorial division - within which, respectively, there are cities and streets - and on them there are houses. Now let's imagine how many people want to have their own corner in such an infrastructure - say, an apartment, or even just a room (it is possible that under the stairs, like in Harry Potter))).
At the same time, the obligation to maintain the ideal state of the infrastructure provides the relevant sub-operators with the rights - any profits derived from the areas provided by them are subject to the obligation of the recipient of the profit to share it with the corresponding sub-operator (at the same time, the entity directly extracting income can communicate with the sub-operator closest to him in rank he will transfer the received fee, minus a certain value, to a higher suboperator - as a result, providing scalable earnings to any suboperator, especially a high-level one (how many lower levels are in his area of responsibility!)
Of course, investing in project infrastructure is a win-win investment (and backed not even by the main system operator, but by the authority of high-level sub-operators - who actually provide the infrastructure.
7) Income from loyalty programs
Income from loyalty programs (an agreement with trade and service enterprises to provide services to customers who came on our recommendation, taking into account partner remuneration in favor of the project - with parity of remuneration to the size of the discount, trade service enterprises receive a huge influx of customers and earn on scale, while the project ensures the influx of new and retention of existing customers from among the subscribers of the System).