An AR app for interacting with celebrities

Solution

Business analysis

We began by elaborating requirements for the applications and defined them as follows: high performance on iOS and Android platforms, high-quality graphics, simple source code, integration with native mobile device capabilities, and scalability. 

Next, proceeding from the requirements, we chose the AR technology stack for the project. In the end, we opted for the ARKit development platform for the iOS app and ARCore for the Android solution, since building with them would guarantee full compliance with app store guidelines, consistent look and feel, and high performance and scalability.  

For the Android application, we chose the MVVM (Model-View-ViewModel) architecture pattern and modular design to make the solution more flexible and scalable. As for the iOS application, we opted for the Coordinator pattern for its MVP as it enabled us to separate the app logic into layers, implement abstractions, and build effective navigation. 

Functionality

Itransition developed iOS and Android applications for interacting with celebrity avatars in real time and buying their exclusive merchandise. The applications provide several membership plans which differ in the scope of features available to users. For example, the free application version allows users to communicate with only one avatar, while subscribers can select three avatars and communicate with them without ads.

Interaction with avatars

Once the app user initiates an interaction with the selected avatar, the system opens a phone camera to scan the environment in front of the device and overlays the avatar onto a horizontal space. 

The application receives the files with avatar data from the server via Rest API HTTP. Using capabilities of ARCore and ARKit, the app establishes the phone’s location in the real world and enables realistic interactions between the 3D avatar and horizontal surfaces. Then, using SceneKit and Sceneform engines, the solution imports 3D avatars that users can place within the room, scale, rotate, and move them around. The app also allows users to change the background, play different animations, and take photos with the avatar.

The application allows users to ask avatars a list of questions which we compiled based on our research and added answers to. The application listens to the user’s question, converts their speech into text, and sends it to the server. The server processes this data and sends back the answer for the avatar.

If the user doesn’t communicate with the avatar for some time, or the system can’t find the answer, or the server response time is too long, the application plays a default set of animations.

Purchasing through Amazon

The application allows users to buy celebrity merchandise by clicking the marker on the 3D model’s outfit. The application then opens the item’s preview from where users can access its Amazon page and purchase it there. 

The purchase process does not interrupt the communication with the avatar, with the avatar showing in the background but keeping silent. The user can also save a specific item to the Favorites list.

Notifications

Avatars can also notify users about the celebrities’ upcoming concerts. If the concert takes place in the user’s city, the app notifies the user about the event and its date and offers to buy tickets. 

To provide these notifications, the application requests permissions to access the user’s geolocation data and send notifications. We applied the native mechanism to enable local notifications and used Firebase Cloud Messaging for push notifications.

Advertising

To enable the display of advertisements for non-subscribers, we integrated the applications with the Amazon Advertising service. We also applied the aspect-oriented programming framework to define the points at which we want to integrate ads and when the engine should display them. 

As a result, the app displays the list of stores and an advertising banner at the bottom of the screen and opens it in full size on another screen that the user can close.

Design enhancement

The customer had already created the applications’ design, but they wanted to refine it and entrusted the task to our team. We introduced some important changes, for example, created gradient backgrounds and configured user interface elements to make them more intuitive.

Application security 

We strengthened the app security with the following tools:

• Themis, a cryptographic services library for copyrighted content encryption
• AWS Identity and Access Management, a service for user permission management
• Amazon Cognito, a service for multi-factor user authentication and encryption of data at rest and in transfer 

We also conducted large-scale security testing to define potential vulnerabilities and security issues in the application. Our QA specialists performed the testing manually using semi-automated tools for labor-intensive tasks like fuzzing and brute force testing, open ports discovery, and more. We also conducted such specific security checks as:

• Network testing to identify security weaknesses in the network
• Local storage testing to assess the security of application data stored on mobile devices 
• Evaluation of how applications interact with their respective operating system 

Results

Itransition delivered AR mobile applications for Android and iOS that allow users to interact with 3D celebrity avatars in real time and purchase their exclusive content and merchandise.

So far, the customer achieved the following results with the applications:

• $1+ million in investments raised
• 3,000+ registered users