Project Name
Project: Themistoklis
Project Category
Build & Integrate
Proposal Earmark
2nd/3rd Grant
Proposal Description
Creation of 3D Engine based on Panda3D in python, with connections to Ocean Marketplace, for synchronization between clients
This way, users will be able to sync scene data between clients in real-time using ocean marketplace inside the engine, so all users can have the same information and map data without needing a real-time map data stream between them.
Grant Deliverables
- Panda3D system, that can load models easily, camera, headless and headfull
- Panda3D examples
- Direct connection between Panda3D clients using TCP protocol
- Automatic sync between clients using the connections, with other clients (based on a pre-selected whitelist)
- Sync with ocean marketplace assets between other clients
Project Description
What is Project Themistoklis?
It is best to illustrate our project through an example. Imagine there is an orange forest farm and you want to count the number of orange trees in the forest.
Source: Apple Orchard Early Morning Aerial by jimmadsen | VideoHive
Source: Orange trees detection with YOLO v5 in UAV Imagery | by Joao Otavio Nascimento Firigato | Medium
Figure 1: A Farm.
How would you do it?
There could be many ways but this is how Themistoklis intend to tackle it.
Firstly, fly a drone using our software stack to map the forest. As the forest is being mapped, data of the world is sent to a World Server that directs the data to its destination. In our case, the data reaches an object detection server. We would have many object detection models hosted on that server. We will use one that detects and counts the number of orange trees in the farm. Once counting is done, the count and the related data will be shared with us through an access control protocol like Ocean Protocol.
Our hardware stack
- We are developing drones for our project fitted with different sensors
- Sensors includes cameras, night vision cameras, and other sensors
Our software stack
-
Desktop/iOS client
-
Drive a drone or group of drones including using NLP
-
Directs data from drone/drones to World Server
-
World Server
-
Data is received from the client
-
Direct data to different servers as per the requirement of the project
-
Object detection models
-
Customised object detection models such as fire detection and orange trees detection models,
In short, Themistoklis aims to make it easy to capture real-world data with drones and offer off-the-shelf models to derive insights from the captured data. It also supports generating 3D of the real world which can be used in VR game development. These real-world data can be monetized through Ocean and other web3 protocols. This is articulated in Figure 3.
Another example is fire detection. The drones will be able to detect early fires in the world, while patrolling around sending a signal in case they catch one, in order to avoid the devastating effects of forest fires.
Figure 2: Fire detector on forest fire image.
Figure 3: Articulates what Themistoklis is.
Figure 4: Themistoklis development stack
Project Themistoklis is a project by artificialLeap. artificialLeap is developing both hardware and software necessary to make it easy for us and anyone to capture data from the real world and monetize it.
Figure 5: How Themistoklis’s different stacks interact
Final Product
Features:
- Simple to use Android, iOS & Desktop (Windows, Linux, MAC) app to connect to remote drones (front-end - client)
- Middle Server that handles the world building (unity or nodejs), a headless app that connects with Thoth (AI) and with the Client, will generate the world around the drones and get world-wise information to the AI
- Everything will communicate together using TCP sockets, for ease of use and faster movement of data.
- The AI will be able to generate commands based on the input from the Drones and the MiddleServer, will be fetching objects around each drone, real-image visuals with objects detected (will be spawned in the World Server too)
Architecture:
The Project will consist of 3 Apps:
- The Client (will be accessible to users and with this will be able to update the data and drones, also will have the connections with the drones, will be able to run on Windows, Linux, Mac, iOS or Android…
- The MiddleServer/World Server, will be built with Unity or NodeJS, will create a 3D world with the world and the drones, will be a headless server, which will be able to run on Linux, Mac or Windows (headfull too).
- The backend Server, which will be Thoth, it will handle the AI directly, will be connected to the World Server (TCP) and to the Client (Web Requests), will be doing all the heavy lifting and generate the output for each command. Thoth also includes a web editor (node graph) to easily create AIs.
Tech Stack:
Front End Application:
- The client will be the API connector with the drones, it will be able to generate multiple connections at once.
- Will be able to get information from the drones (camera, altitude etc)
- Will send the generated commands (move towards, strafe etc) to the drone(s)
- Save/Load Drone Formations, default speeds, max altitude etc
Middle Server - World Server:
- Backend will generate a 3D world around the drone at a smaller scale with information visible to the camera, which will work like a small rendering game camera
- The world will include all the drones that are connected to it
- Each AI in the world will have a unique id, based on the client id for it
- World Server will be able to generate 3D maps from the real world and export them easily as 3D models
- All world servers will sync in an interval through blockchain so all drones will share the same world map
Back End Application - Thoth:
- Thoth will include all the models and AI data that will be needed to generate the commands
- Includes a Web Editor (Node Graph, like UE4) that lets the user create easily graphs for the AI behavior
Models Needed:
- Computer Vision: Visual Object Detector - Python algorithm using ImageAI/Tensorflow to detect objects in images, includes scripts for various clients (screenshots, images, videos, game windows)
- Decision Making: Will use NLP for decision making, probably OpenAI (GPT3) or GPT-J (which is open source and self-hosted). This will create a Word-Based World, which the AI will use and will send word-commands for the movement
- Maybe More?
Drone Prototype:
- There will be a prototype drone with more sensors needed for the best use of the AIs (although it will work normally with other types of drones too)
- It will include all the basic sensors, but also a Thermal Camera and a Night Vision camera, for better results during night.
Web3 Integration - Blockchain:
- Save/Load and Mint Plans for Drones (Formations, good speeds etc)
- Static Website in Blockchain- Manager for the World Server/Drone Manager
- Login System through Blockchain
- Pub-Sub system using Blockchain Canisters ?
- Implement Blockchain to unity Client for login and save/load data
Tests:
- Early Fire Detection:
Value Add Criteria
- Custom game engine built on Panda3D engine
- Multiplayer option for Panda3D
- Panda3D Ocean Marketplace support
Core Team
Alexandros Titonis
Role: AI Engineer & Full Stack Developer
Github: alextitonis · GitHub
LinkedIn: https://www.linkedin.com/in/alexandros-titonis-0ba061176/
Experience:
I’ve been working lately on Thoth a Node Web Editor for AI, i’ve worked on webaverse and xr-engine.
In previous years i focused working on Multiplayer games in Unity.
Funding Requested
15000
Minimum Funding Requested
6450
Wallet Address
0xd5bDc028aaEAb8D8fa1C5Bb577b6C4C3402559F6
