Linuxslate.com Forums

Linuxslate.com Forums
http://linuxslate.com/cgi-bin/forum/YaBB.pl
Original Projects and Builds >> Other Builds and Projects >> Telepresence Robot Build - With Code!
http://linuxslate.com/cgi-bin/forum/YaBB.pl?num=1640407632

Message started by Administrator on Dec 24^th, 2021, 11:47pm

Title: Telepresence Robot Build - With Code!
Post by Administrator on Dec 24^th, 2021, 11:47pm

My son built a Simple Telepresence Robot for a Summer / Stay-at-Home project.

What follows here will be a basic description of the Design, the Finished Robot, and a Basic Description of the Code.

http://linuxslate.com/IMG_TPR_Small.jpg
Basic Telepresence Robot. Photo property of Linuxslate.com

Design Goals for a Basic Telepresence (https://en.wikipedia.org/wiki/Telepresence) Robot:

The Telepresence Robot should allow the operator to communicate with and interact with people at a remote location as if the operator were present at that location.
In other words, it is just like a Zoom, MS Teams, or Google Hangouts/Chat call, but the remote caller can move around from place to place. Their presence is not fixed to a stationary device, or carried by another person.
As with the Teleconference call, the call is not limited to the same area or "range" of a point-to-point connection. The Telepresence Robot can be anyplace that there is an adequate internet (WiFi) connection. This could be in a different building than the operator, or even on a different continent. It is different than "Radio Control".
The Telepresence Robot should be able to move over types of flooring commonly found in a business environment, including carpet, tile, or bare floors. The Telepresence Robot should be able to move over discontinuities typically found in such an environment like thresholds or entering/exiting an elevator. For this implementation, there was no requirement for the Telepresence Robot to navigate steps. It was desired to have some outdoor capabilities such as moving over a surface made of paving stones. The Telepresence Robot should remain stable on such surfaces, including accessibility ramps.
The caller should be close to eye level with other participants. This was implemented as a compromise between seated participants and standing participants.
It is assumed that the operator has access to a PC, phone or tablet with internet capability, however the operator may not have the ability or permissions to install special software or an "app". The robot is controlled via a standard Web Browser. Standard teleconferencing software such as those mentioned above is used for the Audio/visual call.

Use Cases:
What is different about a Telepresence Robot, and a "normal" Zoom Call, or other Teleconference?
The main difference is that the remote user can move around an area at the remote location; much as the remote person could if they were actually present. Here are some possible scenarios to help the reader understand the difference between Telepresence and a Teleconference.

A retail store operator who is quarantined at home due exposure to a contagion (such as COVID-19) could continue to greet customers at the store entrance, interact with them, and guide them toward the products they wish to purchase.
A hotel employee could guide guests to their room, or answer room service calls while continuing to have a real presence in the hotel lobby. Guests may be more comfortable having a robot come to their room than an actual human.
A product purchaser could monitor the production (and possibly testing) of a product for procurement at their convenience and without the attendance or possible influence of an employee of the company producing the product. The purchaser could move within agreed limits to monitor the production or testing process as they choose.
An employee could monitor a process in a hazardous area from a remote (safe) location. They could move so as to observe the process from different vantage points without needing multiple fixed cameras, or the required bandwidth of numerous cameras.

http://linuxslate.com/IMG_TPR2_Small.jpg

Implementation:

The Basic Telepresence Robot is implemented as a "DOIT" Smart Tank (https://www.vvdoit.com/Tank-Chassis-c12244/?currency=USD) Chassis.
The Robot control was implemented as a DOIT provided ESPDuino (https://www.aliexpress.com/item/1005001887564560.html) and 2/4 motor controller daughter card. This provided easy USB connectivity for software development and uploading, as well as ESP 8266 WiFi connectivity (Upper Center of above picture).
Due to the motors used and the desired power, a separate 2 Motor Controller Board from Banggood (https://www.banggood.com/DC-3-36V-15A-Peak-30A-PWM-DC-Dual-Channel-Motor-Driver-Board-High-Power-H-Bridge-Control-Module-p-1255708.html) was used. (Upper left of above picture).
Remote video for navigation is provided by an inexpensive Web-cam (Lower Center of above picture), and OpenWRT router (Lower Left of above picture).
Power is provided by a common hobby type lithuim-ion battery.
The motor controller function of the daughter card is currently not used, but the card is retained for future use for features like manipulators, camera pan/tilt functionality etc.

In the next section, we will talk about the software implementation.

Title: Re: Telepresence Robot Build - With Code!
Post by Administrator on Dec 25^th, 2021, 12:30am

In this section, I will describe the Robot Control Software.

A Few Words about Security:
Before I begin discussing the software, I should emphasize that this is a home (hobbyist) project. It is not a commercial product nor intended to be one.

No IT security is currently implemented. Anyone with access to the network at the robot's location can take control of the robot or view video from the navigation camera.
The firmware version currently used on the OpenWRT router has known vulnerabilities. If the remote location's network was opened to the internet as described below, these vulnerabilities could allow an outside attacker to attack the remote network from the inside.
A network connection to the IP address of the robot must be allowed (Open firewall or DMZ). While this is also an obvious security risk, It can also be used to mitigate the above by letting only the operators IP address (or possibly MAC) to control the robot; Although since the connection is not encrypted, an attacker could easily determine the address needed to spoof the connection.
Access to the facilities IT infrastructure would be need to either assign a static IP address to both the robot control and video connections, or to determine the assigned (DHCP) IP addresses.
To implement this properly, a central server (most likely a 3rd location) would need to be set up to mitigate all of the above. In other words, practical control of the robot would work just like the Teleconferencing software or other IoT (Internet of Things) devices like Thermostats, etc.
In order to explain the need for IT security in commercial products further, consider this. One may not think that the robot control commands would need to be encrypted or obfuscated assuming that actual control of the robot was secure; But consider this: Someone accessing the commands used to drive the robot around a given facility would eventually be able to construct a map of that facility -- including where the robot was used frequently -- which could reveal the locations of important things inside that facility. Commanding should be encrypted and padded.
The chosen Teleconferencing software (and thus it's security/privacy) is not modified or affected by the Robot Control software. The Teleconferencing software does not utilize or connect through the vulnerable router or the robot control connection. The teleconferencing hardware/software is merely along for the ride.
It is also assumed that "good" WiFi access exists throughout the building or facility where the robot is used.

In other words: Don't implement what is described below in a secure network, or as a commercial product. Think when you build software.

User Interface:
As mentioned in the previous post, the Telepresence Robot may be located far away from the operator. This means that common remote control methods such as those used for a Radio Controlled (R/C) car or a connection over Bluetooth would not be applicable. However, basic Bluetooth code such as found here (https://create.arduino.cc/projecthub/samanfern/bluetooth-controlled-car-d5d9ca) was used as a starting point.
It was also desired to have the Robot Control universally accessible, and not require software development on multiple platforms or operating systems.

http://linuxslate.com/IMG_TPR3_Small.jpg
User Interface during use of the Telepresence Robot. Image property of Linuxslate.com

This means that the Robot Control is via a web app that resides on the robot. All one has to do is initiate a standard HTTP connection to the robot's address, and they will be presented with the interface shown above. Note that the person-to-person interface is provided by a Google Hangouts session in another window.

Implementing the user interface in this manner was more complex than originally thought. It is implemented as a ESP8266WebServer running on the ESPDuino board. The ESPDuino web server delivers an HTML page that in turn contains CGI and Javascript.

The concept of Javascript and CGI embedded in HTML, which in turn is embedded in Arduino C code was a bit much for my son to follow, and frankly a stretch for my programming skills. I'll just say that both of us got really good at reading through several levels of escape characters.

Esentially, movement is controlled by starting the respective motors on a mouse down event, and the motors are stopped on a mouse up event for each button.

An iFrame at the top contains the separate video connection, and another iFrame just below that serves as a basic instrument panel showing the main battery voltage and the strength of the wireless signal.

In the next post: The Code.

Title: Re: Telepresence Robot Build - With Code!
Post by Administrator on Dec 25^th, 2021, 1:41pm

Comments on The Code.

This is post #3 of this thread.

The code is too large to fit in a post. A public version can be downloaded here (http://linuxslate.com/files/ESP8266_NodeMCU_WIFI_robo_car_web_Public.ino). (<---Link to be added soon)

Substitute the Access Point Name and the Access Point password in lines 37 and 39.
The IP address of the Navigation Camera (Typically http://192.168.1.1:8080?action=stream) must be inserted in the HTML code in line 118.

Navigation Camera Setup:
As mentioned, the Navigation Camera is implemented via an inexpensive webcam connected to an OpenWRT mini-router. If the OpenWRT router you use is not for configured for the web cam, details can be found here (https://openwrt.org/docs/guide-user/hardware/video/webcam), and will not be further covered in this thread.

Deficiencies and Improvements:
(No special order)

As mentioned -- Security issues. See previous post.
If not fixed by another method such as a remote (3rd location) server, the issue of determining the Robot Control IP address could be mitigated by including an LCD or OLED display. In addition to the Robot Control IP address, Other information could be displayed locally, such as the battery voltage, the remote operators name, or other information to be presented to others at the remote location.
Similarly, the AP name and password are hard-coded. This should be able to be set by a "helper" at the remote site. This could be implemented by an initial point-to-point WiFi connection, or via a display and a simple possibly detachable keyboard.
There is currently no communication between the Robot Control (Code on the Arduino), and the Router that functions as a web cam. The Web cam's IP address must be hard coded into the iFrame for video display. This obviously needs to be fixed.
onmousedown and onmouseup do not support touchscreens adequately. Code needs to be added to support touchscreen devices.
Navigation via physical keyboard arrow keys would be highly desirable. Code should be added to read physical keyboard input for navigation.
More powerful motors than those supplied with the DOIT Car "Smart Tank" kit should be used.
Some sort of charging station and connection should be implemented.
It was originally desired that the Robot should provide power to the Tablet. It is easy to implement via a separate buck converter (on hand), but this has not been added as of this writing.

Join in the Discussion!
Are you building a simiar project? Using some or all of the provided code? Suggestions for improvements?
Please email "john" at this domain, and I will create an account for you. You email will not be made public.

Title: Re: Telepresence Robot Build - With Code!
Post by Administrator on Dec 29^th, 2021, 10:22pm

20 x 4 LCD Display

It has been mentioned previously in this thread that it would be desirable to have a simple display on the Telepresence Robot for information like the Robot's IP address, battery voltage, etc. It may also be useful to allow the remote user to display simple messages such as the remote user's name, or the organization the remote user represents.

To this end, I went to Sky Craft Surplus (https://skycraftsurplus.com/) in Orlando, FL., and purchased 2 LCD displays.

This post will concentrate on a 20 character x 4 line display that was inside some sort of digital audio device. The surplus device was being sold for US$ 9.95. So in addition to the display, I got the steel chassis, a small PC-like power supply, an optical drive, fan, and various other potentially useful parts.

The display is very similar to the one shown here:
http://linuxslate.com/20x4 display.jpg
20 x 4 Alpha-numeric LCD display removed from surplus equipment

I had quite a bit of difficulty finding the Datasheet for this display, or any specific information on connecting it to common Arduino modules.

So I thought I would do a simple write up on getting these boards working with Arduino.

While of course I have only tested this with the display I have, I believe what is here should work with boards marked as
EDT 20-20072, EDT 20-20072-2, EDT 20-20072-5, EW20400YMY, Winstar WH2004A, Raystar 2004A, Raystar 2004A-LLH-JSV, RC QC2004A and others.
The datasheet for the same or compatible device is here (http://linuxslate.com/RC2004A-LLH-JSV.pdf).
This is an 8 bit parallel device. All 8 lines must be used. There is (apparently) no support for 4 bit mode as with similar 2 line displays.
This pinout is similar but not the same as other similar displays. In particular, the last 2 pins (pins 15 and 16) are not used and should be left open.

Here is the pinout from the Datasheet and corresponding Arduino Pins and Functions:

Code:

Arduino LCD Function
GND 1 Ground
5V 2 5.0 VDC LCD Supply
GND    3 Operating voltage for LCD (Contrast Adjustment?)
D12 12 4 RS
GND    5 R/W
D11   11 6 E (Enable)
D2 2    7 DB0
D3 3 8 DB1
D4 4 9 DB2
D5 5 10 DB3
D6 6 11 DB4
D7 7    12 DB5
D8 8    13 DB6
D9 9    14 DB7
15 Vee - Leave unconnected
16       K - Leave unconnected

The code is equally simple. These displays work with the standard Arduino provided LiquidCrystal Library (https://www.arduino.cc/en/Reference/LiquidCrystal); however, some special characters or display functions may need changes to code to appear on the display as desired.
Here is sample code to display a childish message using all 4 lines (rows) of the display:

Code:

#include <LiquidCrystal.h>

LiquidCrystal lcd(12, 11, 2, 3, 4, 5, 6, 7, 8, 9);

void setup() {

lcd.begin(20, 4);
}

void loop() {

lcd.clear();
lcd.setCursor(0, 0);
lcd.print("You are ugly");
lcd.setCursor(0, 1);
lcd.print("and your mother");
lcd.setCursor(0, 2);
lcd.print("dresses you");
lcd.setCursor(0, 3);
lcd.print("funny.");
delay(500);
}

http://linuxslate.com/20x4 display_2.jpg
The display connected to an Arduino Nano, and displaying a childish message.

Another note on the picture:
-- Yes, I tack soldered the wires directly to the Arduino Nano. Pins and connectors would have been better, but this was just for a quick test. It will be properly connected to the pins on the ESPDuino on the Robot. The holes on the Arduino NANO can be easily cleaned, and pins soldered in.
-- This requires almost all of the digital I/O pins, but most of the pins used on the Robot currently are Analog pins. I don't anticipate a problem, but the actual design of connecting the LCD to the Robot has not been done yet.
-- Another solution would be to leave the LCD connected to the NANO, and write a simple serial or I2C - to - parallel program, but in addition to the extra hardware, this would require maintenance of 2 sets of code.

Look for an update -- and updated Robot code -- once the LCD is attached to the robot.

Title: Re: Telepresence Robot Build - With Code!
Post by Administrator on May 18^th, 2022, 12:14pm

There is an effort here at Linuxslate.com to make this project work more like a commercial product.

What does "Work more like a commercial product" mean?

There should not be any special requirements or access needed at the site where the Robot is in operation.
There should not be any special requirements or access needed at the site where the User is located.
At least "Common sense" security should be implemented. More specifically:
- Unauthorized 3rd parties should not be able to control the robot, even if they are on the same network as the Robot or the Operator.
- Unauthorized 3rd parties should not be able to view video from the robot even if they are on the same network as the Robot or the Operator.
- Unauthorized 3rd parties should not be able capture passwords or keys at any point in the communications path.
A helper at the remote location should be able to connect the Robot to their network without special access to, or administrative knowledge of the network at the Robot location.
The interface for configuring the Robot for the network (Network name and password) should be similar to registering other devices on the network.
It should not require possession of cables or other special hardware.
We are maintaining the requirement that no special software is need on the User's device. Control remains through a simple web interface. We call this the "If you can shop, you can drive" approach. This means that if the users Laptop, Tablet, or Phone can access the internet, and the have access to sites like Amazon or eBay, it should be able to control the Robot without additional network access, privileges or configuration.

Implementing the above has proven a challenge, and we are definitely moving forward in a 3 steps forward 2 steps back fashion.

In "a number of items -for- a number of items" trade, I have acquired a Raspberry Pi Zero W. This will replace *both* the ESPDuino and the OpenWRT router on the Robot.
The new method requires the use of a central (3rd point) server. As for any server configuration, the server may require special network settings. For now, we are using a newer OpenWRT Router as a small and simple server.
We currently have secure video routed from the robot to the server, and viewable by the user. However, it is not configured to meet all of the above requirements. We have learned a lot about implementing mjpg-streamer and stunnel4 on the Raspberry Pi.
As of right now, the server is making an HTTPS request for the video channel to the Raspberry Pi Since this won't work over a typical network firewall, some additional work is needed. If the Robot initiates the secure tunnel, then the Server should still be able to initiate an HTTPS request through that tunnel to the Robot requesting the video stream. I'm going to be pretty good at configuring stunnel4 by the time this is over.
There are several solutions out there for headless connection of a Raspberry Pi to a local network. We are attempting to use one that first creates a connection to a Bluetooth phone or tablet, and then utilizes an app running on the phone or tablet to allow the network settings to be entered. In this case, the tablet that is used for the Telepresence conferencing software can be used. This provides a convenient eye-level interface, and also eliminates the need for an assistant at the robot location to install software on a phone or other device. The problems we are encountering here include the fact that the required Python script is written for older versions of Python, as well as getting all of the Python extensions installed for both WiFi and Bluetooth.
The Raspberry Pi does not have any analog input pins. This means we will need additional hardware to implement the battery voltage monitor. The requirement for battery voltage will be put on hold for now, and I will order a Raspberry Pi compatible A to D converter board at some point in the future.
The Raspberry Pi does have 4 PWM compatible pins for motor control. The actual robot drive software will be ported from Arduino C to a script on the robot (Probably some Python Code).
The Javascript that provided the Driver User Interface was extracted from the Adruino C code, and works unmodified on the OpenWRT router. While the interface works, there is no GCI behind it to actually communicate with the robot. The "Meat" of the actual robot control software on both the Robot (Raspberry Pi) and the Server is still ahead of us.
I will solder header pins to the Raspberry Pi Zero W (Essentially converting it to a Raspberry Pi Zero WH) and build a harness to go from the Rasberry Pi to the existing motor control board on the Robot. The ESPDuino and it's daughter card will be removed. Since the +5VDC is provided by the daughter card, a separate Buck converter will be needed to provide +5VDC at sufficient current for the Raspberry Pi Zero and the motor control board.