Help with real robot

[thanhtruongphan]

Hi @hippo5329, see your robot tank very interesting.

now, I don't have a LD19 lidar and Driver for Robots like you, I have an A1M8, and L298. Should I buy a new Driver? (In that case, I must order from abroad. yeah It'll take a long time 😄 )

here is a sample I'm doing with

[hippo5329]

A1M8 and L298 are fine.
What is your battery and motor spec?
The wheel encoder power should be 3.3V. Do not use 5V, which will fry your controller.
You can use a esp32 instead of teensy.

[hippo5329]

L298
AEN and BEN use jumper plug shorted to high.
IN1-IN4 to esp32 pwm

[hippo5329]

Take some picture of your wiring.

esp32 power from usb of rpi.

MPU6050 power from VOUT 5V of esp32.
esp32 I2C SDA SCL need 2.2K to V33 3.3V of esp32 module.

L298 uses on-board 5V jumper plug.

You will need a 3.3V regulator (LM1117 etc) for encoder ,from L298 5V output.

[thanhtruongphan]

yeah, I sovled the problem comunicate SBC with ubuntu laptop linorobot/linorobot2#2 (comment)

now, I need to do some works with my Jeston nano devkit, (i have never used it before)
I will comeback to do with esp soon... 🥲

[hippo5329]

https://nshopvn.com/product/mach-giam-ap-3-3v-ams1117/

[hippo5329]

esp32 i/o pins assignment
motor 19 18 16 17 13 12 4 23, power 12V gnd, 2 PWM pins (INA, INB) driver
encoder 35 34 39 36 32 27 26 25, power 3.3V gnd
i2c scl 22 sda 21, power 3.3v gnd, need 2.2k pull high to 3.3V
battery adc 33, divider 12V-33k-10k-gnd (dac 25, used only during adc calibration)
uart0 txd 1 rxd 3
led 2

[hippo5329]

2WD unused pwm 13 12 4 23 encoder 32 27 26 25 pins leave unconnected.

You need 33K and 10k for battery voltage sensor. It is important to debug motor issue.

[hippo5329]

laptop --wifi -- rpi --usb-- A1M8 and esp32
vscode -- remote development ssh -- rpi -- esp32

[hippo5329]

Jetson nano SDK uses ubuntu 18.04

You can try community image,
https://forums.developer.nvidia.com/t/ubuntu-22-04-image-beta-test-for-jetson-nano/253301

[thanhtruongphan]

https://forums.developer.nvidia.com/t/ubuntu-22-04-image-beta-test-for-jetson-nano/253301

this image is not secure. It's just beta and some people say it has many problems (in the comments of this link)

instead, chatGPT says I can install Ubuntu 20.04 on Jetson, then use LXD to create a container of Ubuntu 22.04 with ROS-humble or Ubuntu 24 with ROS-jazzy, right?
Does it work ok? can you share some things with me? hehe

[thanhtruongphan]

i forgot, i still need the official supported version for the jetson nano devkit for image processing

[thanhtruongphan]

Is this possible? good evening @hippo5329
my idea is: I install the official Ubuntu with SDK support for Jetson nano dev kit as the default OS (ubuntu 18.04 as you said), I do a project on autonomous vehicles with image processing, I use LXD to install ROS2 humble on Ubuntu 22.04, to operate the autonomous vehicle, and the image processing feature is still used on the default OS to take advantage of the default features of Jetson, is that okay? 🦝

[hippo5329]

You can try docker on jetson nono.

[hippo5329]

"Docker is included in JetPack, so running containers on Jetson is easy and does not require any installation. "

https://developer.nvidia.com/embedded/learn/tutorials/jetson-container
https://www.forecr.io/blogs/installation/how-to-install-and-run-docker-on-jetson-nano

https://developer.nvidia.com/embedded/l4t/r32_release_v7.1/jp_4.6.1_b110_sd_card/jeston_nano/jetson-nano-jp461-sd-card-image.zip

[thanhtruongphan]

oh, docker also new for me ...
however worth to try...huh

🥇

[hippo5329]

https://github.com/DominikN/ros2_docker_example

[hippo5329]

https://github.com/Qengineering/Jetson-Nano-image
I've tried this image before. login jetson jetson

https://github.com/jetsonhacks/bootFromUSB
I've also tried this.. Much faster than SD.

https://github.com/Qengineering/Jetson-Nano-Ubuntu-20-image
seem popular, but I don't remember if I tried this

https://kalanaratnayake.medium.com/reviving-jetson-nano-with-ubuntu-20-04-2dc1f82b18f4

[hippo5329]

You can use either docker or lxd. docker is is easier to share and more popular. There are official ros2 docker images.

https://roboticseabass.com/2023/07/09/updated-guide-docker-and-ros2/

https://docs.ros.org/en/jazzy/How-To-Guides/Run-2-nodes-in-single-or-separate-docker-containers.html

https://hub.docker.com/_/ros/

https://docs.ros.org/en/iron/How-To-Guides/Setup-ROS-2-with-VSCode-and-Docker-Container.html

https://github.com/DominikN/ros2_docker_examples

https://divyanshu-raj.medium.com/ros-2-with-docker-part-1-9060f3095811

I use lxd heavily for many years.
sudo snap install lxd --channel=latest/stable

https://documentation.ubuntu.com/lxd/en/latest/tutorial/first_steps/
https://artivis.github.io/post/2022/ros2-humble/
https://dev.to/tsadarsh/lxd-for-runing-different-ros2-versions-362d

[thanhtruongphan]

thank you! 💯

all the links are helpful for me. 👍
because Docker is popular and many guideline use Docker (ex. https://forums.developer.nvidia.com/t/ros2-and-ubuntu-22-on-jetson-nano-developer-kit/280470)
,so I'll use it for start.

[hippo5329]

You should update to Jazzy. Most humble example code will work.

[thanhtruongphan]

It's late. G9. See you.

[thanhtruongphan]

alright!~ @hippo5329

I've done with ROS2-humble on my Jetson nano devkit and test simple
**ros2 run demo_nodes_cpp talker/listener**, it's OK.

I also have learned a lot of command lines with Docker and now I realize that I just need to:

sudo docker run -it ...

sudo docker stop/start

sudo exec...

😸
that's fine.

I'll wiring up everything together, and then
please help me with esp32 tomorrow morning. 🥇
-thank you-

[thanhtruongphan]

I also bought an ESP32 and ASM1117 regulator to 3.3V for encoder.
I'll follow up your instructions above and here is my components:

• esp32​
  https://nshopvn.com/product/kit-rf-thu-phat-wifi-bluetooth-esp32/?variant=134223​
  ​
• mpu6050 (GY-521)​
  https://hshop.vn/cam-bien-6-dof-bac-tu-do-gy-521-mpu6050​
  ​
• Bestep L298​
  https://hshop.vn/mach-dieu-khien-dong-co-dc-l298-1​
  ​
• Motor with incremental hall encoder​
  https://www.thegioiic.com/bo-dong-co-giam-toc-ga25-370-12vdc-encoder-gan-banh-xe-truc-4mm-130rpm​

Ps: It has 130rpm max, ​

And 11 pulses encoder, gearbox-ratio 45:1​

So number of pulses after gearbox with quadrature encoder x4 will be : 11x45x4 = 1980 ppr​
​

• Pack 18650 pin 3s7p (12.6V output max and 9V while low batteries) ​
  -> to a regulator 12V for two motors and​

-> regulator 5V for JetsonNano​
​

• Lidar A1M8​
  ​

• Jetson nano devkit with ROS2-humble container​
  ​

[thanhtruongphan]

now, without Raspi, use Jetson instead, so esp32 will do with base-controller and Jetson like a PC, I think it's same with your robot now, right?
so I'll need some change to do with l298 and a1m8. I can't do it without you haha

[hippo5329]

And 11 pulses encoder, gearbox-ratio 45:1​

So number of pulses after gearbox with quadrature encoder x4 will be : 11x45x4 = 1980 ppr​ ​

11x45 = 495

[hippo5329]

andino looks good.

[thanhtruongphan]

yeah, right.

Okay, time to go to bed. so, see you.

[hippo5329]

It seem the vol mapping does not work.

    volumes:
      - /tmp/.X11-unix:/tmp/.X11-unix:rw
      - $HOME/.Xauthority:/root/.Xauthority
      - ../linorobot2_navigation/maps:/root/linorobot2_ws/src/linorobot2/linorobot2_navigation/maps:rw)

We will need to look into this.

[hippo5329]

You may try update the map and build again.

[hippo5329]

Take care. If you got Shingles,you should see a doctor and have special anti-virus medicine in 72 hours. This is important. In some cases, the pain may last life long. No kidding.

[thanhtruongphan]

Here is my bot now hihi

This bot move quite good.
I was surprised by the smooth, quiet and almost silent movement of the motor (probably because of the 20kHz frequency) only the Jetson and Lidar's cooling fan was a bit noisy.

And another thing is that my Bot's movement doesn't match the mapping. I guess I need to do some configure with the hardware code.

However, do you have any further suggestions?

[hippo5329]

You should run test_acc to find out the max acc and max speed. If you can find a tachometer, you should verify the rpm in test_motors.

[thanhtruongphan]

Take care. If you got Shingles,you should see a doctor and have special anti-virus medicine in 72 hours. This is important. In some cases, the pain may last life long. No kidding.

yah, yes, i heard that shingles can follow me for life. The first time was 5 months ago, I got it in the middle of my forehead, eyelids and on my right arm. It was so swollen that I couldn't open my eyes for the first few days. It was quite funny because it looked like a Buddha stung by a bee with a red dot in the middle of my forehead and swollen eyes. I got it after a long stressful day with some work, and the day before that I was traveling on an unstable bus and I had a bad headache, 2 days later I got it. It lasted about 10 days and then healed. In my hometown, people get it a lot so they have experience with traditional folk remedies. My mother used a medicine made of some special leaves + coarse salt (sea salt) crushed into a thick paste and then applied to the wound. Now I live in the city and don't have those things, but it will be fine if I maintain a healthy lifestyle.

Thanks so much for your advice. I really appreciate it. 🥇

[hippo5329]

The virus infects nerve cells. You should get anti-virus medicine.

[hippo5329]

#define MOTOR_MAX_RPM 150 // motor's max RPM
#define MAX_RPM_RATIO 0.85 // max RPM allowed for each MAX_RPM_ALLOWED = MOTOR_MAX_RPM * MAX_RPM_RATIO
#define MOTOR_OPERATING_VOLTAGE 12 // motor's operating voltage (used to calculate max RPM)
#define MOTOR_POWER_MAX_VOLTAGE 12 // max voltage of the motor's power source (used to calculate max RPM)
#define MOTOR_POWER_MEASURED_VOLTAGE 12 // current voltage reading of the power connected to the motor (used for calibration)
#define COUNTS_PER_REV1 450 // wheel1 encoder's no of ticks per rev
#define COUNTS_PER_REV2 450 // wheel2 encoder's no of ticks per rev
#define COUNTS_PER_REV3 450 // wheel3 encoder's no of ticks per rev
#define COUNTS_PER_REV4 450 // wheel4 encoder's no of ticks per rev
#define WHEEL_DIAMETER 0.0560 // wheel's diameter in meters
#define LR_WHEELS_DISTANCE 0.224 // distance between left and right wheels

[hippo5329]

You should adjust the above parameters to your.

[hippo5329]

Make sure the imu and lidar are in correct orientation.

https://github.com/hippo5329/linorobot2_hardware/wiki#test-the-robot-at-maximum-velocity-and-acceleration-with-full-load

https://github.com/hippo5329/linorobot2_hardware/wiki#configure-urdf-in-linorobot2

https://github.com/hippo5329/linorobot2_hardware/wiki#velocity-smoother---optional

You can describe your problem in more details. Or take videos.

[thanhtruongphan]

I did all your instructions, test full load, configure urdf, especially reconfig encoder-sticks. In case of mine, my motor has a incremental hall encoder 11 pulses, gear ratio 45:1, so one wheel (after gear box) will be 11* 45=495. But maybe the code use mode x2, so my encoder_ticks/per_wheel = 495 *2=990.

I've tested with these, and now it run quite good. It can navigate to goal with capacity avoid obstacle. and yah. Very interesting. haha
here is my map that I've tested.

here is an problem before I configured.

have a good day, @hippo5329 🥇

[hippo5329]

Very good.

[HaiHai-17]

@thanhtruongphan anh ơi em cũng bị tạo bản đồ giống anh, thì có cấu hình lại bộ mã hoá rồi, còn cần cấu hình thêm cái khác không anh

[hippo5329]

@HaiHai-17 Please create a new discussion and use English.

[thanhtruongphan]

as I refered above @HaiHai-17

must modify all URDF file to adapt your robot https://github.com/hippo5329/linorobot2_hardware/wiki#configure-urdf-in-linorobot2

in my case:
2wd_properties.urdf.xacro
robots/2wd.urdf.xacro

especially reconfig encoder-sticks, ... in config.h file

ps: you can run

colcon build
# then
ros2 launch linorobot2_description description.launch.py rviz:=true

to see your URDF if the URDF visuallizer extension doesn't work in VScode.

[thanhtruongphan]

https://www.youtube.com/watch?v=X4mbGX4B3Lo&ab_channel=LucasTetsuyaKuwae%28Lucas%29
hi, @hippo5329 what do you think about the following human feature like in this video?

I see they just need to put their foot in the registration area, after identifying the object to be followed, it will follow and always keep a distance from the object (a person in this case). I think this feature is very good, right? I also want to do this following human feature.

By the way, today is Vietnamese Teachers' Day, you have helped me in the past time, I wish you always strong and successful.

[hippo5329]

Sound great. Thank you.

[hippo5329]

https://automaticaddison.com/how-to-create-an-object-following-robot-ros-2-navigation/

https://docs.nav2.org/tutorials/docs/navigation2_dynamic_point_following.html

[hippo5329]

https://github.com/osrf/subt_hello_world/blob/master/posts/03_perception.md

https://github.com/Wiktor-99/human_detector/tree/humble

[thanhtruongphan]

@hippo5329 hmm, I see, I think It's hard for me now.

Instead, I have a RGB D camera named Kinect V2. I will use YOLOv5 to detect people, then use DeepSORT to tracking. Then use distance from Camera to control motors, the data is like in teleop_twist_keyboard.
I think It's easier, right?

In that case, how I can connect to my ROS2 to control the two motors?

I can do the above parts. But from the data part to connecting to my ROS2, I am confused. Help me too 😄

[hippo5329]

I will do some tests and share with you when I walk through.

[thanhtruongphan]

yah, great! 🛩️

[thanhtruongphan]

@hippo5329
as you can see, I did a program that detect person, and detect a color (fluorescent shirt) by detect HSV color. Then calculate a distance (ex. person 0.65 - 1.6 m, the distance is 1.6 m).

Then create a cmd_vel data to control motor like the teleop_twist_keyboard.

and now, I'm trying to do with ROS2 humble on my Jetson...

here is my code detect:

# works well, about 4-8 FPS, 
# INFO: YOLO v3tiny, Kinect v2 camera, and HSV for detect person with distance

import cv2
import numpy as np
import time
from collections import deque
from pylibfreenect2 import Freenect2, OpenGLPacketPipeline, FrameType, SyncMultiFrameListener

class PersonDetector:
    def __init__(self, input_width=320, input_height=320):  # Significantly reduced input size
                                                            # usually is 416x416, and can reduce to 192x192 or 160x160...
        self.input_width = input_width
        self.input_height = input_height
        self.skip_frames = 2  # Process every nth frame (change to 3, 4... if FPS so low)
        self.frame_count = 0
        
        # Initialize network
        self.net = cv2.dnn.readNet(
            "yolov3-tiny.weights",
            "yolov3-tiny.cfg"
        )
        
        # Use CUDA if available
        if cv2.cuda.getCudaEnabledDeviceCount() > 0:
            self.net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
            self.net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)
        
        self.output_layers = self.net.getUnconnectedOutLayersNames()
        self.fps_counter = deque(maxlen=10)  # Reduced size for more recent FPS calculation
        self.last_detection = None  # Store last detection results

    def detect(self, frame):
        self.frame_count += 1
        height, width = frame.shape[:2]
        
        # Skip frames to improve performance
        if self.frame_count % self.skip_frames != 0 and self.last_detection is not None:
            return self.last_detection
            
        start_time = time.time()
        
        # Create blob
        blob = cv2.dnn.blobFromImage(
            frame, 
            1/255.0, 
            (self.input_width, self.input_height), 
            swapRB=True, 
            crop=False
        )
        
        self.net.setInput(blob)
        outputs = self.net.forward(self.output_layers)
        
        boxes = []
        confidences = []
        
        # Process detections
        for output in outputs:
            for detection in output:
                scores = detection[5:]
                class_id = np.argmax(scores)
                confidence = scores[class_id]
                
                if confidence > 0.4 and class_id == 0:  # Person class
                    center_x = int(detection[0] * width)
                    center_y = int(detection[1] * height)
                    w = int(detection[2] * width)
                    h = int(detection[3] * height)
                    
                    x = max(0, int(center_x - w/2))
                    y = max(0, int(center_y - h/2))
                    
                    boxes.append([x, y, w, h])
                    confidences.append(float(confidence))
        
        indices = cv2.dnn.NMSBoxes(boxes, confidences, 0.4, 0.3)
        
        process_time = time.time() - start_time
        self.fps_counter.append(1 / process_time if process_time > 0 else 0)
        current_fps = sum(self.fps_counter) / len(self.fps_counter)
        
        self.last_detection = (boxes, confidences, indices, current_fps)
        return self.last_detection

def calculate_distance(depth_image, x, y, w, h):
    """Optimized distance calculation with smaller kernel"""
    # Get center point only
    center_x = x + w // 2
    center_y = y + h // 2
    
    # Ensure coordinates are within bounds
    if (center_x >= depth_image.shape[1] or center_y >= depth_image.shape[0] or
        center_x < 0 or center_y < 0):
        return None
    
    # Use a small 3x3 kernel around the center point
    x_start = max(0, center_x - 1)
    x_end = min(depth_image.shape[1], center_x + 2)
    y_start = max(0, center_y - 1)
    y_end = min(depth_image.shape[0], center_y + 2)
    
    # Get depth region
    region = depth_image[y_start:y_end, x_start:x_end]
    valid_depths = region[(region > 0) & (region < 10000)]  # Filter invalid values
    
    if len(valid_depths) > 0:
        return np.median(valid_depths) / 1000.0  # Convert to meters
    return None

def main():
    # Initialize Kinect
    freenect = Freenect2()
    pipeline = OpenGLPacketPipeline()
    device = freenect.openDefaultDevice(pipeline)
    
    listener = SyncMultiFrameListener(FrameType.Color | FrameType.Depth)
    device.setColorFrameListener(listener)
    device.setIrAndDepthFrameListener(listener)
    device.start()
    
    # Initialize detector
    detector = PersonDetector()
    
    print("Starting detection... Press 'q' to quit.")
    
    try:
        while True:
            frames = listener.waitForNewFrame()
            
            # Process color frame with reduced resolution
            color_frame = frames["color"]
            color_image = cv2.cvtColor(color_frame.asarray(np.uint8), cv2.COLOR_RGBA2BGR)
            color_image = cv2.resize(color_image, (640, 480))  # Fixed smaller resolution
            
            # Get depth frame with matching resolution
            depth_frame = frames["depth"]
            depth_image = depth_frame.asarray(np.float32)
            depth_image = cv2.resize(depth_image, (640, 480))
            
            # Run detection
            boxes, confidences, indices, fps = detector.detect(color_image)
            
            # Process detections
            if len(indices) > 0:
                for i in indices:
                    box = boxes[i]
                    x, y, w, h = box
                    confidence = confidences[i]
                    
                    # Calculate distance
                    distance = calculate_distance(depth_image, x, y, w, h)
                    
                    # Draw bounding box
                    cv2.rectangle(color_image, (x, y), (x + w, y + h), (0, 255, 0), 2)
                    
                    # Draw labels
                    label = f'Person {confidence:.2f}'
                    if distance is not None:
                        label += f' - {distance:.1f}m'
                    
                    cv2.putText(color_image, label, (x, y - 10),
                               cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 0, 0), 2)
            
            # Display FPS
            cv2.putText(color_image, f'FPS: {fps:.1f}', (10, 30),
                       cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
            
            cv2.imshow("Kinect v2 Person Detection", color_image)
            
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break
            
            listener.release(frames)
        
        device.stop()
        device.close()
        cv2.destroyAllWindows()
        
    except Exception as e:
        print(f"Error: {str(e)}")

if __name__ == "__main__":
    main()
    

and code create cmd_vel node, support by AI 👯 , and I haven't test it

#!/usr/bin/env python3

import rclpy
from rclpy.node import Node
import cv2
import numpy as np
import time
from collections import deque
from geometry_msgs.msg import Twist
from pylibfreenect2 import Freenect2, OpenGLPacketPipeline, FrameType, SyncMultiFrameListener

class PersonFollower(Node):
    def __init__(self):
        super().__init__('person_follower')
        
        # Parameters
        self.target_distance = 1.0  # Target distance in meters
        self.max_linear_speed = 0.5  # Maximum forward/backward speed
        self.max_angular_speed = 0.5  # Maximum rotation speed
        self.distance_tolerance = 0.1  # Acceptable distance variation
        
        # ROS2 publishers
        self.cmd_vel_pub = self.create_publisher(Twist, '/cmd_vel', 10)
        
        # Kinect setup
        self.setup_kinect()
        
        # Person detector
        self.detector = self.initialize_detector()
        
    def initialize_detector(self):
        return PersonDetector()
    
    def setup_kinect(self):
        freenect = Freenect2()
        pipeline = OpenGLPacketPipeline()
        self.device = freenect.openDefaultDevice(pipeline)
        
        self.listener = SyncMultiFrameListener(FrameType.Color | FrameType.Depth)
        self.device.setColorFrameListener(self.listener)
        self.device.setIrAndDepthFrameListener(self.listener)
        self.device.start()
    
    def calculate_distance(self, depth_image, x, y, w, h):
        center_x = x + w // 2
        center_y = y + h // 2
        
        if (center_x >= depth_image.shape[1] or center_y >= depth_image.shape[0] or
            center_x < 0 or center_y < 0):
            return None
        
        x_start = max(0, center_x - 1)
        x_end = min(depth_image.shape[1], center_x + 2)
        y_start = max(0, center_y - 1)
        y_end = min(depth_image.shape[0], center_y + 2)
        
        region = depth_image[y_start:y_end, x_start:x_end]
        valid_depths = region[(region > 0) & (region < 10000)]
        
        if len(valid_depths) > 0:
            return np.median(valid_depths) / 1000.0
        return None
    
    def calculate_speed_commands(self, distance, box_center, image_width):
        twist = Twist()
        
        if distance is not None:
            distance_error = distance - self.target_distance
            
            if abs(distance_error) > self.distance_tolerance:
                linear_speed = np.clip(distance_error, -self.max_linear_speed, self.max_linear_speed)
                twist.linear.x = linear_speed
            
            image_center = image_width / 2
            angular_error = (box_center[0] - image_center) / image_width
            twist.angular.z = np.clip(angular_error, -self.max_angular_speed, self.max_angular_speed)
        
        return twist
    
    def run(self):
        while rclpy.ok():
            try:
                frames = self.listener.waitForNewFrame()
                
                color_frame = frames["color"]
                color_image = cv2.cvtColor(color_frame.asarray(np.uint8), cv2.COLOR_RGBA2BGR)
                color_image = cv2.resize(color_image, (640, 480))
                
                depth_frame = frames["depth"]
                depth_image = depth_frame.asarray(np.float32)
                depth_image = cv2.resize(depth_image, (640, 480))
                
                boxes, confidences, indices, fps = self.detector.detect(color_image)
                
                if len(indices) > 0:
                    i = indices[0]
                    box = boxes[i]
                    x, y, w, h = box
                    
                    box_center = (x + w/2, y + h/2)
                    distance = self.calculate_distance(depth_image, x, y, w, h)
                    
                    if distance is not None:
                        twist = self.calculate_speed_commands(distance, box_center, color_image.shape[1])
                        self.cmd_vel_pub.publish(twist)
                
                self.listener.release(frames)
                
            except Exception as e:
                self.get_logger().error(f"Error in person following: {str(e)}")
            
            rclpy.spin_once(self)

class PersonDetector:
    def __init__(self, input_width=320, input_height=320):  # Significantly reduced input size
                                                            # usually is 416x416, and can reduce to 192x192 or 160x160...
        self.input_width = input_width
        self.input_height = input_height
        self.skip_frames = 2  # Process every nth frame (change to 3, 4... if FPS so low)
        self.frame_count = 0
        
        # Initialize network
        self.net = cv2.dnn.readNet(
            "yolov3-tiny.weights",
            "yolov3-tiny.cfg"
        )
        
        # Use CUDA if available
        if cv2.cuda.getCudaEnabledDeviceCount() > 0:
            self.net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
            self.net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)
        
        self.output_layers = self.net.getUnconnectedOutLayersNames()
        self.fps_counter = deque(maxlen=10)  # Reduced size for more recent FPS calculation
        self.last_detection = None  # Store last detection results

    def detect(self, frame):
        self.frame_count += 1
        height, width = frame.shape[:2]
        
        # Skip frames to improve performance
        if self.frame_count % self.skip_frames != 0 and self.last_detection is not None:
            return self.last_detection
            
        start_time = time.time()
        
        # Create blob
        blob = cv2.dnn.blobFromImage(
            frame, 
            1/255.0, 
            (self.input_width, self.input_height), 
            swapRB=True, 
            crop=False
        )
        
        self.net.setInput(blob)
        outputs = self.net.forward(self.output_layers)
        
        boxes = []
        confidences = []
        
        # Process detections
        for output in outputs:
            for detection in output:
                scores = detection[5:]
                class_id = np.argmax(scores)
                confidence = scores[class_id]
                
                if confidence > 0.4 and class_id == 0:  # Person class
                    center_x = int(detection[0] * width)
                    center_y = int(detection[1] * height)
                    w = int(detection[2] * width)
                    h = int(detection[3] * height)
                    
                    x = max(0, int(center_x - w/2))
                    y = max(0, int(center_y - h/2))
                    
                    boxes.append([x, y, w, h])
                    confidences.append(float(confidence))
        
        indices = cv2.dnn.NMSBoxes(boxes, confidences, 0.4, 0.3)
        
        process_time = time.time() - start_time
        self.fps_counter.append(1 / process_time if process_time > 0 else 0)
        current_fps = sum(self.fps_counter) / len(self.fps_counter)
        
        self.last_detection = (boxes, confidences, indices, current_fps)
        return self.last_detection

def main(args=None):
    rclpy.init(args=args)
    follower = PersonFollower()
    
    try:
        follower.run()
    except KeyboardInterrupt:
        pass
    finally:
        follower.destroy_node()
        rclpy.shutdown()

if __name__ == '__main__':
    main()
    

[hippo5329]

Nice.

[thanhtruongphan]

But as you known, My Jetson nano is on Ubuntu 20.04, and the robot we made is on ROS2 humble, I still can't add my code detect person to my robot.

GPT said that we can communicate humble with foxy, so I tried.

First,
I installed ROS2 on my Jetson local, yah, its ROS2 foxy. Then I created a node, and then copied that detect code to node, and ran, but it's didn't work, too. So weird... Haha

Second,
I tried a simple talker/listener between foxy and humble, yah, It didn't work 🥲

[hippo5329]

The RMW broadcast does not work between docker and the host. So you will need to choose either one, but not both.

1. Use docker to run person detection and ros2 humble.
2. OR, run person detection and ros2 foxy on host.

To install foxy on host, https://github.com/hippo5329/linorobot2_hardware/wiki#install-linorobot2-on-ubuntu-2004-and-foxy

[thanhtruongphan]

Use the same process as jazzy and replace jazzy with foxy

Oh, you mean, we can make my robot with foxy and not need docker humble?

Then why do we make it with docker humble, I forgot... Haha

[thanhtruongphan]

I'm trying with foxy now, yah, hope it works well 🙏

[hippo5329]

It is because you wanted humble. :)

[thanhtruongphan]

haha how stupid I am... 👯

however, thanks to that I know how to use docker container. It's so cool. 👍

[hippo5329]

I remember the ubuntu 22.04 jetson docker image has some nvidia updates. You can also try person detection inside docker.

[thanhtruongphan]

yah, I'll try

[thanhtruongphan]

omg,

[hippo5329]

micro-ros is not connected. check pico.

[thanhtruongphan]

so weird, does the code in pico disappear after a while?

[hippo5329]

No, it should not. Check if /dev/ttyACM0 exist. Check lsusb , lsusb -t.

[thanhtruongphan]

@hippo5329 I thought like you. It works now, but I had to reupload firmware to pico, 🐳

[thanhtruongphan]

@hippo5329
I've made a simple PCB, and work with new BTS7960 motor driver.

wiring same as the works I did before. But my Pico doesn't work. It doesn't show up a /ttyACM0.

I thought it was dead but when I unplugged the Pico from my PCB, it worked again. 🥲

do you have any ideas... help me, pls.

ah, I also use this motor for a bigger robot https://vi.aliexpress.com/i/4000945809249.html?gatewayAdapt=glo2vnm

[thanhtruongphan]

Or is the encoder on this motor using 5V instead of 3.3V?

[hippo5329]

You should connect the VCC pin on the motor to 3.3V from LM1117-3.3. Do not use 5V.

[thanhtruongphan]

I will fix it step by step...

[hippo5329]

We need to consider,

1. power sequence
2. pico 3.3v tolerance
3. mpu6050 moudle has internal 3.3V regulator

battery 12v -> 5V step down power module to rpi USB-C (power, or pin 2,4 of 40 pins header), and 5V of BTS7960

rpi USB -> pico VBUS (*5V)-> MPU6050 VCC(*5V), and Vin(*5V) of LM1117-3.3, Vout to VCC(*3.3V) of motor encoder

Do not connect 5V from step down power module to pico or MPU6050. They are different powers.

Do not connect LM1117-3.3 output to mpu6050 or pico.

Gnd matches the same rules above.

2.2K I2C SDA SCL pull up to pico 3V3

[thanhtruongphan]

Yah great @hippo5329
I retested all and you're right. I shouldn't connect VCC of MPU with 3.3v of ASM1117. And there must have been a short circuit somewhere (maybe because I soldered too hot), so when I used the old breadboard, it had no errors.

This is the first time I do with handmade PCB, so maybe It has many problems about when I create it. 😂

I will create a new one more beautiful and it should works well 🥲

[thanhtruongphan]

And with new bigger motor, I should modify PID values,
Because It has high overshoot. When receiving cmd_vel, it jerks then stops for 1 second before continuing.

[hippo5329]

Do not tune the PID yet. You should adjust velocity smoother to lower acc.

https://github.com/hippo5329/linorobot2_hardware/wiki#velocity-smoother---optional

And you should add the battery sensor, it is more important for bigger robots.