diff --git a/detector_plugin.md b/detector_plugin.md
new file mode 100644
index 0000000..1f87d1a
--- /dev/null
+++ b/detector_plugin.md
@@ -0,0 +1,30 @@
+# Detector Plugin Architecture
+
+The `object_detection` package follows a plugin-based architecture for allowing the use of different object detection models. These can be loaded at launch time by setting the [`detector_type`](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/2c8152d6a5ae5b5f6e3541648ae97d9ba79ac6a9/object_detection/config/params.yaml#L7P)
+param in the `config/params.yaml` file of the package. Currently the package supports the following detectors out of the box:
+ * YOLOv5
+ * YOLOv8
+ * RetinaNET
+ * EdgeDET
+
+The package provides a [`DetectorBase`](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/detector_plugin_architecture/object_detection/object_detection/DetectorBase.py) class which is an abstract class.
+It uses the python's in-built `abc.ABC` to define the abstract class and `abc.abstractmethod` decorator to define the blueprint for different class methods that the plugin should implement.
+All the detector plugin classes are stored in the [Detectors](https://github.com/atom-robotics-lab/ros-perception-pipeline/tree/detector_plugin_architecture/object_detection/object_detection/Detectors) directory of the
+`object_detection` package.
+
+## Creating Your own Detector Plugin
+To create your own detector plugin, follow the steps below:
+ * Create a file for your Detector class inside the [Detectors](https://github.com/atom-robotics-lab/ros-perception-pipeline/tree/detector_plugin_architecture/object_detection/object_detection/Detectors) directory.
+ * The file should import the `DetectorBase` class from the [`DetectorBase.py`](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/detector_plugin_architecture/object_detection/object_detection/DetectorBase.py) module. You can create a class for your Detector in this file which should inherit the `DetectorBase` abstract class.
+
+ > **Note:** The name of the file and class should be the same. This is required in order to allow the [`object_detection node`](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/ObjectDetection.py#L79) to load the module and its class using the value of the `detector_type` in the param.
+ * Inside the class constructor, make sure to call the constructor of the `DetectorBase` class using `super().__init__()`. This initializes an empty `predictions` list that would be used to store the predictions later. (explained below)
+
+ * After this, the Detector plugin class needs to implement the abstract methods listed below. These are defined in the `DetectorBase` class and provide a signature for the function's implementations. These abstract methods act as a standard API between the Detector plugins and the ObjectDetection node. The plugins only need to match the function signature (parameter and return types) to allow ObjectDetection node to use them.
+ * [`build_model()`](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/DetectorBase.py#L21): It takes 2 strings as parameters: `model_dir_path` and `weight_file_name`. `model_dir_path` is the path which contains the model file and class file. The `weight_file_name` is the name of the weights file (like `.onxx` in case of Yolov5 models). This function should return no parameters and is used by the ObjectDetection node [here](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/ObjectDetection.py#L83). While creating the plugin, you need not worry about the parameters as they are provided by the node through the ROS 2 params. You just need to use their values inside the functions according to your Detector's requirements.
+
+ * [`load_classes()`](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/DetectorBase.py#L25): This is similar to the `build_model()` function. It should load the classes file as per the requirement using the provided `model_dir_path` parameter.
+
+ * [`get_predictions()`](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/DetectorBase.py#L29): This function is [used by the ObjectDetection node](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/ObjectDetection.py#L92) in the subscriber callback to get the predictions for each frame passed. This function should take an opencv image (which is essentially a numpy array) as a parameter and return a list of dictionaries that contain the predictions. This function can implement any kind of checks, formatting, preprocessing, etc. on the frame (see [this](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/Detectors/YOLOv5.py#L131) for example). It only strictly needs to follow the signature described by the abstract method definition in `DetectorBase`. To create the predictions list, the function should call the [`create_predictions_list()`](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/DetectorBase.py#L10) function from the `DetectorBase` class like [this](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/071c63aa4bc71913d4bf5a4c7f9b4fd03b136338/object_detection/object_detection/Detectors/YOLOv5.py#L144). Using the `create_predictions_list()` function is necessary as it arranges the prediction data in a standard format that the `ObjectDetection` node expects.
+
+> **Note:** For reference you can through the [YOLOv5 Plugin class](https://github.com/atom-robotics-lab/ros-perception-pipeline/blob/detector_plugin_architecture/object_detection/object_detection/Detectors/YOLOv5.py#L131) and how it implements all the abstract methods.
\ No newline at end of file
diff --git a/object_detection/config/object_detection.yaml b/object_detection/config/params.yaml
similarity index 54%
rename from object_detection/config/object_detection.yaml
rename to object_detection/config/params.yaml
index 18e9843..4cc92d2 100644
--- a/object_detection/config/object_detection.yaml
+++ b/object_detection/config/params.yaml
@@ -5,5 +5,7 @@ object_detection:
output_img_topic: object_detection/img
model_params:
detector_type: YOLOv8
- model_dir_path: model/yolov8
- weight_file_name: version5.pt
\ No newline at end of file
+ model_dir_path: /home/jayesh/ws/src/ros-perception-pipeline/object_detection/models
+ weight_file_name: version5.pt
+ confidence_threshold : 0.7
+ show_fps : 1
diff --git a/object_detection/efficient.txt b/object_detection/efficient.txt
new file mode 100644
index 0000000..f7e99e0
--- /dev/null
+++ b/object_detection/efficient.txt
@@ -0,0 +1,2 @@
+keras_applications>=1.0.7,<=1.0.8
+scikit-image
diff --git a/object_detection/launch/object_detection.launch.py b/object_detection/launch/object_detection.launch.py
new file mode 100644
index 0000000..19a9af1
--- /dev/null
+++ b/object_detection/launch/object_detection.launch.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2018 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+
+from ament_index_python.packages import get_package_share_directory
+
+from launch import LaunchDescription
+from launch.actions import IncludeLaunchDescription, DeclareLaunchArgument
+from launch.substitutions import LaunchConfiguration
+from launch.launch_description_sources import PythonLaunchDescriptionSource
+from launch_ros.actions import Node
+
+
+def generate_launch_description():
+ pkg_object_detection = get_package_share_directory("object_detection")
+
+ params = os.path.join(
+ pkg_object_detection,
+ 'config',
+ 'params.yaml'
+ )
+
+ node=Node(
+ package = 'object_detection',
+ name = 'object_detection',
+ executable = 'ObjectDetection',
+ parameters = [params],
+ emulate_tty = True,
+ output="screen"
+ )
+
+
+ return LaunchDescription([node])
diff --git a/object_detection/models/classes.txt b/object_detection/models/classes.txt
new file mode 100644
index 0000000..c9d3452
--- /dev/null
+++ b/object_detection/models/classes.txt
@@ -0,0 +1,4 @@
+ESP
+Earpod Case
+Rasberry PI
+Mouse
\ No newline at end of file
diff --git a/object_detection/models/version5.pt b/object_detection/models/version5.pt
new file mode 100644
index 0000000..4261438
Binary files /dev/null and b/object_detection/models/version5.pt differ
diff --git a/object_detection/object_detection/DetectorBase.py b/object_detection/object_detection/DetectorBase.py
new file mode 100644
index 0000000..801b5cc
--- /dev/null
+++ b/object_detection/object_detection/DetectorBase.py
@@ -0,0 +1,30 @@
+from abc import ABC, abstractmethod
+import numpy as np
+
+
+class DetectorBase(ABC):
+
+ def __init__(self) -> None:
+ self.predictions = []
+
+ def create_predictions_list(self, class_ids, confidences, boxes):
+ for i in range(len(class_ids)):
+ obj_dict = {
+ "class_id": class_ids[i],
+ "confidence": confidences[i],
+ "box": boxes[i]
+ }
+
+ self.predictions.append(obj_dict)
+
+ @abstractmethod
+ def build_model(self, model_dir_path: str, weight_file_name: str) -> None:
+ pass
+
+ @abstractmethod
+ def load_classes(self, model_dir_path: str) -> None:
+ pass
+
+ @abstractmethod
+ def get_predictions(self, cv_image: np.ndarray) -> list[dict]:
+ pass
\ No newline at end of file
diff --git a/object_detection/object_detection/Detectors/RetinaNet.py b/object_detection/object_detection/Detectors/RetinaNet.py
index 0b62dc4..8a54122 100755
--- a/object_detection/object_detection/Detectors/RetinaNet.py
+++ b/object_detection/object_detection/Detectors/RetinaNet.py
@@ -1,6 +1,5 @@
#!/usr/bin/env python3
-
from tensorflow import keras
from keras_retinanet import models
from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image
@@ -15,96 +14,25 @@
class RetinaNet:
- def __init__(self, model_dir_path, weight_file_name, conf_threshold = 0.7, score_threshold = 0.4, nms_threshold = 0.25):
-
- self.frame_count = 0
- self.total_frames = 0
- self.fps = -1
- self.start = time.time_ns()
-
- self.predictions = []
- self.conf_threshold = conf_threshold
+ def __init__(self, model_dir_path, weight_file_name, conf_threshold = 0.7,
+ score_threshold = 0.4, nms_threshold = 0.25, is_cuda = 0, show_fps = 1):
self.model_dir_path = model_dir_path
self.weight_file_name = weight_file_name
-
- self.labels_to_names = self.load_classes()
- self.build_model()
-
- def build_model(self) :
-
- try :
- self.model_path = os.path.join(self.model_dir_path, self.weight_file_name)
- self.model = models.load_model(self.model_path, backbone_name='resnet50')
-
- except :
- raise Exception("Error loading given model from path: {}. Maybe the file doesn't exist?".format(self.model_path))
-
-
- def load_classes(self):
- self.class_list = []
- with open(self.model_dir_path + "/classes.txt", "r") as f:
- self.class_list = [cname.strip() for cname in f.readlines()]
- return self.class_list
-
- def create_predictions_list(self, class_ids, confidences, boxes):
- for i in range(len(class_ids)):
- obj_dict = {
- "class_id": class_ids[i],
- "confidence": confidences[i],
- "box": boxes[i]
- }
-
- self.predictions.append(obj_dict)
-
-
- def get_predictions(self, cv_image):
- if cv_image is None:
- # TODO: show warning message (different color, maybe)
- return None,None
-
- else :
-
- # copy to draw on
- self.frame = cv_image
- # preprocess image for network
- input = preprocess_image(cv_image)
- input, scale = resize_image(input)
-
- self.frame_count += 1#!/usr/bin/env python3
-
-
-from tensorflow import keras
-from keras_retinanet import models
-from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image
-from keras_retinanet.utils.visualization import draw_box, draw_caption
-from keras_retinanet.utils.colors import label_color
-import matplotlib.pyplot as plt
-import cv2
-import os
-import numpy as np
-import time
-import matplotlib.pyplot as plt
-
-
-class RetinaNet:
- def __init__(self, model_dir_path, weight_file_name, conf_threshold = 0.7, score_threshold = 0.4, nms_threshold = 0.25):
-
- self.frame_count = 0
- self.total_frames = 0
- self.fps = -1
- self.start = time.time_ns()
- self.frame = None
-
self.predictions = []
self.conf_threshold = conf_threshold
-
- self.model_dir_path = model_dir_path
- self.weight_file_name = weight_file_name
+ self.show_fps = show_fps
+ self.is_cuda = is_cuda
+
+ if self.show_fps :
+ self.frame_count = 0
+ self.total_frames = 0
+ self.fps = -1
+ self.start = time.time_ns()
self.labels_to_names = self.load_classes()
- self.build_model()
+ self.build_model()
def build_model(self) :
@@ -177,15 +105,16 @@ def get_predictions(self, cv_image):
#print(self.labels_to_names[label])
draw_caption(self.frame, b, caption)
- if self.frame_count >= 30:
- self.end = time.time_ns()
- self.fps = 1000000000 * self.frame_count / (self.end - self.start)
- self.frame_count = 0
- self.start = time.time_ns()
+ if self.show_fps :
+ if self.frame_count >= 30:
+ self.end = time.time_ns()
+ self.fps = 1000000000 * self.frame_count / (self.end - self.start)
+ self.frame_count = 0
+ self.start = time.time_ns()
- if self.fps > 0:
- self.fps_label = "FPS: %.2f" % self.fps
- cv2.putText(self.frame, self.fps_label, (10, 25), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
+ if self.fps > 0:
+ self.fps_label = "FPS: %.2f" % self.fps
+ cv2.putText(self.frame, self.fps_label, (10, 25), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)
return (self.predictions, self.frame)
diff --git a/object_detection/object_detection/Detectors/YOLOv5.py b/object_detection/object_detection/Detectors/YOLOv5.py
index 1f22088..571f3d0 100644
--- a/object_detection/object_detection/Detectors/YOLOv5.py
+++ b/object_detection/object_detection/Detectors/YOLOv5.py
@@ -3,45 +3,34 @@
import cv2
import numpy as np
-class YOLOv5:
- def __init__(self, model_dir_path, weight_file_name, conf_threshold = 0.7, score_threshold = 0.4, nms_threshold = 0.25, is_cuda = 0):
+from ..DetectorBase import DetectorBase
- # calculate fps, TODO: create a boolean to enable/diable show_fps
- self.frame_count = 0
- self.total_frames = 0
- self.fps = -1
- self.start = time.time_ns()
+
+class YOLOv5(DetectorBase):
+ def __init__(self, conf_threshold = 0.7, score_threshold = 0.4, nms_threshold = 0.25, is_cuda = 0):
+
+ super().__init__()
# opencv img input
self.frame = None
self.net = None
- self.predictions = []
- self.model_dir_path = model_dir_path
- self.weight_file_name = weight_file_name
self.INPUT_WIDTH = 640
self.INPUT_HEIGHT = 640
self.CONFIDENCE_THRESHOLD = conf_threshold
- self.bb_colors = [(255, 255, 0), (0, 255, 0), (0, 255, 255), (255, 0, 0)]
- self.is_cuda = is_cuda
+ self.is_cuda = is_cuda
-
- # load & build the given model
- self.build_model(self.is_cuda)
-
- # load classes (object labels)
- self.load_classes()
# load model and prepare its backend to either run on GPU or CPU, see if it can be added in constructor
- def build_model(self, is_cuda):
- model_path = os.path.join(self.model_dir_path, self.weight_file_name)
+ def build_model(self, model_dir_path, weight_file_name):
+ model_path = os.path.join(model_dir_path, weight_file_name)
try:
self.net = cv2.dnn.readNet(model_path)
except:
raise Exception("Error loading given model from path: {}. Maybe the file doesn't exist?".format(model_path))
- if is_cuda:
+ if self.is_cuda:
print("is_cuda was set to True. Attempting to use CUDA")
self.net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
self.net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)
@@ -50,6 +39,16 @@ def build_model(self, is_cuda):
self.net.setPreferableBackend(cv2.dnn.DNN_BACKEND_OPENCV)
self.net.setPreferableTarget(cv2.dnn.DNN_TARGET_CPU)
+
+ # load classes.txt that contains mapping of model with labels
+ # TODO: add try/except to raise exception that tells the use to check the name if it is classes.txt
+ def load_classes(self, model_dir_path):
+ self.class_list = []
+ with open(model_dir_path + "/classes.txt", "r") as f:
+ self.class_list = [cname.strip() for cname in f.readlines()]
+ return self.class_list
+
+
def detect(self, image):
# convert image to 640x640
blob = cv2.dnn.blobFromImage(image, 1/255.0, (self.INPUT_WIDTH, self.INPUT_HEIGHT), swapRB=True, crop=False)
@@ -57,13 +56,6 @@ def detect(self, image):
preds = self.net.forward()
return preds
- # load classes.txt that contains mapping of model with labels
- # TODO: add try/except to raise exception that tells the use to check the name if it is classes.txt
- def load_classes(self):
- self.class_list = []
- with open(self.model_dir_path + "/classes.txt", "r") as f:
- self.class_list = [cname.strip() for cname in f.readlines()]
- return self.class_list
# extract bounding box, class IDs and confidences of detected objects
# YOLOv5 returns a 3D tensor of dimension 25200*(5 + n_classes)
@@ -126,6 +118,7 @@ def wrap_detection(self, input_image, output_data):
return result_class_ids, result_confidences, result_boxes
+
# makes image square with dimension max(h, w)
def format_yolov5(self):
row, col, _ = self.frame.shape
@@ -134,18 +127,11 @@ def format_yolov5(self):
result[0:row, 0:col] = self.frame
return result
- def create_predictions_list(self, class_ids, confidences, boxes):
- for i in range(len(class_ids)):
- obj_dict = {
- "class_id": class_ids[i],
- "confidence": confidences[i],
- "box": boxes[i]
- }
-
- self.predictions.append(obj_dict)
-
def get_predictions(self, cv_image):
+ #Clear list
+ self.predictions = []
+
if cv_image is None:
# TODO: show warning message (different color, maybe)
return None,None
@@ -158,35 +144,8 @@ def get_predictions(self, cv_image):
outs = self.detect(inputImage)
class_ids, confidences, boxes = self.wrap_detection(inputImage, outs[0])
- self.create_predictions_list(class_ids, confidences, boxes)
-
- self.frame_count += 1
- self.total_frames += 1
+ super().create_predictions_list(class_ids, confidences, boxes)
- print("Detected ids: ", class_ids)
+ print("Detected ids: ", class_ids)
- # draw bounding box and add label
- for (classid, confidence, box) in zip(class_ids, confidences, boxes):
- color = self.bb_colors[int(classid) % len(self.bb_colors)]
- cv2.rectangle(self.frame, box, color, 2)
- cv2.rectangle(self.frame, (box[0], box[1] - 20), (box[0] + box[2], box[1]), color, -1)
- try :
- cv2.putText(self.frame, self.class_list[classid], (box[0], box[1] - 10), cv2.FONT_HERSHEY_SIMPLEX, .5, (0,0,0))
- except :
- pass
-
- # fps
- if self.frame_count >= 30:
- self.end = time.time_ns()
- self.fps = 1000000000 * self.frame_count / (self.end - self.start)
- self.frame_count = 0
- self.start = time.time_ns()
-
- if self.fps > 0:
- self.fps_label = "FPS: %.2f" % self.fps
- cv2.putText(self.frame, self.fps_label, (10, 25), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
-
- return (self.predictions, self.frame)
-
-
-
+ return self.predictions
\ No newline at end of file
diff --git a/object_detection/object_detection/Detectors/YOLOv8.py b/object_detection/object_detection/Detectors/YOLOv8.py
index fe7f59a..341741d 100755
--- a/object_detection/object_detection/Detectors/YOLOv8.py
+++ b/object_detection/object_detection/Detectors/YOLOv8.py
@@ -3,55 +3,38 @@
import os
import time
-class YOLOv8:
- def __init__(self, model_dir_path, weight_file_name, conf_threshold = 0.7, score_threshold = 0.4, nms_threshold = 0.25):
-
- #FPS
- self.frame_count = 0
- self.total_frames = 0
- self.fps = -1
- self.start = time.time_ns()
- self.frame = None
-
+from ..DetectorBase import DetectorBase
- self.model_dir_path = model_dir_path
- self.weight_file_name = weight_file_name
+class YOLOv8(DetectorBase):
+ def __init__(self, conf_threshold = 0.7, is_cuda = 1):
+
+ super().__init__()
self.conf_threshold = conf_threshold
- self.predictions = []
- self.build_model()
- self.load_classes()
+ self.is_cuda = is_cuda
-
- def build_model(self) :
+ def build_model(self, model_dir_path, weight_file_name) :
try :
- model_path = os.path.join(self.model_dir_path, self.weight_file_name)
- self.model = YOLO(model_path)
+ model_path = os.path.join(model_dir_path, weight_file_name)
+ if self.is_cuda:
+ self.model = YOLO(model_path)
+ self.model.to('cuda')
except :
raise Exception("Error loading given model from path: {}. Maybe the file doesn't exist?".format(model_path))
- def load_classes(self):
+
+ def load_classes(self, model_dir_path):
self.class_list = []
- with open(self.model_dir_path + "/classes.txt", "r") as f:
+ with open(model_dir_path + "/classes.txt", "r") as f:
self.class_list = [cname.strip() for cname in f.readlines()]
return self.class_list
- # create list of dictionary containing predictions
- def create_predictions_list(self, class_ids, confidences, boxes):
-
- for i in range(len(class_ids)):
- obj_dict = {
- "class_id": class_ids[i],
- "confidence": confidences[i],
- "box": boxes[i]
- }
- self.predictions.append(obj_dict)
def get_predictions(self, cv_image):
@@ -61,35 +44,20 @@ def get_predictions(self, cv_image):
else :
self.frame = cv_image
- self.frame_count += 1
- self.total_frames += 1
class_id = []
confidence = []
- bb = []
- result = self.model.predict(self.frame, conf = self.conf_threshold) # Perform object detection on image
+ boxes = []
+ result = self.model.predict(self.frame, conf = self.conf_threshold, verbose = False) # Perform object detection on image
row = result[0].boxes
for box in row:
- class_id.append(box.cls)
- confidence.append(box.conf)
- bb.append(box.xyxy)
-
- self.create_predictions_list(class_id,confidence,bb)
- result = self.model.predict(self.frame, conf = self.conf_threshold)
- output_frame = result[0].plot() # Frame with bounding boxes
-
- print("frame_count : ", self.frame_count)
-
- if self.frame_count >= 30 :
- self.end = time.time_ns()
- self.fps = 1000000000 * self.frame_count / (self.end - self.start)
- self.frame_count = 0
- self.start = time.time_ns()
-
- if self.fps > 0:
- self.fps_label = "FPS: %.2f" % self.fps
- cv2.putText(output_frame, self.fps_label, (10, 25), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
-
- return self.predictions, output_frame
-
\ No newline at end of file
+ class_id.append(box.cls.numpy().tolist())
+ confidence.append(box.conf.numpy().tolist())
+ boxes.append(box.xyxy.numpy().tolist())
+
+ super().create_predictions_list(class_id,confidence,boxes)
+
+
+ return self.predictions
+
diff --git a/object_detection/object_detection/ObjectDetection.py b/object_detection/object_detection/ObjectDetection.py
index 549726a..2d38677 100644
--- a/object_detection/object_detection/ObjectDetection.py
+++ b/object_detection/object_detection/ObjectDetection.py
@@ -1,80 +1,110 @@
#! /usr/bin/env python3
+import os
+import importlib
+
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
-from vision_msgs.msg import BoundingBox2D
-
-
-from .Detectors import YOLOv5, YOLOv8, EfficientDet, RetinaNet
+#from vision_msgs.msg import BoundingBox2D
from cv_bridge import CvBridge
+import cv2
class ObjectDetection(Node):
def __init__(self):
super().__init__('object_detection')
+ # create an empty list that will hold the names of all available detector
+ self.available_detectors = []
+
+ # fill available_detectors with the detectors from Detectors dir
+ self.discover_detectors()
+
self.declare_parameters(
namespace='',
parameters=[
- ('input_img_topic', 'color_camera/image_raw'),
- ('output_bb_topic', 'object_detection/img_bb'),
- ('output_img_topic', 'object_detection/img'),
- ('model_params.detector_type', 'YOLOv5'),
- ('model_params.model_dir_path', 'model/yolov5'),
- ('model_params.weight_file_name', "auto_final.onnx")
+
+ ('input_img_topic', ""),
+ ('output_bb_topic', ""),
+ ('output_img_topic', ""),
+ ('model_params.detector_type', ""),
+ ('model_params.model_dir_path', ""),
+ ('model_params.weight_file_name', ""),
+ ('model_params.confidence_threshold', 0.7),
+ ('model_params.show_fps', 1),
]
)
+ # node params
self.input_img_topic = self.get_parameter('input_img_topic').value
self.output_bb_topic = self.get_parameter('output_bb_topic').value
self.output_img_topic = self.get_parameter('output_img_topic').value
+
+ # model params
self.detector_type = self.get_parameter('model_params.detector_type').value
self.model_dir_path = self.get_parameter('model_params.model_dir_path').value
self.weight_file_name = self.get_parameter('model_params.weight_file_name').value
-
-
- if self.detector_type == "YOLOv5" :
- print("Using detector : {}".format(self.detector_type))
- self.detector = YOLOv5.YOLOv5(self.model_dir_path, self.weight_file_name)
-
- elif self.detector_type == "YOLOv8" :
- print("Using detector : {}".format(self.detector_type))
- self.detector = YOLOv8.YOLOv8(self.model_dir_path, self.weight_file_name)
-
- elif self.detector_type == "RetinaNet" :
- print("Using detector : {}".format(self.detector_type))
- self.detector = RetinaNet.RetinaNet(self.model_dir_path, self.weight_file_name)
-
- elif self.detector_type == "EfficientDet" :
- print("Using detector : {}".format(self.detector_type))
- self.detector = EfficientDet.EfficientDet(self.model_dir_path, self.weight_file_name)
-
- else :
- print("The detector type : {} is not supported".format(self.detector_type))
-
+ self.confidence_threshold = self.get_parameter('model_params.confidence_threshold').value
+ self.show_fps = self.get_parameter('model_params.show_fps').value
+
+ # raise an exception if specified detector was not found
+ if self.detector_type not in self.available_detectors:
+ raise ModuleNotFoundError(self.detector_type + " Detector specified in config was not found. " +
+ "Check the Detectors dir for available detectors.")
+ else:
+ self.load_detector()
+
+
self.img_pub = self.create_publisher(Image, self.output_img_topic, 10)
self.bb_pub = None
self.img_sub = self.create_subscription(Image, self.input_img_topic, self.detection_cb, 10)
self.bridge = CvBridge()
+
+ def discover_detectors(self):
+ curr_dir = os.path.dirname(__file__)
+ dir_contents = os.listdir(curr_dir + "/Detectors")
+
+ for entity in dir_contents:
+ if entity.endswith('.py'):
+ self.available_detectors.append(entity[:-3])
+
+ self.available_detectors.remove('__init__')
+
+
+ def load_detector(self):
+ detector_mod = importlib.import_module(".Detectors." + self.detector_type, "object_detection")
+ detector_class = getattr(detector_mod, self.detector_type)
+ self.detector = detector_class()
+
+ self.detector.build_model(self.model_dir_path, self.weight_file_name)
+ self.detector.load_classes(self.model_dir_path)
+
+ print("Your detector : {} has been loaded !".format(self.detector_type))
+
+
def detection_cb(self, img_msg):
- print("detection_cb")
- input = self.bridge.imgmsg_to_cv2(img_msg, "bgr8")
+ cv_image = self.bridge.imgmsg_to_cv2(img_msg, "bgr8")
- predictions, frame = self.detector.get_predictions(cv_image = input)
+ predictions = self.detector.get_predictions(cv_image=cv_image)
if predictions == None :
print("Image input from topic : {} is empty".format(self.input_img_topic))
else :
- output = self.bridge.cv2_to_imgmsg(frame, "bgr8")
- self.img_pub.publish(output)
-
- print(predictions)
+ for prediction in predictions:
+ left, top, width, height = prediction['box']
+ right = left + width
+ bottom = top + height
+
+ #Draw the bounding box
+ cv_image = cv2.rectangle(cv_image,(left,top),(right, bottom),(0,255,0),1)
+ output = self.bridge.cv2_to_imgmsg(cv_image, "bgr8")
+ self.img_pub.publish(output)
def main():
rclpy.init()
diff --git a/object_detection/retina.txt b/object_detection/retina.txt
new file mode 100644
index 0000000..be3448e
--- /dev/null
+++ b/object_detection/retina.txt
@@ -0,0 +1,8 @@
+keras-retinanet==1.0.0
+matplotlib==3.5.4
+numpy==1.25.0
+opencv-python==4.7.0.72
+pandas==2.0.3
+pillow==9.5.0
+tensorflow==2.12.0
+tensorflow-hub==0.13.0
diff --git a/object_detection/setup.py b/object_detection/setup.py
index 3f0c5b4..6cec5fe 100644
--- a/object_detection/setup.py
+++ b/object_detection/setup.py
@@ -13,6 +13,8 @@
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
(os.path.join('share', package_name, 'config'), glob('config/*.yaml')),
+ (os.path.join('share', package_name, 'launch'), glob('launch/*.launch.py')),
+
],
install_requires=['setuptools'],
zip_safe=True,
diff --git a/object_detection/yoloV8.txt b/object_detection/yoloV8.txt
new file mode 100644
index 0000000..cf77e48
--- /dev/null
+++ b/object_detection/yoloV8.txt
@@ -0,0 +1,14 @@
+matplotlib>=3.2.2
+opencv-python>=4.6.0
+pillow>=7.1.2
+pyyaml>=5.3.1
+requests>=2.23.0
+scipy>=1.4.1
+torch>=1.7.0
+torchvision>=0.8.1
+tqdm>=4.64.0
+pandas>=1.1.4
+seaborn>=0.11.0
+psutil
+py-cpuinfo
+ultralytics
diff --git a/object_detection/yolov5.txt b/object_detection/yolov5.txt
new file mode 100644
index 0000000..2c96974
--- /dev/null
+++ b/object_detection/yolov5.txt
@@ -0,0 +1,49 @@
+# YOLOv5 requirements
+# Usage: pip install -r requirements.txt
+
+# Base ------------------------------------------------------------------------
+gitpython>=3.1.30
+matplotlib>=3.3
+numpy>=1.22.2
+opencv-python>=4.1.1
+Pillow>=7.1.2
+psutil # system resources
+PyYAML>=5.3.1
+requests>=2.23.0
+scipy>=1.4.1
+thop>=0.1.1 # FLOPs computation
+torch>=1.7.0 # see https://pytorch.org/get-started/locally (recommended)
+torchvision>=0.8.1
+tqdm>=4.64.0
+ultralytics>=8.0.145
+# protobuf<=3.20.1 # https://github.com/ultralytics/yolov5/issues/8012
+
+# Logging ---------------------------------------------------------------------
+# tensorboard>=2.4.1
+# clearml>=1.2.0
+# comet
+
+# Plotting --------------------------------------------------------------------
+pandas>=1.1.4
+seaborn>=0.11.0
+
+# Export ----------------------------------------------------------------------
+# coremltools>=6.0 # CoreML export
+# onnx>=1.10.0 # ONNX export
+# onnx-simplifier>=0.4.1 # ONNX simplifier
+# nvidia-pyindex # TensorRT export
+# nvidia-tensorrt # TensorRT export
+# scikit-learn<=1.1.2 # CoreML quantization
+# tensorflow>=2.4.0 # TF exports (-cpu, -aarch64, -macos)
+# tensorflowjs>=3.9.0 # TF.js export
+# openvino-dev>=2023.0 # OpenVINO export
+
+# Deploy ----------------------------------------------------------------------
+setuptools>=65.5.1 # Snyk vulnerability fix
+# tritonclient[all]~=2.24.0
+
+# Extras ----------------------------------------------------------------------
+# ipython # interactive notebook
+# mss # screenshots
+# albumentations>=1.0.3
+# pycocotools>=2.0.6 # COCO mAP
diff --git a/perception_bringup/launch/playground.launch.py b/perception_bringup/launch/playground.launch.py
index 41a22aa..83868b0 100644
--- a/perception_bringup/launch/playground.launch.py
+++ b/perception_bringup/launch/playground.launch.py
@@ -27,7 +27,7 @@
def generate_launch_description():
pkg_perception_bringup = get_package_share_directory("perception_bringup")
- pkg_ros_gz_sim = get_package_share_directory("ros_gz_sim")
+ #pkg_ros_gz_sim = get_package_share_directory("ros_gz_sim")
world_name = "playground"
@@ -37,10 +37,18 @@ def generate_launch_description():
world_sdf = pkg_perception_bringup + "/worlds/" + world_name + ".sdf"
- gz_sim = IncludeLaunchDescription(
+ '''gz_sim = IncludeLaunchDescription(
PythonLaunchDescriptionSource(
os.path.join(pkg_ros_gz_sim, 'launch', 'gz_sim.launch.py')),
- )
+ )'''
+
+ gz_sim_share = get_package_share_directory("ros_gz_sim")
+ gz_sim = IncludeLaunchDescription(
+ PythonLaunchDescriptionSource(os.path.join(gz_sim_share, "launch", "gz_sim.launch.py")),
+ launch_arguments={
+ "gz_args" : world_sdf
+ }.items()
+ )
parameter_bridge = Node(package="ros_gz_bridge", executable="parameter_bridge",
parameters = [
diff --git a/perception_bringup/models/adhesive/model.sdf b/perception_bringup/models/adhesive/model.sdf
index 38fd2ed..1a6ca88 100644
--- a/perception_bringup/models/adhesive/model.sdf
+++ b/perception_bringup/models/adhesive/model.sdf
@@ -28,9 +28,8 @@
0 0.5 0.5 1
- 0 0 0 1
- 0 0 0 1
- 0 0 0 1
+ 0 0.5 0.5 1
+ 0 0.5 0.5 1
diff --git a/perception_bringup/worlds/playground.sdf b/perception_bringup/worlds/playground.sdf
index c5cc417..99df154 100644
--- a/perception_bringup/worlds/playground.sdf
+++ b/perception_bringup/worlds/playground.sdf
@@ -1,13 +1,13 @@
-
+
-
+
3D View
false
docked
-
+
ogre2
scene
0.4 0.4 0.4
@@ -17,87 +17,87 @@
-
+
floating
5
5
false
-
+
-
+
false
5
5
floating
false
-
+
-
+
false
5
5
floating
false
-
+
-
+
false
5
5
floating
false
-
+
-
+
false
5
5
floating
false
-
+
-
+
false
5
5
floating
false
-
+
-
+
false
5
5
floating
false
-
+
-
+
false
5
5
floating
false
-
+
-
+
World control
false
false
@@ -108,7 +108,7 @@
-
+
true
true
true
@@ -117,7 +117,7 @@
-
+
World stats
false
false
@@ -129,7 +129,7 @@
-
+
true
true
true
@@ -138,7 +138,7 @@
-
+
false
0
0
@@ -147,12 +147,12 @@
floating
false
#666666
-
+
-
+
false
250
0
@@ -161,12 +161,12 @@
floating
false
#666666
-
+
-
+
false
0
50
@@ -175,31 +175,31 @@
floating
false
#777777
-
+
-
+
false
docked
-
+
-
+
false
docked
-
+
-
+
false
docked
-
+
color_camera/image_raw
@@ -208,13 +208,11 @@
1
1000
-
-
-
-
-
+
+
+
+
+
ogre2
0 0 -9.8000000000000007
@@ -277,7 +275,7 @@
true
- -0.77 0.00 1.03 0 0.0 1.57
+ -0.77 0.20 1.41 0 0.37 1.57
true
0 0 0 0 0 0