A problem about timestep in Deepbots

[speedhawk]

Hi, Sorry for trouble you again and hope you are well. I got a problem in creating the environments of reinforcement learning training.
I just remember that you have proposed me before a thought that with deepbots we can construct a similar world as many cases of gym environments, in which the the agent moves by not continuous driving but actions moving one space in a different direction (such as cliff walking which is a typical grid world). Thanks to your enlightment, I am now trying to create a grid world with arena map with black and white plaid over it.
Now, my main thought is to create a space including actions by which the robot agent can move a constant distance, just like the same distance to one grid step ahead to different direction (forward, back, left, right). However, I got a problem with timestep.
As you know, the timestep is an attribute of Supervisor class, which is the super class of even all of relative subclasses. When the iteration starts, the function step(), which is the function of Supervisor class, will be called by the step() function in subclass and automatically call the timestep attribute from Supervisor class so that the 3D simulating animation can push forward along the timeline. The trouble is, if I create a space with 'grid moving style' actions (such as moving for a grid distance, or rotate a given angle in one step of iteration), the function also use timestep. For example, this is the original step function in Supervisor class:

    def step(self, action):
        self.handle_emitter(action)
        if super(Supervisor, self).step(self.timestep) == -1:
            exit()

        return (
            self.get_observations(),
            self.get_reward(action),
            self.is_done(),
            self.get_info(),
        )

And based on some resources online, one method to implement 'grid moving style' action can be achieved like this:

def turn_left(angle):
    l_speed = 1.0
    r_speed = -1.0
    leftMotor.setVelocity(l_speed)
    rightMotor.setVelocity(r_speed)
    
    while robot.step(self.timestep) != -1:
        if robot.getFromDef('agent').getField('rotation').getSFRotation[3] >= 1.5708:
            leftMotor.setVelocity(0.0)
            rightMotor.setVelocity(0.0)
        break

Here, timestep cannot be called simultanerously by two processes. Therefore, do you have any ideas to solve the problem? Or any better solutions for create grid world in Webots? Many thanks for any help or advice!

[tsampazk]

Hey @speedhawk! From what i understand from your description, the way you are approaching the problem is not ideal if i am not mistaken. Instead of implementing movement through the motors, I would suggest to use the supervisor to move the robot to the required grid position instantly, or rotate it instantly by setting its position and rotation. This would probably require some groundwork for the grid so you would know exactly at which world coordinates each grid square is.

I am not sure whether this solution fits your problem, hope it gives you some ideas. Let me know if i can offer any additional help.

[speedhawk]

Hi @tsampazk ! Really graceful for your reply with some advice! Do you have any idea for the implementation of grid-world construction groundworks (such as occupancy grid mapping)? Is it able to be achieved by deepbots or requiring other plugins or resources?

Instead of implementing movement through the motors, I would suggest to use the supervisor to move the robot to the required grid position instantly, or rotate it instantly by setting its position and rotation.

Additionally, this is my first time to hear that the robot agent can move like this in Webots without motor LoL. It is really interesting. Could you please give me some enlighting or advice about how to implement it? Many thanks!

[tsampazk]

You're welcome @speedhawk!

Additionally, this is my first time to hear that the robot agent can move like this in Webots without motor LoL. It is really interesting. Could you please give me some enlighting or advice about how to implement it? Many thanks!

You can see an example of setting the rotation of a node here and an example of setting the position of node here. The full documentation is here for further reading.

Do you have any idea for the implementation of grid-world construction groundworks (such as occupancy grid mapping)? Is it able to be achieved by deepbots or requiring other plugins or resources?

Implementing an occupancy grid or anything similar is up to you unfortunately as it is not really related to deepbots itself. It should be fairly straightforward. On another note, there are many webots examples to check out the usage of various sensors and devices. I find this one really interesting, but it's not exactly an occupancy grid (it might be an easy and nice way to visualize it though). If you feel your question is answered feel free to mark it.

[speedhawk]

Hi, I am really sorry for replying so late! Firstly I really appreciate you advice in such a great detail. After re-thought about the implementation and reviewed the document of the parts of Webot step() function, I desire to achieved, the main problem may appear as the indivisibility of control steps. Briefly, although one control step consists of several simulator steps, they cannot be re-configured or designed by self-definition. That means several basic actions cannot be edited together for a new independent action.
Nevertheless, I still feel interested on Webots real-time simulation. This problem happened during my last project which by now has been cancelled. After I finish my current work, I would like to try more about self-defined actions in Webots with other method!