feat: improving manipulation demo

[jmatejcz]

Purpose

Improve performance of manipulation demo

Proposed Changes

• examples/manipulation-demo-streamlit.py now launches everything - simulation, robo stack and UI. No need for second terminal.
• Changes to embodiment prompt to improve performance
• Added new scene configs
• Now all configs that are available in the demo contains only - red cubes, carrots and apples
  Yellow cubes, blue cubes, tomatoes and corn were removed because our vision model do not recognize colors so these were easy to confuse
• Fixed a bug in connector where unique id was created during import time, which resulted in all connector nodes having the same name
• Fixed description of arguments in MoveObjectFromToToolInput
• Updated command in docs and added example prompts
• Using Vacuum Gripper which performs better in our demo (switched branch in demos.repos)

Warning

For now we have binary with vacuum gripper only for jazzy and do not have them on aws. But these will be added, for now you need to download it manually: https://robotecai.sharepoint.com/:u:/g/EemJ_CqFGd5JqFVYxJhFDx0BQigipS6uJuui940h2Catjw?e=BwMqs1
After that you will need to rebuild with colcon and resource env

Warning

If you already have rai-manipulaiton-demo cloned, VCS wont pull changes automatically. You need to either pull manually or remove the repo and clone again.

Notes and Doubts

I didn't test it extensively on rai-perception refactor. It was developed and tested mainly when rai-openset-vision existed, so the tools were slightly different. But they are very similar so i' m betting that it works fine.

in experimental setup reset arm was added straight to agent after every tool call, so the arm is not stuck in weird positions. In production we obviously can't leave it like that. Reset arm tool can be added, but it will introduce more complexity to demo.

I tried making adding launch methods to common file here a4f8495, but it seems like python package import do not work in launch files so I removed it. If you know if this can be done, please let me know

If now one streamlit file launches everything, do we need the rest of the example mani files?

Testing

Run manipulation demo, it should perform better

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.36%. Comparing base (8700550) to head (592547b).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #747      +/-   ##
==========================================
+ Coverage   65.34%   65.36%   +0.02%     
==========================================
  Files          78       78              
  Lines        3388     3390       +2     
==========================================
+ Hits         2214     2216       +2     
  Misses       1174     1174              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Juliaj]

What do you think about these ?

Suggested change

	x: float = Field(description="The x coordinate of the point to move to")
	y: float = Field(description="The y coordinate of the point to move to")
	z: float = Field(description="The z coordinate of the point to move to")
	x: float = Field(description="The x coordinate of the point to move from")
	y: float = Field(description="The y coordinate of the point to move from")
	z: float = Field(description="The z coordinate of the point to move from")
	x: float = Field(description="The x coordinate of the source point (pick-up location) in the manipulator frame")
	y: float = Field(description="The y coordinate of the source point (pick-up location) in the manipulator frame")
	z: float = Field(description="The z coordinate of the source point (pick-up location) in the manipulator frame")
	x1: float = Field(description="The x coordinate of the destination point (drop-off location) in the manipulator frame")
	y1: float = Field(description="The y coordinate of the destination point (drop-off location) in the manipulator frame")
	z1: float = Field(description="The z coordinate of the destination point (drop-off location) in the manipulator frame")

The thought about adding "in the manipulator frame" to the coordinate descriptions is to clarify the required coordinate system and help the LLM agents chain tools correctly. So when GetObjectPositionsTool returns positions in the target/manipulator frame, the agent can use them directly with MoveObjectFromToTool without confusion.

[jmatejcz]

i didn't encounter problems where llm confuses 1st point with 2nd point, but it looks like a good change

[Juliaj]

Suggested change

	# Ensure the object has the correct _prefix field
	# Set the ROS2 message type namespace

[jmatejcz]

deleted 437f25d

[Juliaj]

In theory, this is not needed because the _prefix is already set in BaseGeometryModel. Were you seeing runtime failures ?

[jmatejcz]

I asked @boczekbartek , but it was long ago and he do not remember why he added this, but we agreed that it is probably not needed as _prefix has default value ( as you mentioned), so i deleted that line 437f25d

[Juliaj]

Suggested change

	"Before starting the task, by looking at the image, determine if the task can be completed, of yes - proceed without asking. Otherwise report.",
	"Before starting the task, by looking at the image, determine if the task can be completed. If yes - proceed without asking. Otherwise report.",

[jmatejcz]

applied 62e5985

[Juliaj]

Suggested change

	"Cubes height is 5cm. Remember to adjust z coordinate while stacking.",
	"Cubes are 5cm tall. When stacking, add 5cm to the z coordinate for each additional cube.",

[jmatejcz]

applied 62e5985

[Juliaj]

Suggested change

	"Try to consider the typical size og objects when placing them (width, height, length)",
	"Try to consider the typical size of objects when placing them (width, height, length)",

[jmatejcz]

applied 62e5985

[Juliaj]

Suggested change

	"If you repeat the task, make sure to run detection again, to check objects positions",
	"If you repeat the task, make sure to run detection again to check object positions",

[jmatejcz]

applied 62e5985

[Juliaj]

Suggested change

	"Coordinate system and reachable space: x - front to back (positive is forward, [0.3, 0.5]), y - left to right (positive is right, [-0.5, 0.5]), z - up to down (positive is up)."
	"Coordinate system: x-axis (front to back, positive is forward, reachable range [0.3, 0.5] m), y-axis (left to right, positive is right, reachable range [-0.5, 0.5] m), z-axis (vertical, positive is up)."

[jmatejcz]

applied 62e5985

[Juliaj]

I tried making adding launch methods to common file here a4f8495, but it seems like python package import do not work in launch files so I removed it.

What error were you seeing? My guess is that the launch file was run in a different Python context, so examples/ might not be on sys.path .

in the launch file, you could try following

import sys
from pathlib import Path

# Add examples directory to path
examples_path = Path(__file__).parent.parent / "examples"
sys.path.insert(0, str(examples_path))

from manipulation_common import launch_description  # or whatever function name

[jmatejcz]

I tried making adding launch methods to common file here a4f8495, but it seems like python package import do not work in launch files so I removed it.

What error were you seeing? My guess is that the launch file was run in a different Python context, so examples/ might not be on sys.path .

in the launch file, you could try following

import sys
from pathlib import Path

# Add examples directory to path
examples_path = Path(__file__).parent.parent / "examples"
sys.path.insert(0, str(examples_path))

from manipulation_common import launch_description  # or whatever function name

I get import error. I tried these approaches with adding path, but it seems like when you use ros2 launch python modules are not recognized properly

[jmatejcz]

PERFORMANCE TESTS

I tried simple prompts that require 1 tool call and some more advanced , for example these (which are in docs now):

• "Place every apple on top of the cube"
• "Stack all cubes into a tower"
• "Arrange objects in a line"
• "Move cubes to the left side of the table"

Problems that we encountered on old version:

• Objects were falling from the gripper mid manipulation
• Positions of cubes were detected wrongly , tomatoes with apples and corn was just hard to identify
• Arm was bumping into other objects
• Arm was not behaving properly on the ends of table
• Agent struggled with understanding objects sizes (bad alignments and stacking, not leaving any margin)

on the above prompts demo had ~50% acc.

We found out that vision model do not recognize colors -> so I left in demo only configs with red cubes, removed tomatoes and corn.

We decided that this should be showcase of agentic Ai, not a robot capability, so we replaced real gripper with vacuum one ( which is not adequate to what you see in simulation). We removed collisions of gripper's fingers. @knicked did all that in https://github.com/RobotecAI/rai-manipulation-demo.

We changed the embodiment prompt so that agent understand the sizes of objects, leaves margins, etc. We also added the arm range in prompt and made configs so that the objects do not exceed the range.

The results:

• Easy prompts that require 1 tool call -> almost 100% acc
• Harder ones (these above) -> 80% , after follow app prompts from user even 95%

[Juliaj]

Please consider making this as a dict with values as layout descriptions (i.e. three red cubes) . The keys are too cryptic for the user to select as a layout.

[Juliaj]

Adding a unit test for this is worthwhile. The unit tests enabled a safe code refactoring.