Isaac Lab v1.2.0 is out!

[Dhoeller19]

👀 Overview

We leverage the new release of Isaac Sim, 4.2.0, and bring RTX-based tiled rendering, support for multi-agent environments, and introduce many bug fixes and improvements.

Additionally, we have published an example for generating rewards using an LLM based on Eureka, available here https://github.com/isaac-sim/IsaacLabEureka.

Full Changelog: v1.1.0...v1.2.0

🔆 Highlighted Features

shadowhand_cube_tiled_rendering.mp4

shadowhand_multi_agent_hand_over.mp4

deformable_teddy_pick_up.mp4

✨ New Features

• Adds RTX-based tiled rendering. This improves the overall rendering speed and quality.
• Adds the direct workflow perceptive Shadowhand Cube Repose environment Isaac-Repose-Cube-Shadow-Vision-Direct-v0 by @kellyguo11.
• Adds support for multi-agent environments with the Direct workflow, with support for MAPPO and IPPO in SKRL by @Toni-SM
• Adds the direct workflow multi-agent environments Isaac-Cart-Double-Pendulum-Direct-v0 and Isaac-Shadow-Hand-Over-Direct-v0 by @Toni-SM
• Adds throughput benchmarking scripts for the different learning workflows by @kellyguo11 in Adds throughput benchmarking scripts for different learning workflows #759
• Adds results for the benchmarks in the documentation here for different types of hardware by @kellyguo11
• Adds the direct workflow Allegro hand environment by @kellyguo11 in Adds direct workflow Allegro hand environment #709
• Adds video recording to the play scripts in RL workflows by @j3soon in Adds video recording to the play scripts in RL workflows #763
• Adds comparison tables for the supported RL libraries here by @kellyguo11
• Add APIs for deformable asset by @masoudmoghani in Add APIs for deformable asset #630
• Adds support for MJCF converter by @qqqwan in Adds support for MJCF converter #957
• Adds a function to define camera configs through intrinsic matrix by @pascal-roth in Adds function to define camera configs through intrinsic matrix #617
• Adds configurable modifiers to observation manager by @jtigue-bdai in Adds configurable modifiers to observation manager #830
• Adds the Hydra configuration system for RL training by @Dhoeller19 in Adds the Hydra configuration system for RL training #700

🔧 Improvements

• Uses PhysX accelerations for rigid body acceleration data by @Mayankm96 in Uses PhysX accelerations for rigid body acceleration data #760
• Adds documentation on the frames for asset data by @Mayankm96 in Adds documentation on the frames for asset data #742
• Renames Unitree configs in locomotion tasks to match properly by @Mayankm96 in Renames Unitree configs in locomotion tasks to match properly #714
• Adds option to set the height of the border in the TerrainGenerator by @pascal-roth in Adds option to set the height of the border in the TerrainGenerator #744
• Adds a cli arg to run_all_tests.py for testing a selected extension by @jsmith-bdai in Adds a cli arg to run_all_tests.py for testing a selected extension #753
• Decouples rigid object and articulation asset classes by @Mayankm96 in Decouples rigid object and articulation asset classes #644
• Adds performance optimizations for domain randomization by @kellyguo11 in Adds performance optimizations for domain randomization #494
• Allows having hybrid dimensional terms inside an observation group by @Mayankm96 in Allows having hybrid dimensional terms inside an observation group #772
• Adds a flag to preserve joint order inside JointActionCfg action term by @xav-nal in Adds flag to preserve joint order inside JointActionCfg action term #787
• Adds the ability to resume training from a checkpoint with rl_games by @sizsJEon in Adds the ability to resume training from a checkpoint with rl_games #797
• Adds windows configuration to VS code tasks by @johnBuffer in Adds windows configuration to VS code tasks #963
• Adapts A and D button bindings in the keyboard device by @zoctipus in Adapts A and D button bindings in the keyboard device #910
• Uses torch.einsum for quat_rotate and quat_rotate_inverse operations by @dxyy1 in Uses torch.einsum for quat_rotate and quat_rotate_inverse operations #900
• Expands on articulation test for multiple instances and devices by @jsmith-bdai in Expands on articulation test for multiple instances and devices #872
• Adds setting of environment seed at initialization by @Mayankm96 in Adds setting of environment seed at initialization #940
• Disables default viewport when headless but cameras are enabled by @kellyguo11 in Disables default viewport when headless but cameras are enabled #851
• Simplifies the return type for parse_env_cfg method by @Mayankm96 in Simplifies the return type for parse_env_cfg method #965
• Simplifies the if-elses inside the event manager apply method by @Mayankm96 in Simplify the if-elses inside the event manager apply method #948

🐛 Bug Fixes

• Fixes rendering frame delays. Rendered images now faithfully represent the latest state of the physics scene. We added the flag
  rerender_on_reset in the environment configs to toggle an additional render step when a reset happens. When activated, the images/observation always represent the latest state of the environment, but this also reduces performance.
• Fixes wrap_to_pi function in math utilities by @Mayankm96 in Fixes wrap_to_pi function in math utilities #771
• Fixes setting of pose when spawning a mesh by @masoudmoghani in Fixes setting of pose when spawning a mesh #692
• Fixes caching of the terrain using the terrain generator by @Mayankm96 in Fixes caching of the terrain using the terrain generator #757
• Fixes running train scripts when rsl_rl is not installed by @Dhoeller19 in Fixes running train scripts when rsl_rl is not installed #784, Fixes importing rsl_rl configs #789
• Adds flag to recompute inertia when randomizing the mass of a rigid body by @Mayankm96 in Adds flag to recompute inertia when randomizing the mass of a rigid body #989
• Fixes support for classmethod when defining a configclass by @Mayankm96 in Fixes support for classmethod when defining a configclass #901
• Fixes Sb3VecEnvWrapper to clear buffer on reset by @EricJin2002 in Fixes Sb3VecEnvWrapper to clear buffer on reset #974
• Fixes venv and conda pip installation on windows by @kellyguo11 in Fixes venv and conda pip installation on windows #970
• Sets native livestream extensions to Isaac Sim 4.1-4.0 defaults by @jtigue-bdai in Sets native livestream extensions to Isaac Sim 4.1-4.0 defaults #954
• Defaults the gym video recorder fps to match episode decimation by @ozhanozen in Defaults the gym video recorder fps to match episode decimation #894
• Fixes the event manager's apply method by @kellyguo11 in Fixes the event manager's apply method #936
• Updates camera docs with world units and introduces new test for intrinsics by @pascal-roth in Updates camera docs with world units and introduces new test for intrinsics #886
• Adds the ability to resume training from a checkpoint with rl_games by @sizsJEon in Adds the ability to resume training from a checkpoint with rl_games #797

💔 Breaking Changes

• Simplifies device setting in SimulationCfg and AppLauncher by @Dhoeller19 in Simplifies device setting in SimulationCfg and AppLauncher #696
• Fixes conflict in teleop-device command line argument in scripts by @Dhoeller19 in Fixes conflict in teleop-device command line argument in scripts #791
• Converts container.sh into Python utilities by @hhansen-bdai in f565c33
• Drops support for TiledCamera for Isaac Sim 4.1

✈️ Migration Guide

Setting the simulation device into the simulation context

Details

Previously, changing the simulation device to CPU required users to set other simulation parameters (such as disabling GPU physics and GPU pipelines). This made setting up the device appear complex. We now simplify the checks for device directly inside the simulation context, so users only need to specify the device through the configuration object.

Before:

sim_utils.SimulationCfg(device="cpu", use_gpu_pipeline=False, dt=0.01, physx=sim_utils.PhysxCfg(use_gpu=False))

Now:

sim_utils.SimulationCfg(device="cpu", dt=0.01, physx=sim_utils.PhysxCfg())

Setting the simulation device from CLI

Details

Previously, users could specify the device through the command line argument --device_id. However, this made it ambiguous when users wanted to set the device to CPU. Thus, instead of the device ID, users need to specify the device explicitly through the argument --device. The valid options for the device name are:

• "cpu": runs simulation on CPU
• "cuda": runs simulation on GPU with device ID at default index
• "cuda:N": runs simulation on GPU with device ID at N. For instance, "cuda:0" will use device at index "0".

Due to the above change, the command line interaction with some of the scripts has changed.

Before:

./isaaclab.sh -p source/standalone/workflows/sb3/train.py --task Isaac-Cartpole-v0 --headless --cpu

Now:

./isaaclab.sh -p source/standalone/workflows/sb3/train.py --task Isaac-Cartpole-v0 --headless --device cpu

Renaming of teleoperation device CLI in standalone scripts

Details

Since --device is now an argument provided by the AppLauncher, it conflicted with the command-line argument used for specifying the teleoperation-device in some of the standalone scripts. Thus, to fix this conflict, the teleoperation-device now needs to be specified through --teleop_device argument.

Before:

./isaaclab.sh -p source/standalone/environments/teleoperation/teleop_se3_agent.py --task Isaac-Lift-Cube-Franka-IK-Rel-v0 --num_envs 1 --device keyboard

Now:

./isaaclab.sh -p source/standalone/environments/teleoperation/teleop_se3_agent.py --task Isaac-Lift-Cube-Franka-IK-Rel-v0 --num_envs 1 --teleop_device keyboard

Using Python-version of container utility script

Details

The prior container.sh became quite complex as it had many different use cases in one script. For instance, building a docker image for "base" or "ros2", as well as cluster deployment. As more users wanted to have the flexibility to overlay their own docker settings, maintaining this bash script became cumbersome. Hence, we migrated its features into a Python script in this release. Additionally, we split the cluster-related utilities into their own script inside the docker/cluster directory.

We still maintain backward compatibility for container.sh. Internally, it calls the Python script container.py. We request users to use the Python script directly.

Before:

./docker/container.sh start

Now:

./docker/container.py start

Using separate directories for logging videos in RL workflows

Details

Previously, users could record videos during the RL training by specifying the --video flag to the train.py script. The videos would be saved inside the videos directory in the corresponding log directory of the run.

Since many users requested to also be able to record videos while inferencing the policy, recording videos have also been added to the play.py script. Since changing the prefix of the video file names is not possible, the videos from the train and play scripts are saved inside the videos/train and videos/play directories, respectively.

Drops support for the tiled camera with Isaac Sim 4.1

Details

Various fixes have been made to the tiled camera rendering pipeline in Isaac Sim 4.2. This made supporting the tiled camera with Isaac Sim 4.1 difficult. Hence, for the best experience, we advice switching to Isaac Sim 4.2 with this release of Isaac Lab.

🤗 New Contributors

• @xav-nal made their first contribution in Adds flag to preserve joint order inside JointActionCfg action term #787
• @sizsJEon made their first contribution in Adds the ability to resume training from a checkpoint with rl_games #797
• @jtigue-bdai made their first contribution in Adds configurable modifiers to observation manager #830
• @StrainFlow made their first contribution in Updates installation verfication for Windows #835
• @mpgussert made their first contribution in Adds images for the tutorial docs #827
• @Symars made their first contribution in Fixes initialization of normal velocity command generator #898
• @martinmatak made their first contribution in Fixes comment in run_articulation.py #876
• @bearpaw made their first contribution in Fixes folder typo in the create_scene.rst tutorial #945
• @dxyy1 made their first contribution in Uses torch.einsum for quat_rotate and quat_rotate_inverse operations #900
• @qqqwan made their first contribution in Adds support for MJCF converter #957
• @johnBuffer made their first contribution in Adds windows configuration to VS code tasks #963
• @EricJin2002 made their first contribution in Fixes Sb3VecEnvWrapper to clear buffer on reset #974
• @iamnambiar made their first contribution in Fixes typo in the Environment ID of the unitree robots in the environments.rst file #986