#IndexError: list index out of range when running "python -m scripts.preprocess_libero --suite libero_spatial"

[Khlann]

Dear author:
I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

[Robot-K]

same issue

[ahadjawaid]

You have to run the preprocessing step where they split the dataset into validation and training set. You can do this by doing python3 scripts/split_libero_dataset.py --folder data/atm_libero --train_ratio 0.9.

[hanbit-o]

#7 (comment)
this didn't work to me.

[AlvinWen428]

Hi, I guess this problem is because the downloaded libero dataset is not correctly placed into the folder ATM/data/. Would you please check the folder structure of ATM/data/? It should be like this:

data

libero_spatial

pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate_demo.hdf5
pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate_demo.hdf5
...

libero_object
libero_goal
libero_10
libero_90

Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

[lib36157]

Hi, I guess this problem is because the downloaded libero dataset is not correctly placed into the folder ATM/data/. Would you please check the folder structure of ATM/data/? It should be like this:

data

libero_spatial

pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate_demo.hdf5
pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate_demo.hdf5
...

libero_object
libero_goal
libero_10
libero_90

Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

same error. I don't think it's the problem of data structure. Should be some other reasons.

[lib36157]

Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

just put all these data files into a folder named libero. The directory should be ATM/data/libero/data files. That works for me.

[Robot-K]

Solved it, there's a .DS_Store file under the location, use
for source_h5 in os.listdir(suite_dir):
source_h5_path = os.path.join(suite_dir, source_h5)
if 'hdf5' not in source_h5: # to pass files like DS_store
continue

[RenMing-Huang]

Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!

(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
    main()
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
    task_bert_embs_dict = get_task_bert_embs(root)
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
    task_embs = get_task_embs(cfg, task_names).cpu().numpy()
  File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
    tokens = tz(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
    return self.batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range

just put all these data files into a folder named libero. The directory should be ATM/data/libero/data files. That works for me.

thanks, it works for me!