Index out of range during emulation

[Jackbaude]

Start StarryNet.
rm: cannot remove '/home/baude/git/StarryNet/./starlink-5-5-550-53-grid-LeastDelay/*.txt': No such file or directory
Reset docker environment for constellation emulation ...
Remove legacy containers.
['constellation-test\r\n']
['7079db422c3c\r\n', '7886ffe15304\r\n', '5fb5d8e4245d\r\n', '35d3cef30cb1\r\n', '36ed52b392e3\r\n', 'f51f755569b9\r\n', 'f2cef938db7b\r\n', 'ab0caa4d6c89\r\n', '6dc4e9b054d4\r\n', '84f2792313d5\r\n', '98d110f38424\r\n', 'a9ac27621578\r\n', '1f56d1bfe18e\r\n', 'ec1ede8a4436\r\n', '33f49166eeff\r\n', '8a9078868e0a\r\n', '6d8779148e5f\r\n', '6e066b002e5a\r\n', 'e9aa76bc5644\r\n', 'e4b3db6d7fc0\r\n', 'bbfe3d8fce77\r\n', 'b3a24fc9c9ac\r\n', '63fc5427a3e1\r\n', '4b3cb242977e\r\n', '3d47b3c66332\r\n', 'cab3eb0be7a0\r\n', 'babcda903424\r\n']
Remove legacy emulated ISLs.
Creating new containers...
Rename all containers ...
Constellation initialization done. 27 have been created.
Create Links.
Run in link init thread.
Initializing links ...
Link initialization done.
Copy bird configuration file to each container and run routing process.
Initializing routing ...
Routing initialized!
Bird routing in all containers are running.
node_distance (km): 8148.249253913003
neighbors_index: [2, 5, 6, 21]
GSes are: [26]
LLA: 53.003651,-2.890712,542.611248

IP: []
Emulation in No.2 second.
Emulation in No.3 second.
Exception in thread Thread-10 (sn_ping):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/baude/git/StarryNet/starrynet/sn_utils.py", line 732, in sn_ping
    " ping " + str(des_IP[0][:-1]) + " -c 4 -i 0.01 ")
IndexError: list index out of range
Emulation in No.4 second.
Emulation in No.5 second.
Randomly setting damaged links...

iperf server success
Exception in thread Thread-11 (sn_perf):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/baude/git/StarryNet/starrynet/sn_utils.py", line 762, in sn_perf
    " iperf3 -c " + str(des_IP[0][:-1]) + " -t 5 ")
IndexError: list index out of range
Damage done.

Emulation in No.6 second.
Emulation in No.7 second.
Emulation in No.8 second.
Emulation in No.9 second.
Emulation in No.10 second.
Delay updating done.

Recovering damaged links...

Link recover done.

Emulation in No.11 second.
Emulation in No.12 second.
Emulation in No.13 second.
Emulation in No.14 second.
Emulation in No.15 second.
Emulation in No.16 second.
Emulation in No.17 second.
Emulation in No.18 second.
Emulation in No.19 second.
Emulation in No.20 second.
Delay updating done.

Exception in thread Thread-9:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/baude/git/StarryNet/starrynet/sn_utils.py", line 438, in run
    sn_sr(self.sr_src[index_num],
  File "/home/baude/git/StarryNet/starrynet/sn_utils.py", line 705, in sn_sr
    " ip route del " + str(des_IP[0][:-3]) + "0/24")
IndexError: list index out of range
Deleting all native bridges and containers...

I have built and installed StarryNet as instructed, and I am seeing these errors when I run the example.

{
    "Name": "starlink",
    "Altitude (km)": 550,
    "Cycle (s)": 5731,
    "Inclination": 53,
    "Phase shift": 1,
    "# of orbit": 5,
    "# of satellites": 5,
    "Duration (s)": 100,
    "update_time (s)": 10,
    "satellite link bandwidth (\"X\" Gbps)": 5,
    "sat-ground bandwidth (\"X\" Gbps)": 5,
    "satellite link loss (\"X\"% )": 1,
    "sat-ground loss (\"X\"% )": 1,
    "GS number": 2,
    "antenna number": 1,
    "antenna_inclination_angle": 25,
    "remote_machine_IP": "REDACT",
    "remote_machine_username": "ubuntu",
    "remote_machine_password": "REDACT",
    "Satellite link": "grid",
    "IP version": "IPv4",
    "Intra-AS routing": "OSPF",
    "Inter-AS routing": "BGP",
    "Link policy": "LeastDelay",
    "Handover policy": "instant handover",
    "multi-machine (\"0\" for no, \"1\" for yes)": 0
}

Why is this bug happening? is this an issue with my configuration or with docker or with the code base?

[xiex386]

Looks like the problem of ifconfig command. Is ifconfig or package net-tools installed on the remote machine?

[Jackbaude]

Yes ifconfig and net-tools and iperf, and iperf3 are installed on the remote machine.

$ ifconfig --version
net-tools 2.10

I did a bit more digging but have not been able to resolve this issue. I have modified the sn_ping function in sn_utils.py to print out the commands it is going to run before it runs them.

def sn_ping(src, des, time_index, constellation_size, container_id_list,
            file_path, configuration_file_path, remote_ssh):
    print(f"container id list {container_id_list}")
    print(f"src {src} des {des}")
    **print(f"src {container_id_list[src - 1]} des {container_id_list[des - 1]}")**
    des_IP = []
    if des <= constellation_size:
        **print(f"ifconfig_output_command= docker exec -it {container_id_list[des - 1]} ifconfig | sed 's/[ \t].*//;/^\(eth0\|\)\(lo\|\)$/d'")**
        ifconfig_output = sn_remote_cmd(
            remote_ssh, "docker exec -it " + str(container_id_list[des - 1]) +
            " ifconfig | sed 's/[ \t].*//;/^\(eth0\|\)\(lo\|\)$/d'")
        **print(f"ifconfig_output {ifconfig_output}")**
        des_IP = sn_remote_cmd(
            remote_ssh, "docker exec -it " + str(container_id_list[des - 1]) +
            " ifconfig " + ifconfig_output[0][:-1] +
            "|awk -F '[ :]+' 'NR==2{print $4}'")
        print(f"des_IP {des_IP}")
    else:
        des_IP = sn_remote_cmd(
            remote_ssh, "docker exec -it " + str(container_id_list[des - 1]) +
            " ifconfig B" + str(des) +
            "-default |awk -F '[ :]+' 'NR==2{print $4}'")
        **print(f"des_IP ELSE {des_IP}")**
    ping_result = sn_remote_cmd(
        remote_ssh, "docker exec -i " + str(container_id_list[src - 1]) +
        " ping " + str(des_IP[0][:-1]) + " -c 4 -i 0.01 ")
    f = open(
        configuration_file_path + "/" + file_path + "/ping-" + str(src) + "-" +
        str(des) + "_" + str(time_index) + ".txt", "w")
    f.writelines(ping_result)
    f.close()

Modifying the sn_ping with some debug statements, this is what I see:

src 0ebfacbbfd03 des 6f205da00e9b
ifconfig_output_command= docker exec -it 6f205da00e9b ifconfig | sed 's/[       ].*//;/^\(eth0\|\)\(lo\|\)$/d'
ifconfig_output ['\r\n', '\r\n']
des_IP []

Then running these commands in my remote server this is what I see:

$ sudo docker exec -it 6f205da00e9b ifconfig | sed 's/[ \t].*//;/^\(eth0\|\)\(lo\|\)$/d'

if I run it without the regex I am getting a result

$ sudo docker exec -it 6f205da00e9b ifconfig 
eth0      Link encap:Ethernet  HWaddr 22:7c:31:bf:d7:c4  
          inet addr:172.17.0.19  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:218 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:17064 (17.0 KB)  TX bytes:13096 (13.0 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

[xiex386]

The links with names like B1-eth6 don't seem to be established.
Could you check the docker network configurations? Here is an output example.

$ docker network ls
NETWORK ID     NAME              DRIVER    SCOPE
59f0e91119e2   GSL_7-26          bridge    local
47a65ded2fa5   GSL_13-27         bridge    local
2a379334c5e6   GS_26             bridge    local
a8e1232e2bc8   GS_27             bridge    local
d11cc0d92e32   La_0-0_0-1        bridge    local
6eb419bc095c   La_0-1_0-2        bridge    local
606fdf23b898   La_0-2_0-3        bridge    local
c2f3fbc8618a   La_0-3_0-4        bridge    local
...

[Jackbaude]

I am not seeing the bridge on the remote machine

$ docker network ls
NETWORK ID     NAME              DRIVER    SCOPE
7dd94570d608   bridge            bridge    local
a031f596fddf   docker_gwbridge   bridge    local
0157ecfcc7ce   host              host      local
n7ejuky1dfot   ingress           overlay   swarm
6bc53d22dfab   none              null      local

$ sudo docker network inspect docker_gwbridge 
[
    {
        "Name": "docker_gwbridge",
        "Id": "703c27f4fc6d4d83127ea24fdc7c9c2b2fdc0ff8a1cd7168a7f7129a1781854f",
        "Created": "2025-04-10T04:04:28.694975658Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv4": true,
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "gateway_ingress-sbox",
                "EndpointID": "a36be0f58c4168f3d60d4b54e3c06c8201aa758d86693dc675bc60ce9dc37603",
                "MacAddress": "d2:f7:20:e7:e7:74",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]

remote machine docker containers

$ sudo docker ps -a
CONTAINER ID   IMAGE                    COMMAND                CREATED          STATUS          PORTS      NAMES
4025eff5700f   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_1
c4b1c02b7693   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_2
a85a912a0125   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_3
f5ffc2dca6fe   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_4
3b8ab3aba4b3   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_5
5344a663f6a4   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_6
60befa821cdc   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_7
b754d29034bd   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_8
47d56688b84b   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_9
b481e99162a2   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_10
35c004cb39b3   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_11
8e391cb51ce4   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_12
9094a7d090db   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_13
3689bc1b8350   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_14
1f1f44730c2f   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_15
ed090632d4a2   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_16
fb2a1330f70b   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_17
44d6cf7e68e1   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_18
18d19c48bf4c   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_19
c9e60a816ba0   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_20
9f01f975f972   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_21
8be043318d2f   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_22
14285e62430e   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_23
7e893e98767c   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_24
d7e6ad0c4067   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_25
af73bcdfa94e   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_26
e64f1bfa0e77   lwsen/starlab_node:1.0   "ping www.baidu.com"   19 seconds ago   Up 18 seconds   9999/tcp   ovs_container_27

this remote machine should init a docker swarm correct?

like
docker swarm init

[xiex386]

Yes, the remote machine needs to initialize a docker swarm first.

[YifanHu0102]

I also have the exact problem even after initialization of docker swarm, have you solved the issue?