Core: Compose: compose.yml does not build development environment

[andrewkwolek]

I tried using docker to build the development environment but found that the cloud-telemetry image does not exist. The current image in the master branch is cloud-telemetry: image: public.ecr.aws/h4c9p1o0/blueos-cloud-agent-dev.

I found an active image in AWS ECR called public.ecr.aws/blueos/bcloud-agent which allowed me to successfully pull and run the environment. However, I received an exception in cloud-telemetry about non-existent files.

Is public.ecr.aws/blueos/bcloud-agent the correct image for cloud-telemetry or does it no longer exist?

Thanks!

[patrickelectric]

Ping @voorloopnul

[voorloopnul]

Hello Andrew, you are right, the correct image is: public.ecr.aws/blueos/bcloud-agent
Can you share the error you are receiving?

[andrewkwolek]

This is the issue I am receiving. I see you made a PR so I can try that as well.

blueos-cloud-telemetry | Traceback (most recent call last):
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
blueos-cloud-telemetry | response = self._make_request(
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 495, in _make_request
blueos-cloud-telemetry | conn.request(
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 441, in request
blueos-cloud-telemetry | self.endheaders()
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/http/client.py", line 1281, in endheaders
blueos-cloud-telemetry | self._send_output(message_body, encode_chunked=encode_chunked)
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/http/client.py", line 1041, in _send_output
blueos-cloud-telemetry | self.send(msg)
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/http/client.py", line 979, in send
blueos-cloud-telemetry | self.connect()
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/transport/unixconn.py", line 26, in connect
blueos-cloud-telemetry | sock.connect(self.unix_socket)
blueos-cloud-telemetry | FileNotFoundError: [Errno 2] No such file or directory
blueos-cloud-telemetry |
blueos-cloud-telemetry | During handling of the above exception, another exception occurred:
blueos-cloud-telemetry |
blueos-cloud-telemetry | Traceback (most recent call last):
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
blueos-cloud-telemetry | resp = conn.urlopen(
blueos-cloud-telemetry | ^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen
blueos-cloud-telemetry | retries = retries.increment(
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 474, in increment
blueos-cloud-telemetry | raise reraise(type(error), error, _stacktrace)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
blueos-cloud-telemetry | raise value.with_traceback(tb)
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
blueos-cloud-telemetry | response = self._make_request(
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 495, in _make_request
blueos-cloud-telemetry | conn.request(
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 441, in request
blueos-cloud-telemetry | self.endheaders()
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/http/client.py", line 1281, in endheaders
blueos-cloud-telemetry | self._send_output(message_body, encode_chunked=encode_chunked)
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/http/client.py", line 1041, in _send_output
blueos-cloud-telemetry | self.send(msg)
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/http/client.py", line 979, in send
blueos-cloud-telemetry | self.connect()
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/transport/unixconn.py", line 26, in connect
blueos-cloud-telemetry | sock.connect(self.unix_socket)
blueos-cloud-telemetry | urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
blueos-cloud-telemetry |
blueos-cloud-telemetry | During handling of the above exception, another exception occurred:
blueos-cloud-telemetry |
blueos-cloud-telemetry | Traceback (most recent call last):
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 223, in _retrieve_server_version
blueos-cloud-telemetry | return self.version(api_version=False)["ApiVersion"]
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
blueos-cloud-telemetry | return self._result(self._get(url), json=True)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 44, in inner
blueos-cloud-telemetry | return f(self, *args, **kwargs)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 246, in _get
blueos-cloud-telemetry | return self.get(url, **self._set_request_timeout(kwargs))
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
blueos-cloud-telemetry | return self.request("GET", url, **kwargs)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
blueos-cloud-telemetry | resp = self.send(prep, **send_kwargs)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
blueos-cloud-telemetry | r = adapter.send(request, **kwargs)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 682, in send
blueos-cloud-telemetry | raise ConnectionError(err, request=request)
blueos-cloud-telemetry | requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
blueos-cloud-telemetry |
blueos-cloud-telemetry | The above exception was the direct cause of the following exception:
blueos-cloud-telemetry |
blueos-cloud-telemetry | Traceback (most recent call last):
blueos-cloud-telemetry | File "/opt/app/major_tom.py", line 36, in <module>
blueos-cloud-telemetry | cli()
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
blueos-cloud-telemetry | return self.main(*args, **kwargs)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
blueos-cloud-telemetry | rv = self.invoke(ctx)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
blueos-cloud-telemetry | return _process_result(sub_ctx.command.invoke(sub_ctx))
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
blueos-cloud-telemetry | return ctx.invoke(self.callback, **ctx.params)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
blueos-cloud-telemetry | return __callback(*args, **kwargs)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/opt/app/major_tom.py", line 28, in cloud
blueos-cloud-telemetry | from src.cloud import cloud as Cloud
blueos-cloud-telemetry | File "/opt/app/src/cloud/cloud.py", line 13, in <module>
blueos-cloud-telemetry | from src.cloud.container_task import ContainerTask
blueos-cloud-telemetry | File "/opt/app/src/cloud/container_task.py", line 5, in <module>
blueos-cloud-telemetry | client = docker.from_env()
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 94, in from_env
blueos-cloud-telemetry | return cls(
blueos-cloud-telemetry | ^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
blueos-cloud-telemetry | self.api = APIClient(*args, **kwargs)
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 207, in __init__
blueos-cloud-telemetry | self._version = self._retrieve_server_version()
blueos-cloud-telemetry | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blueos-cloud-telemetry | File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 230, in _retrieve_server_version
blueos-cloud-telemetry | raise DockerException(
blueos-cloud-telemetry | docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
blueos-cloud-telemetry exited with code 1

[patrickelectric]

@andrewkwolek can you try the latest master ?

[andrewkwolek]

@patrickelectric sorry for the delayed reply. I pulled and merged the latest master and am still seeing the same error in blueos-cloud-telemetry

Seeing this error: FileNotFoundError: [Errno 2] No such file or directory: '/host/etc/blueos/hardware-uuid'

[andrewkwolek]

@patrickelectric I was able to get both images running on my machine! Your fix worked. I realized I didn't pull the updated docker image, I just pulled the code to my local git repo. I am seeing the following error in cloud-telemetry:
blueos-cloud-telemetry | 2025-01-14 20:48:10.803 | ERROR | src.helper:unblock_once_bag_is_up:19 - The Bag service seems to be unreachable!! I will try again in 20 seconds

I have a BlueROV2 arriving in a few days so I should be able to test on a real robot shortly. Thanks!

[patrickelectric]

I'll look into it, but of you are not using BlueOS cloud features you should be fine.

[patrickelectric]

@andrewkwolek I just checked now and the error is not there, did it appears only once ?

[andrewkwolek]

Hey @patrickelectric. It continues to appear every 20 seconds as it can not find the bag service. Should I be running QGroundControl or some other process in addition to BlueOS? Maybe I've missed something.

[andrewkwolek]

Hey @patrickelectric I am unable to access any of the services through my localhost when I run the container. It does not show any ports being exposed to my system. Am I missing something? This is the full log when I spin up the container.

✔ Container blueos-core Created 0.3s
✔ Container blueos-cloud-telemetry Created 0.2s
Attaching to blueos-cloud-telemetry, blueos-core
blueos-core | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
blueos-core | bash: no job control in this shell
blueos-core | BlueOS 1.4.0-beta.10-1-gc965e094
blueos-core | Available disk space: 971581MB
blueos-core | Total RAM size: 7832 MB
blueos-core | Waring: Using dockers's resolv.conf, this may cause DNS issues if the host's network configuration changes.
blueos-core | Warning: Identity file /root/.config/.ssh/id_rsa not accessible: No such file or directory.
blueos-core | ssh: connect to host localhost port 22: Connection refused
blueos-core | SSH is not working, skipping cgroups preparation
blueos-core | Starting high priority services..
blueos-core | Service: autopilot: nice --19 /home/pi/services/ardupilot_manager/main.py with memory limit: 7832 MB
blueos-core | Service: cable_guy is disabled
blueos-core | Service: video: nice --19 mavlink-camera-manager --default-settings BlueROVUDP --mavlink tcpout:127.0.0.1:5777 --mavlink-system-id 1 --gst-feature-rank omxh264enc=0,v4l2h264enc=250,x264enc=260 --log-path /var/logs/blueos/services/mavlink-camera-manager --stun-server stun://stun.l.google.com:19302 --verbose with memory limit: 7832 MB
blueos-core | Service: mavlink2rest: mavlink2rest --connect=udpout:127.0.0.1:14001 --server [::]:6040 --system-id 1 --component-id 194 with memory limit: 7832 MB
blueos-cloud-telemetry | 2025-01-23 00:20:19 | INFO | src.telemetry.telemetry:run:105 - Starting Major Tom, session UUID: 639e7380-aab1-464e-91d3-06ab472760db
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /ardupilot-manager/v1.0/firmware_info (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbb2c890>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /ardupilot-manager/v1.0/vehicle_type (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbcea550>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /ardupilot-manager/v1.0/firmware_vehicle_type (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbb2f810>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /ardupilot-manager/v1.0/board (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbcea090>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /version-chooser/v1.0/version/current (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbce81d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /system-information/model (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbce22d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /system-information/system/disk (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbce0190>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /system-information/system/memory (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbcb9e90>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /wifi-manager/v1.0/scan (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbce12d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.snapshot:get_api_snapshot:48 - HTTPConnectionPool(host='blueos', port=80): Max retries exceeded with url: /kraken/v2.0/extension/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdadbce3650>: Failed to establish a new connection: [Errno 111] Connection refused'))
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.telemetry.telemetry:clean_api_snapshot:36 - Failed to clean disk information
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.telemetry.telemetry:clean_api_snapshot:37 - 'NoneType' object is not subscriptable
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.telemetry.telemetry:clean_api_snapshot:44 - Failed to clean wifi information
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.telemetry.telemetry:clean_api_snapshot:45 - object of type 'NoneType' has no len()
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.telemetry.telemetry:clean_api_snapshot:52 - Failed to clean extension information
blueos-cloud-telemetry | 2025-01-23 00:20:19 | ERROR | src.telemetry.telemetry:clean_api_snapshot:53 - 'NoneType' object is not iterable
blueos-cloud-telemetry | 2025-01-23 00:20:19 | WARNING | src.helper:get_latency:78 - Getting Latency from SpeedTest
blueos-cloud-telemetry | 2025-01-23 00:20:19.416 | INFO | src.cloud.cloud:run:97 - Starting up Major Tom Cloud
blueos-cloud-telemetry | 2025-01-23 00:20:19.417 | ERROR | src.helper:unblock_once_bag_is_up:19 - The Bag service seems to be unreachable!! I will try again in 20 seconds
blueos-cloud-telemetry | INFO: Started server process [76202]
blueos-cloud-telemetry | INFO: Waiting for application startup.
blueos-cloud-telemetry | INFO: Application startup complete.
blueos-cloud-telemetry | INFO: Uvicorn running on http://0.0.0.0:7000 (Press CTRL+C to quit)
blueos-cloud-telemetry | 2025-01-23 00:20:20 | INFO | src.telemetry.engine:__call__:67 - Record 1 sent to Ground Control
blueos-core | Starting other services..
blueos-core | Service: kraken: nice -19 /home/pi/services/kraken/main.py with memory limit: 7832 MB
blueos-core | Service: wifi is disabled
blueos-core | Service: zenohd: ZENOH_BACKEND_FS_ROOT=/home/pi/tools/zenoh zenohd -c /home/pi/tools/zenoh/blueos-zenoh.json5 with memory limit: 7832 MB
blueos-core | Service: beacon: /home/pi/services/beacon/main.py with memory limit: 7832 MB
blueos-core | Service: bridget: nice -19 sudo -u blueos /home/pi/services/bridget/main.py with memory limit: 7832 MB
blueos-core | Service: commander is disabled
blueos-core | Service: nmea_injector: nice -19 /home/pi/services/nmea_injector/nmea_injector/main.py with memory limit: 7832 MB
blueos-core | Service: helper: /home/pi/services/helper/main.py with memory limit: 7832 MB
blueos-core | Service: iperf3: iperf3 --server --port 5201 with memory limit: 7832 MB
blueos-core | Service: linux2rest: linux2rest --log-path /var/logs/blueos/services/linux2rest --log-settings netstat=30,platform=10,serial-ports=10,system-cpu=10,system-disk=30,system-info=10,system-memory=10,system-network=10,system-process=60,system-temperature=10,system-unix-time-seconds=10 with memory limit: 7832 MB
blueos-core | Service: filebrowser: nice -19 filebrowser --database /etc/filebrowser/filebrowser.db --baseurl /file-browser with memory limit: 7832 MB
blueos-core | Service: versionchooser: /home/pi/services/versionchooser/main.py with memory limit: 7832 MB
blueos-core | Service: pardal: nice -19 /home/pi/services/pardal/main.py with memory limit: 7832 MB
blueos-core | Service: ping: nice -19 sudo -u blueos /home/pi/services/ping/main.py with memory limit: 7832 MB
blueos-core | Service: user_terminal: cat /etc/motd with memory limit: 7832 MB
blueos-core | Service: ttyd: nice -19 ttyd -p 8088 sh -c "/usr/bin/tmux attach -t user_terminal || /usr/bin/tmux new -s user_terminal" with memory limit: 7832 MB
blueos-core | Service: nginx: nice -18 nginx -g "daemon off;" -c /home/pi/tools/nginx/nginx.conf with memory limit: 7832 MB
blueos-core | Service: log_zipper: nice -20 /home/pi/services/log_zipper/main.py '/shortcuts/system_logs/\\*\\*/\\*.log' --max-age-minutes 60 with memory limit: 7832 MB
blueos-core | Service: bag_of_holding: /home/pi/services/bag_of_holding/main.py with memory limit: 7832 MB
blueos-core | BlueOS running!
blueos-cloud-telemetry | 2025-01-23 00:20:39.422 | ERROR | src.helper:unblock_once_bag_is_up:19 - The Bag service seems to be unreachable!! I will try again in 20 seconds
blueos-cloud-telemetry | 2025-01-23 00:20:59.426 | ERROR | src.helper:unblock_once_bag_is_up:19 - The Bag service seems to be unreachable!! I will try again in 20 seconds
blueos-cloud-telemetry | 2025-01-23 00:21:19.431 | ERROR | src.helper:unblock_once_bag_is_up:19 - The Bag service seems to be unreachable!! I will try again in 20 seconds

[andrewkwolek]

Hi @patrickelectric. I figured it out! Long story short I had another process running on port 80 which is why cloud telemetry couldn't connect, and then for whatever reason network_mode: host did not work on my Ubuntu 24.04 system so I had to manually expose all of the ports by uncommenting them in the .yml file. I also removed the version line at the top of the .yml file as that is no longer used in Docker. If you have any insight as to why network_mode: host works on your system I'd be curious to know.

[patrickelectric]

Having a service running at port 80 explains why. network_mode host should just work, if you are running inside a VM or something like it may explain. There are also some problems in macos related to network_mode host. That's why we have all the ports mapped

[patrickelectric]

Closing for now, be free to reopen.