Auto control stopped working

support
1

Auto control stopped working. I would like to know where to start troubleshooting.

  • What package and version are you using ?
    Brain: 1.3.0
    Dashboard: 1.9.0, updated to 2.0.0.

  • Please, describe the issues you are seeing ?
    I went to test auto control; virtual joystick stopped working. The dashboard transitions from auto ready to auto active but the motors do not move. You can see I’m moving the joystick and the dashboard speed/accel meter is moving but the chassis is stationary (and the E-stop is obviously off).

20231116_205826
20231116_2058262944×1652 1.05 MB

(You can also see the camera is completely white, I’m unsure why; the recorder app works.)

20231116_205709
20231116_2057094032×3024 505 KB

The Amiga will move when using the pendant and the motors appear to be healthy.

20231116_205740
20231116_2057403444×1976 1.6 MB

Auto control was working yesterday when using the amiga-xbox-controller interface and I don’t believe anything changed. I updated the dashboard firmware to 2.0.0 but the issue persists.

I thought updating the Brain may alleviate the problem but I’m not able to retrieve any updates (it’s connected to the internet; I connected via SSH and pinged an external IP).

20231116_205936
20231116_2059364032×3024 1.08 MB

Is there a diagnostic utility for the Brain terminal? Is there a hardware or software utility to capture CAN bus packets (apart from just looking at the circuit on an oscope)?

Thanks in advance!

2

Hello @anb ,

This is unexpected behavior considering the symptoms you have documented here (very thorough and clear, by the way :+1:). There are a few things going on here, so let’s address each one.

Dashboard auto mode

With the dashboard kicking into Auto Active and displaying the commanded linear and angular velocities, it seems the brain is correctly sending the auto command messages. The motors are all healthy and there are no CAN errors (that’s the CAN state, the “rec” (receive error count), & “tec” (transmit error count)). I would expect similar behavior / issues when operating with the pendant, but you indicate that is not the case.

:question: You indicated auto control was working with the amiga-xbox-controller. Is that still working or has that also stopped working?

Brain OS upgrades

There will not be any over-the-air (OTA) updates available beyond v1.3. We have just released v2.0 (which requires one physical flashing step for the upgrade before unlocking all future OTA updates). As noted in the release notes, you can reach out to support@farm-ng.com to coordinate the upgrade.

Corresponding with brain OS release, we released updated pip packages for farm-ng-amiga and farm-ng-core, which both increment from v0.X.X → v2.0.0

It could be possible these pip packages updated in the venv for your virtual-joystick or amiga-xbox-controller and are mismatched with the services v1.3.0 brainOS. You can check with:

cd <path/to>/virtual-joystick
source venv/bin/activate
pip list

To which you should see (among other packages)

farm-ng-amiga            0.0.9
farm-ng-core               0.1.4

If not, you can pin these two pip packages to these versions (e.g., pip install farm-ng-amiga==v0.0.9).

CAN tools

If you wish to inspect the packets on the canbus, you can ssh into the brain and install the can-utils apt package.

sudo apt update
sudo apt install can-utils

Two good command line tools for inspecting CAN messages are:

I’m looking forward to getting to the bottom of this.

– Kyle

3

I found the problem. If the Brain is connected to a network (there exists a default route in the IP routing table), and there is an internet connectivity issue, virtually everything stops working (the cameras, the joystick, the Xbox controller interface). It does not matter if the default route is via WiFi (192.168.0.0/24) or Ethernet (10.95.76.0/24). If the Brain is not connected to a network (there is no default route), or, if the Brain is connected and there is no internet connectivity issue, then everything works fine. Is this something that was patched in a recent Brain OS update? Either way, I think I should update and check again.

4

Also, do you think it would be possible in a future update to reveal some more log files in the container? This is what’s in my /var/log directory

MicrosoftTeams-image (3)

Some of these are binary files. I think it would help troubleshooting some weird or intermittent issues.

5

Hi @anb,

Thank you for diagnosing that. It had been some time since we’d seen that issue so it was not top of mind when you reported the symptoms.

Regarding the connections

I am filing a ticket for us to internally stress test the grpc connections under fluctuating network connectivity on the new Amiga OS 2.0. If the issue persists, we will resolve it in a future minor release.

Regarding an upgrade

The upgrade to Amiga OS 2.0 requires a physical reflash of the brain, which will require us to send you a shipping label and for you to send us your brain to perform the upgrade. Please reach out to support@farm-ng.com to coordinate a convenient time for you to be without your brain so we can perform this upgrade.

Regarding logs

This is exposed in the new OS, which uses systemd for starting apps and services, through journalctl commands.

– Kyle

6

Okay, thank you. Sorry for the belated reply; I sent support an email. Before mailing it back to you, do you want any specific diagnostic information given that I know how to reproduce the issue?

7

Thank you for the offer @anb , but you do not need to!

We were able to replicate this behavior in the past on brains running AmigaOS 1.0. We will find a way to replicate this condition for the brains running AmigaOS 2.0 and collect logs on them, as there is so much changed in the back end that it is not clear if logs from AmigaOS 1.0 will be valid.

Thanks!
Kyle