Issue with Instance Launching and Build Process After Updating to Anbox 1.23

Anbox Cloud Users
1

Hello Anbox Cloud Support Team,

I hope this message finds you well.

After updating Anbox to version 1.23, I’ve encountered an issue where I’m no longer able to launch new instances. Each time I attempt to launch an instance using the following command:

amc launch prova -s +adb --enable-graphics --enable-streaming --tags "emulator_name=test_01"

the instance remains stuck in the “started” phase for about 5 to 10 minutes before eventually failing.

I’ve attached the manifest of the application “prova” for your reference:

name: prova
instance-type: g4.3
image: jammy:android13:arm64
resources:
  memory: 4GB
  disk-size: 10GB
addons:
  - frida-server
  - sqlite3

Additionally, I’ll include the output of anbox-cloud-appliance.buginfo and the system.log that I was able to retrieve from the instance:

anbox-cloud-appliance.buginfo
system.log

Furthermore, I tried performing a clean installation of the appliance on another machine (not on AWS), but I encountered a different issue there. I was unable to build the application as the build instance remained stuck at “created” without generating any logs.

If you require additional logs or information, I’m happy to provide them privately via email or another method.

Thank you for your assistance.

Best regards,
Marco

2

Hi!

Looking at the system.log file you linked, it looks to me like Android is failing to boot (there is no Started Anbox. message).

Could you share the android logs with us? You should be able to do that from the instance in the error state. You could also create another instance, and retrieve them while it is stuck in the ‘started’ phase.

Thanks!

Alexis

3

Hi Alexis,

Thank you for the prompt reply. I’ve retrieved the Android logs as you requested. You can find them here.

Please let me know if you need any further information.

Thanks again for your help!

Best regards,
Marco

4

It looks like surfaceflinger is failing to set up graphics.

Could you try to run an instance with software rendering?

In your case, that should be as simple as changing the resource type from g4.3 to a4.3 in the application manifest:

name: prova
instance-type: a4.3
image: jammy:android13:arm64
resources:
  memory: 4GB
  disk-size: 10GB
addons:
  - frida-server
  - sqlite3

If that works without crashing, that probably means communication with the GPU is not working somehow.

5

Hi Alexis,

I tried running an instance with software rendering, but it crashed immediately upon launch. You can find the logs here.

Please let me know if there’s anything else I should try.

Thank you for your continued assistance!

Best regards,
Marco

6

Thanks @collez!

Can you try to run the 1.22.2 image and see if that works? You can import images of previous versions by running

amc image add android13-1.22 jammy:android13:arm64@1.22.2

and then use android13-1.22 in your manifest.yaml for the images:field, e.g.

name: prova
instance-type: a4.3
image: android13-1.22
resources:
  memory: 4GB
  disk-size: 10GB
addons:
  - frida-server
  - sqlite3

Can you also provide me the output of

lsmod
sudo cat /sys/module/nvidia_drm/parameters/modeset

Thanks!

7

Hello Simon,

Thank you for the suggestions. I tried downloading the 1.22.2 image, but I received the following error:

Error: version 1.22.2 for image jammy:android13:arm64 not found

However, I was able to download the 1.22.1 version using:

amc image add android13-1.22 jammy:android13:arm64@1.22.1

When I attempt to update the image with:

amc application update prova applications/prova/

I encounter the error:

Error: websocket: close 1006 (abnormal closure): unexpected EOF

If I delete the image and recreate it, the process works correctly, and I can confirm that using the 1.22.1 image allows the instance to start successfully.

I also have a question regarding the new “Seamless ADB connection” feature introduced in version 1.23, as mentioned in the release notes. Could you provide more details about this feature? Will it replace the traditional -s +adb option, or is it a separate functionality?

As requested, here is the output of lsmod:

lsmod output 
Module                  Size  Used by
tls                   143360  0
nft_masq               12288  2
nft_chain_nat          12288  2
nf_nat                 61440  2 nft_masq,nft_chain_nat
bridge                380928  0
stp                    12288  1 bridge
llc                    16384  2 bridge,stp
nf_tables             323584  111 nft_masq,nft_chain_nat
nfnetlink              20480  1 nf_tables
binfmt_misc            24576  1
nls_iso8859_1          12288  1
nvidia_uvm           4640768  0
nvidia_drm            114688  0
nvidia_modeset       1363968  1 nvidia_drm
nvidia              54775808  2 nvidia_uvm,nvidia_modeset
zfs                  5369856  6
aes_ce_blk             32768  0
spl                   176128  1 zfs
aes_ce_cipher          12288  1 aes_ce_blk
crct10dif_ce           12288  1
polyval_ce             12288  0
polyval_generic        12288  1 polyval_ce
ghash_ce               24576  0
sm4                    12288  0
sha2_ce                16384  0
sha256_arm64           24576  1 sha2_ce
sha1_ce                12288  0
video                  69632  1 nvidia_modeset
drm_kms_helper        274432  1 nvidia_drm
ena                   139264  0
qemu_fw_cfg            24576  0
dm_multipath           49152  0
scsi_dh_rdac           16384  0
scsi_dh_emc            12288  0
scsi_dh_alua           24576  0
sch_fq_codel           24576  9
nf_conntrack          212992  2 nf_nat,nft_masq
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         12288  1 nf_conntrack
libcrc32c              12288  3 nf_conntrack,nf_nat,nf_tables
virt_wifi              24576  0
cfg80211             1163264  1 virt_wifi
anbox_sync             20480  0
binder_linux          217088  0
drm                   794624  4 drm_kms_helper,nvidia,nvidia_drm
efi_pstore             12288  0
ip_tables              36864  0
x_tables               69632  1 ip_tables
autofs4                53248  2

And the output of sudo cat /sys/module/nvidia_drm/parameters/modeset is N.

Thank you for your continued assistance.

Best regards,
Marco

1 Like
8

However, I was able to download the 1.22.1 version using:

Hm, let me look into that but 1.22.1 should be enough to proof the point.

If I delete the image and recreate it, the process works correctly, and I can confirm that using the 1.22.1 image allows the instance to start successfully.

Ok, that is good and we will try next to reproduce this and see what we find. Thanks a lot for the information!

I also have a question regarding the new “Seamless ADB connection” feature introduced in version 1.23, as mentioned in the release notes. Could you provide more details about this feature? Will it replace the traditional -s +adb option, or is it a separate functionality?

It is additional functionality to allow you connecting securely over ADB from your local laptop (for example) to a cloud instance running on AWS. ADB by itself is not build for this kind of use case so we put a secure WebRTC based communication channel underneath it with the seamless ADB feature.

The existing port forwarding option will remain as-is and not going away but depending on your use-case, specifically when you need access to the instance over the internet, you should prefer seamless ADB from now on instead.

You can find the documentation for the feature here

9

@collez one more thing: Can you confirm if you executed the script from the output of anbox-cloud-appliance prepare-node-script?

10

Hello Simon,

I can confirm that I’ve successfully run the script using the following command before initializing the Anbox Cloud Appliance:

anbox-cloud-appliance prepare-node-script | sudo bash -ex

I also tested the new Seamless ADB Connection feature, but after running the command:

anbox-connect <url>

the program gets stuck at “Establishing ADB connection…” and doesn’t proceed further.

Additionally, I’d like to know if there is a way to programmatically obtain the URL for the ADB connection other than the CLI and the Web Dashboard. It would be fantastic if this could be done using the AMS SDK.

Thank you for your help.

Best regards,
Marco

11

That’s puzzling but we will investigate and try to understand what happened. Thanks for confirming!

Are you trying this against a 1.22.x image? I guess so because that is the one you got working. You will need >= 1.23.0 for seamless ADB to work as there are changes on the image itself needed for this to work.

Yes, you can either use sudo anbox-cloud-appliance.gateway session share <id> where you can find the session id in the amc ls output for the particular instance. Alternatively you can use the /1.0/sessions/<id>/share API endpoint to generate the URL yourself. This is the same endpoint the UI and CLI are using.

12

@collez can you check what content /etc/modprobe.d/anbox-nvidia.conf has? It should have a line options nvidia_drm modeset=1.

13

Thank you for looking into this. If you need any additional information about the environment or the AWS machine, please feel free to ask.

Thanks again for the clarifications. One last question: will there be a hotfix for this issue, or should we expect to wait until September 18th for version 1.23.1?

Best regards,
Marco

14

Hello Simon,

I checked the /etc/modprobe.d/ directory, but I couldn’t find the anbox-nvidia.conf file there.

15

We typically don’t do out-of-order patch releases, so next release will be on 1.23.1 and if there is a fix in Anbox Cloud necessary itself it will be included.

Ok, this smells suspicious like the output of anbox-cloud-appliance prepare-node-script was not correct. Did you install the NVIDIA driver manually or did the script install it?

16

Hello Simon,

After running anbox-cloud-appliance init, the program encountered an error indicating that it couldn’t parse the NVIDIA driver version. Upon checking, I noticed that the drivers were not installed, so I manually installed version 550.90.07.

Should I uninstall this version and try running the script again?

Thank you for your guidance.

17

Thanks!

I was able to reproduce the issue now and as I suspected it boils down to options nvidia_drm modeset=1 not being set which is required by Anbox in order to make use of certain functionality the NVIDIA driver provides only in this mode. You can find some more details here.

To solve the problem here, you can do the following:

  1. Delete or stop all AMS instances
  2. Unload the nvidia_drm kernel module: $ sudo rmmod nvidia_drm
  3. Create /etc/modprobe.d/anbox-nvidia.conf with the following content
options nvidia NVreg_RegistryDwords=RMIncreaseRsvdMemorySizeMB=256;RMBar1ApertureSizeMB=256
options nvidia_drm modeset=1
  1. Load the nvidia_drm kernel module again: $ sudo modprobe nvidia_drm

After this 1.23.0 based instances should come up without any further problem.

The second option options nvidia NVreg_RegistryDwords=RMIncreaseRsvdMemorySizeMB=256;RMBar1ApertureSizeMB=256 is not as critical and you can wait until you reboot the system the next time, unless you want to run a larger number of instances in parallel.

I will make sure we better document and at best hint towards this being a problem if the option is not set for the kernel driver.

1 Like
18

Hello Simon,

I wanted to confirm that the solution you provided worked perfectly. Thank you so much for your assistance and support throughout this process.

Best regards,
Marco