Anbox container change disk-size doesn't work

dfelici87 · January 26, 2024, 4:38pm

Hi,

I try to increase disk-size value of manifest.yml of my application, but if I change disk-size value the application doesn’t start and remaining in “initializing” status (I attend 24 hours).

This is the application detail:
id: cmptnbi0pvn6093ei2ng
name: test_v1
created-at: 2024-01-26 16:19:58 +0000 UTC
status: initializing
published: false
immutable: false
inhibit-auto-updates: false
tags: []
node-selector: []
parent-image: jammy:android13:arm64
abi: arm64-v8a
vm: false
config:
instance-type: g4.3
versions:
0:
image: jammy:android13:arm64 (version 2)
published: false
status: initializing
addons:
- sqlite3 (version 1)
extra-data: {}
video-encoder: gpu-preferred
watchdog:
disabled: false
allowed-packages: []
features:
- allow_custom_system_signatures
resources:
cpus: 2
memory: 4GB
disk-size: 8GB

anbox-cloud version is: 1.20.1.

Someone help me please?

Thanks,
Demis.

gary-wzl77 · January 29, 2024, 7:47am

Hey dfelici87
Your deployment likely ran out of disk capacity. Are you using Anbox Cloud appliance? If so, could you please dump the logs with

  anbox-cloud-appliance.buginfo

If not(E.g. for regular deployment), please share the output of the following commands`

  amc ls
  amc app ls
  amc node show lxd0

This would help us to figure out why the instance got stuck to “initializing” status.

Thanks

BR
Gary

dfelici87 · January 29, 2024, 9:20am

Hi,

yes, I use anbox cloud appliance.
This is the command result (I masked some information E.g.: addon names):

…
2024-01-26T20:08:09Z anbox-cloud-appliance.traefik[1293]: time=“2024-01-26T20:08:09Z” level=error msg=“Error while handling TCP connection: readfrom tcp 240.0.86.1:60954->240.0.86.97:8444: read tcp 172.31.0.86:8444->118.194.250.2:35436: read: connection reset by peer”
2024-01-26T22:34:20Z anbox-cloud-appliance.traefik[1293]: time=“2024-01-26T22:34:20Z” level=error msg=“Error while handling TCP connection: readfrom tcp 240.0.86.1:47854->240.0.86.97:8444: read tcp 172.31.0.86:8444->167.248.133.190:49304: read: connection reset by peer”
2024-01-28T03:23:59Z anbox-cloud-appliance.traefik[1293]: time=“2024-01-28T03:23:59Z” level=error msg=“Error while handling TCP connection: readfrom tcp 240.0.86.1:47838->240.0.86.97:8444: read tcp 172.31.0.86:8444->118.123.105.93:41608: read: connection reset by peer”
2024-01-28T03:23:59Z anbox-cloud-appliance.traefik[1293]: time=“2024-01-28T03:23:59Z” level=error msg=“Error while handling TCP connection: readfrom tcp 240.0.86.1:48658->240.0.86.97:8444: read tcp 172.31.0.86:8444->118.123.105.93:45892: read: connection reset by peer”
2024-01-28T03:24:03Z anbox-cloud-appliance.traefik[1293]: time=“2024-01-28T03:24:03Z” level=error msg=“Error while handling TCP connection: readfrom tcp 240.0.86.1:48676->240.0.86.97:8444: read tcp 172.31.0.86:8444->118.123.105.93:35558: read: connection reset by peer”
2024-01-28T20:15:11Z anbox-cloud-appliance.traefik[1293]: time=“2024-01-28T20:15:11Z” level=error msg=“Error while handling TCP connection: readfrom tcp 240.0.86.1:39538->240.0.86.97:8444: read tcp 172.31.0.86:8444->167.248.133.122:34472: read: connection reset by peer”

Juju Status

Model Controller Cloud/Region Version SLA Timestamp
anbox-cloud appliance appliance/default 2.9.42 unsupported 08:59:10Z

App Version Status Scale Charm Channel Rev Exposed Message
ams 1.20.0 active 1 ams 1.20/stable 241 no
anbox-cloud-dashboard 1.20.0 active 1 anbox-cloud-dashboard 1.20/stable 228 no
anbox-stream-agent 1.20.0 active 1 anbox-stream-agent 1.20/stable 241 no
anbox-stream-gateway 1.20.0 active 1 anbox-stream-gateway 1.20/stable 246 no
coturn 4.5.2 active 1 coturn 1.20/stable 230 yes
etcd 3.4.22 active 1 etcd stable 742 no Healthy with 1 known peer
etcd-ca 3.0.1 active 1 easyrsa stable 39 no Certificate Authority connected.
internal-ca 3.0.1 active 1 easyrsa stable 39 no Certificate Authority connected.
lxd-integrator active 1 lxd-integrator stable 4 no
nats active 1 nats-charmers-nats stable 11 no

Unit Workload Agent Machine Public address Ports Message
ams/0* active idle 1 240.0.86.97 8444/tcp,20002/tcp
anbox-cloud-dashboard/0* active idle 1 240.0.86.97 5000/tcp
anbox-stream-agent/0* active idle 1 240.0.86.97 8082/tcp
anbox-stream-gateway/0* active idle 1 240.0.86.97 4000/tcp,7000/tcp
coturn/0* active idle 1 240.0.86.97 5349/tcp,5349/udp,60000-60100/udp
etcd-ca/0* active idle 0 240.0.86.196 Certificate Authority connected.
etcd/0* active idle 1 240.0.86.97 2379/tcp Healthy with 1 known peer
internal-ca/0* active idle 0 240.0.86.196 Certificate Authority connected.
lxd-integrator/0* active idle 0 240.0.86.196
nats/0* active idle 1 240.0.86.97 4222/tcp

Machine State Address Inst id Series AZ Message
0 started 240.0.86.196 juju-bcc362-0 jammy Running
1 started 240.0.86.97 juju-bcc362-1 jammy Running

Juju Config

coturn.internal_addresses:
coturn.location: 18.158.139.177
gateway.location: ec2-18-158-139-177.eu-central-1.compute.amazonaws.com
dashboard.location: ec2-18-158-139-177.eu-central-1.compute.amazonaws.com

AMS config

config:
application.addons: “”
application.auto_publish: true
application.auto_update: true
application.default_abi: “”
application.max_published_versions: 3
container.apt_mirror: “”
container.default_platform: “”
container.features: “”
container.security_updates: true
core.debug_address: “”
core.proxy_http: “”
core.proxy_https: “”
core.proxy_ignore_hosts: “”
core.trust_password: false
cpu.limit_mode: pinning
gpu.allocation_mode: single
gpu.type: nvidia
images.allow_insecure: false
images.auth: true
images.update_interval: 5m
images.url: https://images.anbox-cloud.io/stable/
images.version_lockstep: true
instance.apt_mirror: “”
instance.default_platform: “”
instance.features: “”
instance.security_updates: “true”
load_balancer.url: “”
node.queue_size: 100
node.workers_per_queue: 4
registry.filter: “”
registry.fingerprint: “”
registry.mode: pull
registry.update_interval: 1h
registry.url: “”
scheduler.strategy: spread

AMS Nodes

AMS Containers

AMS Applications

±---------------------± | ID ±---------------------± | clmtq620pvn60b34j1hg | marco_v13 | | | | | | | | | | | | | | | | ±---------------------± | cmj796q0pvn60b73hpsg | nik_v1 | | | | | | | | | | ±---------------------± | cmnabpi0pvn6093ei1ig | demis_v13b | | | | | | | | | | | | | | | | ±---------------------± | cmptnbi0pvn6093ei2ng | | | | | | | | | | | | | | | | | | | | | | | | | | | | ±---------------------± ----------------±--------------±-------------------±-----±----------±-------------±--------------------±------+
| NAME | INSTANCE TYPE | ADDONS | TAGS | PUBLISHED | STATUS | LAST UPDATED | VM |
----------------±--------------±-------------------±-----±----------±-------------±--------------------±------+
| g4.3 | 1111 | | true | ready | 2024-01-25 14:30:52 | false |
| | 2222 | | | | | |
| | 3333 | | | | | |
| | 4444 | | | | | |
| | 5555 | | | | | |
| | frida-server-arm64 | | | | | |
| | 6666 | | | | | |
| | 7777 | | | | | |
| | 8888 | | | | | |
----------------±--------------±-------------------±-----±----------±-------------±--------------------±------+
| g4.3 | 1111 | | true | ready | 2024-01-17 09:53:35 | false |
| | 2222 | | | | | |
| | sqlite3 | | | | | |
| | frida-server-arm64 | | | | | |
| | 4444 | | | | | |
| | 5555 | | | | | |
----------------±--------------±-------------------±-----±----------±-------------±--------------------±------+
| g4.3 | 1111 | | true | ready | 2024-01-22 17:29:10 | false |
| | brave | | | | | |
| | 2222 | | | | | |
| | 3333 | | | | | |
| | sqlite3 | | | | | |
| | frida-server-arm64 | | | | | |
| | twitter | | | | | |
| | 5555 | | | | | |
| | 6666 | | | | | |
----------------±--------------±-------------------±-----±----------±-------------±--------------------±------+
| phoneScanner_v1 | g4.3 | 1111 | | false | initializing | 2024-01-26 16:19:58 | false |
| | 2222 | | | | | |
| | 3333 | | | | | |
| | 4444 | | | | | |
| | 5555 | | | | | |
| | 6666 | | | | | |
| | 7777 | | | | | |
| | sqlite3 | | | | | |
| | 8888 | | | | | |
| | 9999 | | | | | |
| | 0000 | | | | | |
| | xxxx | | | | | |
| | twitter | | | | | |
| | yyyy | | | | | |
| | zzzz | | | | | |
----------------±--------------±-------------------±-----±----------±-------------±--------------------±------+

AMS Addons

±-------------------±---------±------------------------------------------------------------------------------------+
| NAME | VERSIONS | USED BY |
±-------------------±---------±------------------------------------------------------------------------------------+
| brave | 2 | cmnabpi0pvn6093ei1ig |
±-------------------±---------±------------------------------------------------------------------------------------+
| frida-server | 2 | |
±-------------------±---------±------------------------------------------------------------------------------------+
| frida-server-arm64 | 8 | clmtq620pvn60b34j1hg,cmj796q0pvn60b73hpsg,cmnabpi0pvn6093ei1ig |
±-------------------±---------±------------------------------------------------------------------------------------+
| multi-user-V1 | 1 | |
±-------------------±---------±------------------------------------------------------------------------------------+
| multi-user-V2 | 1 | |
±-------------------±---------±------------------------------------------------------------------------------------+
| sqlite3 | 2 | clmtq620pvn60b34j1hg,cmj796q0pvn60b73hpsg,cmnabpi0pvn6093ei1ig,cmptnbi0pvn6093ei2ng |
±-------------------±---------±------------------------------------------------------------------------------------+
| twitter | 1 | cmnabpi0pvn6093ei1ig,cmptnbi0pvn6093ei2ng |
±-------------------±---------±------------------------------------------------------------------------------------+
| wallpaper | 1 | |
±-------------------±---------±------------------------------------------------------------------------------------+

AMS Tasks

AMS Operations

AMS node “lxd0”

architecture: aarch64
config:
cpu-allocation-rate: 4
cpus: 8
gpu-encoder-slots: 32
gpu-slots: 32
memory: 16GB
memory-allocation-rate: 2
public-address: 18.158.139.177
tags: []
disk:
pool: data
size: 96GB
gpus:
“0”:
encoder-slots: 32
numa-node: 0
pci-address: 0000:00:1f.0
render-name: renderD128
slots: 32
managed: false
name: lxd0
network:
address: 172.31.0.86
bridge-mtu: 1500
status: online
unscheduable: false
unschedulable: false
vpus: {}

AMS application “clmtq620pvn60b34j1hg”

id: clmtq620pvn60b34j1hg
name: marco_v13
created-at: 2023-12-04 14:07:20 +0000 UTC
status: ready
published: true
immutable: false
inhibit-auto-updates: false
tags: []
node-selector: []
parent-image: jammy:android13:arm64
abi: arm64-v8a
vm: false
config:
instance-type: g4.3
versions:
35:
image: jammy:android13:arm64 (version 2)
published: true
status: active
addons:
- 1111 (version 9)
- 2222 (version 1)
- 3333 (version 0)
- 4444 (version 21)
- sqlite3 (version 1)
- frida-server-arm64 (version 7)
- 5555 (version 2)
- 6666 (version 0)
- 7777 (version 3)
extra-data: {}
video-encoder: gpu-preferred
watchdog:
disabled: false
allowed-packages: []
features:
- allow_custom_system_signatures

AMS application “cmj796q0pvn60b73hpsg”

id: cmj796q0pvn60b73hpsg
name: nik_v1
created-at: 2024-01-16 12:20:43 +0000 UTC
status: ready
published: true
immutable: false
inhibit-auto-updates: false
tags: []
node-selector: []
parent-image: jammy:android13:arm64
abi: arm64-v8a
vm: false
config:
instance-type: g4.3
versions:
0:
image: jammy:android13:arm64 (version 1)
published: true
status: active
addons:
- 1111 (version 9)
- 2222 (version 21)
- 3333 (version 1)
- frida-server-arm64 (version 7)
- 4444 (version 3)
- 5555 (version 1)
extra-data: {}
video-encoder: gpu-preferred
watchdog:
disabled: false
allowed-packages: []
features:
- allow_custom_system_signatures
1:
image: jammy:android13:arm64 (version 2)
published: true
status: active
addons:
- 1111 (version 9)
- 2222 (version 21)
- sqlite3 (version 1)
- frida-server-arm64 (version 7)
- 3333 (version 3)
- 4444 (version 1)
extra-data: {}
video-encoder: gpu-preferred
watchdog:
disabled: false
allowed-packages: []
features:
- allow_custom_system_signatures

AMS application “cmnabpi0pvn6093ei1ig”

id: cmnabpi0pvn6093ei1ig
name: demis_v13b
created-at: 2024-01-22 17:29:10 +0000 UTC
status: ready
published: true
immutable: false
inhibit-auto-updates: false
tags: []
node-selector: []
parent-image: jammy:android13:arm64
abi: arm64-v8a
vm: false
config:
instance-type: g4.3
versions:
0:
image: jammy:android13:arm64 (version 2)
published: true
status: active
addons:
- 1111 (version 5)
- brave (version 1)
- 2222 (version 1)
- 3333 (version 21)
- sqlite3 (version 1)
- frida-server-arm64 (version 7)
- twitter (version 0)
- 4444 (version 0)
- 5555 (version 0)
extra-data: {}
video-encoder: gpu-preferred
watchdog:
disabled: false
allowed-packages: []
features:
- allow_custom_system_signatures
resources:
memory: 4GB
disk-size: 20GB

AMS application “cmptnbi0pvn6093ei2ng”

id: cmptnbi0pvn6093ei2ng
name: phoneScanner_v1
created-at: 2024-01-26 16:19:58 +0000 UTC
status: initializing
published: false
immutable: false
inhibit-auto-updates: false
tags: []
node-selector: []
parent-image: jammy:android13:arm64
abi: arm64-v8a
vm: false
config:
instance-type: g4.3
versions:
0:
image: jammy:android13:arm64 (version 2)
published: false
status: initializing
addons:
- 1111 (version 0)
- 2222 (version 3)
- 3333 (version 0)
- 4444 (version 1)
- 5555 (version 21)
- 6666 (version 9)
- 7777 (version 3)
- sqlite3 (version 1)
- 8888 (version 0)
- 9999 (version 1)
- 0000 (version 0)
- xxxx (version 1)
- twitter (version 0)
- yyyy (version 0)
- zzzz (version 5)
extra-data: {}
video-encoder: gpu-preferred
watchdog:
disabled: false
allowed-packages: []
features:
- allow_custom_system_signatures
resources:
cpus: 2
memory: 4GB
disk-size: 8GB

AMS addon “brave”

name: brave
versions:
0:
size: 302.65MB
created-at: 2023-11-22 15:40:28 +0000 UTC
1:
size: 570.68MB
created-at: 2023-11-23 09:19:42 +0000 UTC
used_by:

cmnabpi0pvn6093ei1ig

AMS addon “1111”

name: 1111
versions:
0:
size: 44.41MB
created-at: 2024-01-19 13:46:15 +0000 UTC
used_by:

cmptnbi0pvn6093ei2ng

AMS addon “2222”

name: 2222
versions:
3:
size: 16.70MB
created-at: 2024-01-22 14:29:01 +0000 UTC
used_by:

cmptnbi0pvn6093ei2ng

AMS addon “3333”

name: 3333
versions:
0:
size: 16.26MB
created-at: 2024-01-22 17:26:58 +0000 UTC
used_by:

cmnabpi0pvn6093ei1ig
cmptnbi0pvn6093ei2ng

AMS addon “4444”

name: 4444
versions:
0:
size: 138.22MB
created-at: 2023-11-22 15:43:02 +0000 UTC
1:
size: 138.22MB
created-at: 2024-01-05 11:14:41 +0000 UTC
used_by:

cmnabpi0pvn6093ei1ig
cmptnbi0pvn6093ei2ng

AMS addon “frida-server”

name: frida-server
versions:
0:
size: 103.16MB
created-at: 2023-11-22 15:57:30 +0000 UTC
1:
size: 103.16MB
created-at: 2023-11-27 16:04:25 +0000 UTC
used_by: []

AMS addon “frida-server-arm64”

name: frida-server-arm64
versions:
0:
size: 48.65MB
created-at: 2023-11-22 15:57:55 +0000 UTC
1:
size: 48.65MB
created-at: 2023-11-27 16:20:20 +0000 UTC
2:
size: 48.65MB
created-at: 2023-11-27 16:36:12 +0000 UTC
3:
size: 48.65MB
created-at: 2023-11-27 16:58:27 +0000 UTC
4:
size: 48.65MB
created-at: 2023-11-28 07:52:43 +0000 UTC
5:
size: 48.65MB
created-at: 2023-11-28 10:41:36 +0000 UTC
6:
size: 48.65MB
created-at: 2023-11-28 15:10:14 +0000 UTC
7:
size: 48.65MB
created-at: 2024-01-11 14:29:14 +0000 UTC
used_by:

clmtq620pvn60b34j1hg
cmj796q0pvn60b73hpsg
cmnabpi0pvn6093ei1ig

AMS addon “5555”

name: 5555
versions:
0:
size: 3.10MB
created-at: 2023-11-22 15:58:06 +0000 UTC
1:
size: 3.10MB
created-at: 2023-11-27 16:09:09 +0000 UTC
used_by:

clmtq620pvn60b34j1hg

AMS addon “6666”

name: 6666
versions:
0:
size: 3.10MB
created-at: 2023-12-22 10:26:52 +0000 UTC
1:
size: 3.10MB
created-at: 2023-12-22 10:38:13 +0000 UTC
2:
size: 3.10MB
created-at: 2023-12-22 10:45:46 +0000 UTC
3:
size: 3.10MB
created-at: 2023-12-22 10:49:01 +0000 UTC
4:
size: 3.10MB
created-at: 2023-12-22 11:09:53 +0000 UTC
5:
size: 3.10MB
created-at: 2023-12-22 11:13:21 +0000 UTC
6:
size: 3.10MB
created-at: 2023-12-22 11:17:03 +0000 UTC
7:
size: 3.10MB
created-at: 2023-12-22 11:37:25 +0000 UTC
8:
size: 3.28MB
created-at: 2023-12-22 14:39:50 +0000 UTC
9:
size: 3.28MB
created-at: 2023-12-22 15:39:29 +0000 UTC
10:
size: 2.77MB
created-at: 2023-12-22 15:45:17 +0000 UTC
11:
size: 2.77MB
created-at: 2023-12-22 16:16:38 +0000 UTC
12:
size: 2.77MB
created-at: 2024-01-08 08:33:03 +0000 UTC
13:
size: 5.54MB
created-at: 2024-01-09 11:11:37 +0000 UTC
14:
size: 5.54MB
created-at: 2024-01-09 11:18:26 +0000 UTC
15:
size: 5.52MB
created-at: 2024-01-09 12:01:06 +0000 UTC
16:
size: 5.58MB
created-at: 2024-01-09 12:29:14 +0000 UTC
17:
size: 5.55MB
created-at: 2024-01-09 13:03:45 +0000 UTC
18:
size: 5.55MB
created-at: 2024-01-09 14:27:11 +0000 UTC
19:
size: 5.55MB
created-at: 2024-01-09 14:55:54 +0000 UTC
20:
size: 5.55MB
created-at: 2024-01-09 15:04:35 +0000 UTC
21:
size: 5.57MB
created-at: 2024-01-09 15:42:23 +0000 UTC
used_by:

cmj796q0pvn60b73hpsg
cmnabpi0pvn6093ei1ig
cmptnbi0pvn6093ei2ng

AMS addon “7777”

name: 7777
versions:
0:
size: 14.18kB
created-at: 2024-01-16 10:08:39 +0000 UTC
1:
size: 14.17kB
created-at: 2024-01-16 10:10:40 +0000 UTC
2:
size: 8.83kB
created-at: 2024-01-16 10:56:14 +0000 UTC
3:
size: 8.82kB
created-at: 2024-01-16 11:02:34 +0000 UTC
4:
size: 8.84kB
created-at: 2024-01-16 11:15:15 +0000 UTC
5:
size: 21.53kB
created-at: 2024-01-16 11:25:28 +0000 UTC
6:
size: 23.70kB
created-at: 2024-01-16 11:31:50 +0000 UTC
7:
size: 18.97kB
created-at: 2024-01-16 11:38:08 +0000 UTC
8:
size: 18.97kB
created-at: 2024-01-16 11:40:29 +0000 UTC
9:
size: 8.63kB
created-at: 2024-01-16 12:20:33 +0000 UTC
used_by:

cmj796q0pvn60b73hpsg
cmptnbi0pvn6093ei2ng

AMS addon “8888”

name: 8888
versions:
0:
size: 63.02MB
created-at: 2023-11-30 15:10:25 +0000 UTC
1:
size: 63.02MB
created-at: 2023-11-30 15:45:40 +0000 UTC
2:
size: 63.02MB
created-at: 2023-12-04 09:58:54 +0000 UTC
3:
size: 63.02MB
created-at: 2023-12-04 10:03:17 +0000 UTC
used_by:

cmj796q0pvn60b73hpsg
cmptnbi0pvn6093ei2ng

AMS addon “9999”

name: 9999
versions:
0:
size: 277.20MB
created-at: 2023-12-01 14:43:18 +0000 UTC
1:
size: 282.23MB
created-at: 2023-12-04 13:53:55 +0000 UTC
2:
size: 282.23MB
created-at: 2023-12-11 14:36:23 +0000 UTC
used_by:

clmtq620pvn60b34j1hg

AMS addon “multi-user-V1”

name: multi-user-V1
versions:
0:
size: 2.76kB
created-at: 2023-11-22 15:58:20 +0000 UTC
used_by: []

AMS addon “multi-user-V2”

name: multi-user-V2
versions:
0:
size: 3.06kB
created-at: 2023-11-22 15:58:26 +0000 UTC
used_by: []

AMS addon “sqlite3”

name: sqlite3
versions:
0:
size: 706.11kB
created-at: 2023-11-22 15:58:38 +0000 UTC
1:
size: 1.30MB
created-at: 2023-11-24 16:51:34 +0000 UTC
used_by:

clmtq620pvn60b34j1hg
cmj796q0pvn60b73hpsg
cmnabpi0pvn6093ei1ig
cmptnbi0pvn6093ei2ng

AMS addon “0000”

name: 0000
versions:
0:
size: 26.40MB
created-at: 2024-01-25 12:19:26 +0000 UTC
used_by:

cmptnbi0pvn6093ei2ng

AMS addon “xxxx”

name: xxxx
versions:
0:
size: 2.06MB
created-at: 2023-11-22 15:58:52 +0000 UTC
1:
size: 2.06MB
created-at: 2024-01-11 09:56:54 +0000 UTC
used_by:

cmptnbi0pvn6093ei2ng

AMS addon “yyyy”

name: yyyy
versions:
0:
size: 36.64MB
created-at: 2023-11-22 15:59:00 +0000 UTC
used_by:

cmptnbi0pvn6093ei2ng

AMS addon “zzzz”

name: zzzz
versions:
0:
size: 263.27MB
created-at: 2023-12-04 14:26:14 +0000 UTC
1:
size: 263.27MB
created-at: 2023-12-04 14:30:01 +0000 UTC
used_by:

cmj796q0pvn60b73hpsg
cmptnbi0pvn6093ei2ng

AMS addon “twitter”

name: twitter
versions:
0:
size: 100.38MB
created-at: 2023-11-22 15:59:38 +0000 UTC
used_by:

cmnabpi0pvn6093ei1ig
cmptnbi0pvn6093ei2ng

AMS addon “qqqq”

name: qqqq
versions:
0:
size: 180.24MB
created-at: 2024-01-11 15:54:55 +0000 UTC
used_by:

cmnabpi0pvn6093ei1ig
cmptnbi0pvn6093ei2ng

AMS addon “wallpaper”

name: wallpaper
versions:
0:
size: 110.05kB
created-at: 2023-11-22 15:59:52 +0000 UTC
used_by: []

AMS addon “wwww”

name: wwww
versions:
0:
size: 160.95MB
created-at: 2023-11-22 16:00:05 +0000 UTC
1:
size: 160.95MB
created-at: 2023-11-23 16:30:04 +0000 UTC
2:
size: 160.95MB
created-at: 2023-11-24 08:35:38 +0000 UTC
3:
size: 160.95MB
created-at: 2023-11-24 08:41:28 +0000 UTC
4:
size: 160.95MB
created-at: 2023-11-24 09:32:57 +0000 UTC
5:
size: 168.69MB
created-at: 2023-11-24 13:55:55 +0000 UTC
used_by:

cmnabpi0pvn6093ei1ig
cmptnbi0pvn6093ei2ng

AMS addon “eeee”

name: eeee
versions:
0:
size: 160.95MB
created-at: 2023-11-24 10:48:13 +0000 UTC
used_by: []

AMS container “clg63lq0pvn60b59i85g”

id: clg63lq0pvn60b59i85g
name: ams-clg63lq0pvn60b59i85g
status: error
node: lxd0
created_at: 2023-11-24 08:41:59 +0000 UTC
application:
name: deleted (clg62220pvn60b59i82g)
id: clg62220pvn60b59i82g
version: 1
network:
address: 192.168.96.9
public_address: 18.158.139.177
services: []
stored_logs: []
error_message: “”
status_message: “”
config: {}
app_id: clg62220pvn60b59i82g
app_version: 1
image_id: “”
image_version: 0

AMS container “cm2rcta0pvn60b73hik0”

id: cm2rcta0pvn60b73hik0
name: ams-cm2rcta0pvn60b73hik0
status: error
node: lxd0
created_at: 2023-12-22 16:16:53 +0000 UTC
application:
name: deleted (cm2qucq0pvn60b73hihg)
id: cm2qucq0pvn60b73hihg
version: 1
network:
address: 192.168.96.7
public_address: 18.158.139.177
services: []
stored_logs: []
error_message: “”
status_message: “”
config: {}
resources:
memory: 4.00GB
disk-size: 32.00GB
app_id: cm2qucq0pvn60b73hihg
app_version: 1
image_id: “”
image_version: 0

AMS container “cmj4i5a0pvn60b73hoqg”

id: cmj4i5a0pvn60b73hoqg
name: ams-cmj4i5a0pvn60b73hoqg
status: error
node: lxd0
created_at: 2024-01-16 09:15:01 +0000 UTC
application:
name: deleted (cmim5120pvn60b73hon0)
id: cmim5120pvn60b73hon0
version: 1
network:
address: 192.168.96.2
public_address: 18.158.139.177
services: []
stored_logs: []
error_message: “”
status_message: “”
config: {}
resources:
memory: 4.00GB
disk-size: 32.00GB
app_id: cmim5120pvn60b73hon0
app_version: 1
image_id: “”
image_version: 0

AMS container “cmj9f3a0pvn60b73hpv0”

id: cmj9f3a0pvn60b73hpv0
name: ams-cmj9f3a0pvn60b73hpv0
status: running
node: lxd0
created_at: 2024-01-16 14:49:49 +0000 UTC
application:
name: nik_v1
id: cmj796q0pvn60b73hpsg
network:
address: 192.168.96.4
public_address: 18.158.139.177
services:

port: 5559
node_port: 10002
expose: true
protocols:
- tcp
  name: adb
  stored_logs: []
  error_message: “”
  status_message: “”
  config:
  platform: webrtc
  app_id: cmj796q0pvn60b73hpsg
  app_version: 0
  image_id: “”
  image_version: 0

AMS container “cmnafr20pvn6093ei1k0”

id: cmnafr20pvn6093ei1k0
name: ams-cmnafr20pvn6093ei1k0
status: running
node: lxd0
created_at: 2024-01-22 17:37:48 +0000 UTC
application:
name: demis_v13b
id: cmnabpi0pvn6093ei1ig
network:
address: 192.168.96.3
public_address: 18.158.139.177
services:

port: 5559
node_port: 10001
expose: true
protocols:
- tcp
  name: adb
  stored_logs: []
  error_message: “”
  status_message: “”
  config:
  platform: webrtc
  resources:
  memory: 4.00GB
  disk-size: 20.00GB
  app_id: cmnabpi0pvn6093ei1ig
  app_version: 0
  image_id: “”
  image_version: 0

AMS container “cmptnbi0pvn6093ei2o0”

id: cmptnbi0pvn6093ei2o0
name: “”
status: created
node: “”
created_at: 2024-01-26 16:19:58 +0000 UTC
application:
name: phoneScanner_v1
id: cmptnbi0pvn6093ei2ng
network:
address: “”
public_address: “”
services: []
stored_logs: []
error_message: “”
status_message: “”
config: {}
resources:
cpus: 2
memory: 4.00GB
disk-size: 8.00GB
app_id: cmptnbi0pvn6093ei2ng
app_version: 0
image_id: “”
image_version: 0

AMS container “cmrlsk20pvn6093ei2p0”

id: cmrlsk20pvn6093ei2p0
name: ams-cmrlsk20pvn6093ei2p0
status: running
node: lxd0
created_at: 2024-01-29 08:14:08 +0000 UTC
application:
name: marco_v13
id: clmtq620pvn60b34j1hg
version: 35
network:
address: 192.168.96.1
public_address: 18.158.139.177
services:

port: 5559
node_port: 10000
expose: true
protocols:
- tcp
  name: adb
  stored_logs: []
  error_message: “”
  status_message: “”
  config:
  platform: webrtc
  app_id: clmtq620pvn60b34j1hg
  app_version: 35
  image_id: “”
  image_version: 0

Error: Network ACL not found

LXD containers

±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| ams-clg63lq0pvn60b59i85g | STOPPED | | | CONTAINER | 0 | lxd0 |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| ams-cm2rcta0pvn60b73hik0 | STOPPED | | | CONTAINER | 0 | lxd0 |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| ams-cmj4i5a0pvn60b73hoqg | STOPPED | | | CONTAINER | 0 | lxd0 |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| ams-cmj9f3a0pvn60b73hpv0 | RUNNING | 192.168.96.4 (eth0) | | CONTAINER | 0 | lxd0 |
| | | 192.168.250.1 (anbox0) | | | | |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| ams-cmnafr20pvn6093ei1k0 | RUNNING | 192.168.96.3 (eth0) | | CONTAINER | 0 | lxd0 |
| | | 192.168.250.1 (anbox0) | | | | |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| ams-cmrlsk20pvn6093ei2p0 | RUNNING | 192.168.96.1 (eth0) | | CONTAINER | 0 | lxd0 |
| | | 192.168.250.1 (anbox0) | | | | |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| juju-7143b6-0 | RUNNING | 240.0.86.108 (eth0) | | CONTAINER | 0 | lxd0 |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| juju-bcc362-0 | RUNNING | 240.0.86.196 (eth0) | | CONTAINER | 0 | lxd0 |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+
| juju-bcc362-1 | RUNNING | 240.0.86.97 (eth0) | | CONTAINER | 0 | lxd0 |
±-------------------------±--------±-----------------------±-----±----------±----------±---------+

LXD storage

±-----------±-------±------------±--------±--------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
±-----------±-------±------------±--------±--------+
| data | zfs | | 17 | CREATED |
±-----------±-------±------------±--------±--------+
| juju-btrfs | btrfs | | 0 | CREATED |
±-----------±-------±------------±--------±--------+
| juju-zfs | zfs | | 0 | CREATED |
±-----------±-------±------------±--------±--------+

LXD networks

±--------±---------±--------±-----------------±-----±-------------------±--------±--------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
±--------±---------±--------±-----------------±-----±-------------------±--------±--------+
| amsbr0 | bridge | YES | 192.168.100.1/20 | none | AMS network bridge | 3 | CREATED |
±--------±---------±--------±-----------------±-----±-------------------±--------±--------+
| ens5 | physical | NO | | | | 0 | |
±--------±---------±--------±-----------------±-----±-------------------±--------±--------+
| lxdfan0 | bridge | YES | | | | 4 | CREATED |
±--------±---------±--------±-----------------±-----±-------------------±--------±--------+

LXD Cluster

LXD Network ACLs

n/a

LXD Network forwards

LXD resources

CPU (aarch64):
Vendor: AWS
Name: AWS Graviton2
Caches:
- Level 1 (type: Data): 64KiB
- Level 1 (type: Instruction): 64KiB
- Level 2 (type: Unified): 1MiB
- Level 3 (type: Unified): 32MiB
Cores:
- Core 0
Frequency: 0Mhz
Threads:
- 0 (id: 0, online: true, NUMA node: 0)
- Core 1
Frequency: 0Mhz
Threads:
- 0 (id: 1, online: true, NUMA node: 0)
- Core 2
Frequency: 0Mhz
Threads:
- 0 (id: 2, online: true, NUMA node: 0)
- Core 3
Frequency: 0Mhz
Threads:
- 0 (id: 3, online: true, NUMA node: 0)
- Core 4
Frequency: 0Mhz
Threads:
- 0 (id: 4, online: true, NUMA node: 0)
- Core 5
Frequency: 0Mhz
Threads:
- 0 (id: 5, online: true, NUMA node: 0)
- Core 6
Frequency: 0Mhz
Threads:
- 0 (id: 6, online: true, NUMA node: 0)
- Core 7
Frequency: 0Mhz
Threads:
- 0 (id: 7, online: true, NUMA node: 0)

Memory:
Free: 1.53GiB
Used: 14.22GiB
Total: 15.75GiB

GPU:
NUMA node: 0
Vendor: NVIDIA Corporation (10de)
PCI address: 0000:00:1f.0
Driver: nvidia (535.129.03)
DRM:
ID: 0
Card: card0 (226:0)
Render: renderD128 (226:128)
NVIDIA information:
Architecture: 7.5
Brand: Nvidia
Model: NVIDIA T4G
CUDA Version: 12.2
NVRM Version: 535.129.03
UUID: GPU-2802a2e1-4ba0-77a3-e92d-4984f9aed0e6

NIC:
NUMA node: 0
Vendor: Amazon.com, Inc. (1d0f)
Product: Elastic Network Adapter (ENA) (ec20)
PCI address: 0000:00:05.0
Driver: ena (6.2.0-1017-aws)
Ports:
- Port 0 (ethernet)
ID: ens5
Address: 0a:d6:fe:18:1c:27
Auto negotiation: false
Link detected: true

Disks:
Disk 0:
NUMA node: 0
ID: nvme0n1
Device: 259:1
Model: Amazon Elastic Block Store
Type: nvme
Size: 50.00GiB
WWN: nvme.1d0f-766f6c3030636465373231363832366135663364-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001
Read-Only: false
Removable: false
Partitions:
- Partition 1
ID: nvme0n1p1
Device: 259:4
Read-Only: false
Size: 49.90GiB
- Partition 15
ID: nvme0n1p15
Device: 259:5
Read-Only: false
Size: 99.00MiB
Disk 1:
NUMA node: 0
ID: nvme1n1
Device: 259:0
Model: Amazon Elastic Block Store
Type: nvme
Size: 100.00GiB
WWN: nvme.1d0f-766f6c3065633939313864363238646364613135-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001
Read-Only: false
Removable: false
Partitions:
- Partition 1
ID: nvme1n1p1
Device: 259:2
Read-Only: false
Size: 99.99GiB
- Partition 9
ID: nvme1n1p9
Device: 259:3
Read-Only: false
Size: 8.00MiB

Zpool status

pool: data
state: ONLINE
scan: scrub repaired 0B in 00:09:25 with 0 errors on Sun Jan 14 00:33:26 2024
config:

NAME        STATE     READ WRITE CKSUM
data        ONLINE       0     0     0
  nvme1n1   ONLINE       0     0     0

errors: No known data errors

pool: juju-zfs
state: ONLINE
scan: scrub repaired 0B in 00:00:00 with 0 errors on Sun Jan 14 00:24:02 2024
config:

NAME                                           STATE     READ WRITE CKSUM
juju-zfs                                       ONLINE       0     0     0
  /var/snap/lxd/common/lxd/disks/juju-zfs.img  ONLINE       0     0     0

errors: No known data errors

thanks a lot,
Demis.

gary-wzl77 · January 29, 2024, 10:53am

As you can see, the total disk capacity for ‘lxd0’ node is 96GB.
The disk sizes for the following instances are respectively as follows:

clg63lq0pvn60b59i85g (status: error): 3GiB
cm2rcta0pvn60b73hik0 (status: error): 32GB
cmj4i5a0pvn60b73hoqg (status: error): 32GB
cmj9f3a0pvn60b73hpv0 (status: running): 3GiB
cmnafr20pvn6093ei1k0 (status: running): 20GB

So, in addition to the ‘created’ instance ‘cmptnbi0pvn6093ei2o0’, you have 90GB disk space used, and only 6GB left, as the application ‘phoneScanner_v1’ requires 8GB to function. This is why it got stuck at ‘created’ status. As documented in the Capacity Planning:

If an instance stops with an error, its disk space is preserved for inspection. Other resources are released. Therefore, if you have many instances with an ‘error’ status, you might run out of disk space.

Hence, as long as you drop those errored instances to free up some disk space, the instance cmptnbi0pvn6093ei2o0 will be scheduled by AMS shortly.

BR
Garu

dfelici87 · January 29, 2024, 12:06pm

ok, it’s clear…

but when I Try to delete a container in error status I received this error:

ubuntu@ip-172-31-0-86:~/applications$ amc delete cm2rcta0pvn60b73hik0
The following instance will be REMOVED:

cm2rcta0pvn60b73hik0
Do you want to continue? [Y/n]: y
Error: operation not allowed

Why?
And How can I delete this container?

Thanks a lot,
Demis.

gary-wzl77 · January 30, 2024, 2:40am

Looks like you hit the following bug

Would it be acceptable if you could delete one of the running instances, such as cmnafr20pvn6093ei1k0 , to free up some disk space? Alternatively, we could provide you with a workaround by removing the instance object from the store.

Gary

dfelici87 · January 30, 2024, 8:17am

Hi,

I don’t understand how can I apply the workaroud?
Can you explain me please?

thanks a lot,
Demis.

gary-wzl77 · January 30, 2024, 9:47am

Sure.

Enter the ams/0 unit

 /snap/anbox-cloud-appliance/current/bin/juju ssh -m appliance:anbox-cloud ams/0

Dump the lxd0 node struct from the db

sudo ETCDCTL_API=3 /snap/etcd/current/bin/etcdctl --debug --cert=/var/snap/ams/common/etcd/client-cert.pem --key=/var/snap/ams/common/etcd/client-key.pem --insecure-transport=true --endpoints=240.0.86.97:2379 get /ams/1.0/nodes/lxd0 --print-value-only > lxd0.txt

Make a backup for the lxd0 data
```
cp lxd0.txt{,.bak}
```
Query that errored instance id like clg63lq0pvn60b59i85g from the lxd0.txt file and drop it from the used_network_endpoints and used_disk_space fields

Update the lxd0 struct in the db:

cat lxd0.txt | sudo ETCDCTL_API=3 /snap/etcd/current/bin/etcdctl --debug --cert=/var/snap/ams/common/etcd/client-cert.pem --key=/var/snap/ams/common/etcd/client-key.pem --insecure-transport=true --endpoints=240.0.86.97:2379 put /ams/1.0/nodes/lxd0

Delete the errored instance in the db:

 sudo ETCDCTL_API=3 /snap/etcd/current/bin/etcdctl --debug --cert=/var/snap/ams/common/etcd/client-cert.pem --key=/var/snap/ams/common/etcd/client-key.pem --insecure-transport=true --endpoints=240.0.86.97:2379 del /ams/1.0/containers/clg63lq0pvn60b59i85g

Restart the AMS
```
sudo snap restrat ams
```
Logout from the ams/0 unit and delete the relevant instance from the LXD
```
lxc delete ams-clg63lq0pvn60b59i85g
```

And then check the output of the ams ls.
Please give it a try.

NOTE: It is strongly advised not to interact with the database using etcdctl . The above comments are only intended to work around the issue where a base container cannot be deleted after its corresponding app has been removed from AMS.

BR
Gary

dfelici87 · January 30, 2024, 10:06am

thank you very much for your explanation.

BR
Demis.