@mlasch For cross compiling I would probably use something like sbuild(1). It manages temporary build chroots independent from your host system and can even create cross-arch chroots. There is a relatively simple guide to set that up in the Ubuntu wiki.
For cross building you would essentially mk-sbuild --target arm64 orcular and then follow the guide above and replace the build step with:
$ fakeroot ./debian/rules clean
$ sbuild --host=arm64